TOC

• What are message queues?
• Using message queues
• Tools and patterns
• Background (Microservices, Serverless)

An Introduction to Message Queues

Message Queues

• Queue software or service
• Message (payload application specific)
• Senders ("to whom it may concern")
• Receivers ("subscriber")
• Asynchronous (sender does not wait)
• Resilient (buffered when receivers are down)
• Acknowledgement and Redelivery

Working with Messages: AMQP

• exchange (publish to)
• queue (consume from, bound to exchange)

• direct ("round robin")
• fanout ("broadcast")
• topic (like fanout, but filter on routing key)

=> tryrabbitmq.com/

For Example

            beanstalkd

      


    SQS    Azure    IBM MQ

Leverage Message Queues

Start using message queues

Producer - Consumer

• Split into defined tasks
• Worker classes
• Message when a task is done, next worker listens to message

AMQP 0.9 libraries

• php-amqlib: PHP only implementation
• ext-amqp: Needs a compiled PHP module
• BunnyPHP: PHP only, also works with ReactPHP

STOMP libraries

• php-stomp: PHP only implementation
• ext-stomp: Needs a compiled PHP module

queue-interop

queue-interop tries to identify and standardize a common way for PHP programs to create, send, receive and read MQ messages to achieve interoperability.

use M6Web\Bundle\AmqpBundle\Amqp\Producer;
use App\Message\ImageUploaded;

class ImageUploadedProducer
{
  /** @var Producer */
  private $producer;

  public function publish(ImageUploaded $message) 
  {
    return $this->producer->publishMessage(
        json_encode($message),
        AMQP_NOPARAM,
        $message->getMessageProperties()
    );
  }
}
            

namespace App\Message;

class ImageUploaded implements \JsonSerializable
{
    private $path;
    private $mimeType;
    public function __construct($path, ...
    public function jsonSerialize() {
      $message = ['path' => $this->path];
      if ($this->mimeType) {
        $message['mimeType'] = $this->mimeType;
      }
        
      return $message;
    }
}
            

class ImageConvertProcessor
{
    public function process(ImageUpdated $msg)
    {
        if (!$msg->getMimeType()) {
            return;
        }
        
        if ($msg->getMimeType() === 'png') {
            ... $msg->getPath() ...
            

Patterns and Tools

Pattern: Delay slow task

• Fast return, delay slow task
• Cheap solution: Write to DB and cronjob
• If result is needed by client: Poll for status of task that was triggered

e.g. Rokka.io on AWS

• Respond with lower quality image transformation
• Low cache lifetime
• Send notification message
• AWS lambda to do expensive transformation
• Serve that image with long cache lifetime

Pattern: Decouple Event Listeners

• Do not handle events inline
• But send them to queue
• Filter which events are relevant
• Process queue and reproduce events

class WorkflowListener {
    public function eventHappened(
        string $type, 
        string $itemId
    ) {
      foreach ($this->handlers as $handler) {
        if ($handler->canHandle($type)) {
            $this->messageProducer->publish(
                $type, 
                $itemId
            );
            return;
        }
      }
    }
}
            

class WorkflowListener {
    ...    
    public function handleEvent(
        string $type, 
        string $itemId
    ) {
      $item = $this->repo->getEntity($itemId);
      foreach ($this->handlers as $handler) {
        if ($handler->canHandle($type)) {
          $handler->handle($type, $item);
        }
      }
    }
}
            

Pattern: Processing lots of data

• Split into subtasks
• Each step has a persistent outcome
  & new message(s)
• Scale individually
• Different entry points
• Priorities

e.g. Compile data into Elasticsearch

e.g. Compile data into Elasticsearch

• SAP sends data - need to handle push as fast as possible, data delivery process is blocking in SAP
• Dump data into message
• Consumer to store data locally
• New message that product needs reindexing
• Price changes: higher priority
• Changes from other data sources: Also message to reindex product

Workers in PHP?

• Long running tasks and PHP...
• Careful with simple arrays as "caching"
• Detach entities when using ORM
• Rejuvenation: exit after X messages processed

Task Runners

$ bin/console app:messages:image-worker

$ rabbitmq-cli-consumer \
  -e "bin/console app:messages:image-worker" \
  -c config.ini
            

# supervisor/imageworker.conf
[program:imageworker]
command=/usr/local/bin/rabbitmq-cli-consumer ...
process_name=imageworker_%(process_num)s
numprocs=4
autorestart=true
            

Worker scheduler

We built an autoscaler (in GO) that monitors queue sizes in RabbitMQ and adjusts the number of running worker instances to queue size

What can go wrong?

Everything

Log, Monitor, Alert

Missing Aknowledgement

• Network connection lost
• When client crashes without confirming
• Heartbeat: PHP has no threads.
  Long processing - no heartbeat

e.g. Blowing up the queues

• Brand information on each product
• Message that brand changed, send reindex message for every product with that brand
• Large brand, have to queue 50k products
• Brand worker ran out of memory and crashed hard after 40k messages
• Worker restarted, message immediately redelivered
• ad infinitum

The Redelivery Trap: Retry

• Catch exceptions, when recoverable: Copy message into delayed exchange that forwards it later.
• Keep a retry count, only retry if not > max retries
• When message redelivered (connection lost or hard crash): Put into retry exchange

Conclusions

Benefits

• Decoupling
• Redundancy
• Scaling
• Asynchronous

• Background processing in PHP
• Adapters for popular message queues
• Also "poor man's queues" with redis or Doctrine

Microservices

• Message queue like event system
• No need to know sender / receiver
• Mixed programming languages

Serverless

• System to run code fragments
• Queue-like communication
• Handled by the "Serverless" infrastructure

Questions / Input / Feedback ?

https://joind.in/talk/6b11a

Twitter: @dbu