Decoupling an Application
With Symfony Messenger
SymfonyCon Disneyland Paris - November 18th, 2022
© David Buchmann
We built a data kraken
flickr.com/photos/w-tommerdich/32823758672
Product API
massive central index of product data
- Constantly evolving since 10 years
- Collecting data from ~40 source systems
Legacy and future sources, always in the process of migrating to new services
- Cache all data locally in MySQL for quick access
- Aggregate data in Elasticsearch
First approach
- Cronjobs
- Incremental imports: last-modified
- Record command runs and use the timestamp
- Does not scale
- Error during processing: Start over
- Update products many times for different things
Encapsulate Logic
Extract the logic out of commands and controllers.
=> Refactored into Importer classes that can handle a single item at a time.
Decouple with message queue
- Importer can be called by command but also from message processor
- Importing triggers events on data changes
- Event listeners dispatch further messages as needed for next steps
- Perfect if source system can send messages
- Otherwise cronjob to only pull data and store it, then dispatch messages
Reaping the benefits
- Parallel processing of data
- Dynamically scale the number of workers
- Failure only affects single items, does not block
- easy retry
- abort after several failed attempts
Messages within the application
- Also split up heavy tasks in the application
- Multiple message-driven steps
- Easy to trigger the next step
- Using Symfony event system to decouple
- Event listeners can dispatch messages: e.g. we have 2 sources of product data, decide based on category which is relevant
e.g. Compile data into Elasticsearch
- SAP data must be processed as fast as possible, data delivery process is blocking in SAP
- Dispatch message with data
- First consumer stores data locally
- And dispatches an event, which makes another listener queue this product for reindexing
- Different priorities to process urgent changes fast (e.g. prices)
Flexible on the outside,
consistent on the inside
- Some source systems send messages on changes
- Other systems are polled for changes
- Daily import of CSV files with full data
- Temporal data with time limited validity
- We process everything as message
Any questions so far?
@dbu
Did i hear «microservices»?
Une Bataille, ~1750, François-Joseph Casanova
Messaging != microservices
- Our application has ~30 types of message and their processors
- Messages and processors all in the same application
- I would not want to maintain 30 tiny Symfony applications just for one single worker each
- Some separate applications that have nothing directly to do with the main data api
Symfony Messenger
Symfony Event System
... but decoupled
Symfony 6.3: External Events
Message System
- Message (payload application specific)
- Senders ("to whom it may concern")
- Receivers ("subscriber")
- Decoupled (mix systems or languages)
- Asynchronous (sender does not wait on receiver)
- Resilient (buffered when receivers are down)
- Reliable (acknowledgement and redelivery)
Why Symfony Messenger?
- High level functionality
- Abstract from specific message transport
- Nice and modern architecture
- Fully integrated into Symfony
- Previous attempts: queue-interop and enqueue
The flow of Symfony Messenger
https://symfony.com/doc/current/components/messenger.html#concepts
Send Message
Receiving Message
Message
- This is your class. It is serialized by the messenger.
- Small (Entity ID rather than the full entity data)
- Specific
class UpdatePromotion
{
public function __construct(
public string $id
) {}
}
Bus and Middleware
private MessageBusInterface $bus;
...
$this->bus->dispatch(new UpdatePromotion($id));
Envelope and Stamps
$stamps[] = new DelayStamp(20);
$this->bus->dispatch(
new UpdatePromotion($id),
$stamps
);
Handler
class UpdatePromotionProcessor
implements MessageHandlerInterface
{
public function __invoke(
UpdatePromotion $message
): void
{
// business logic update promotion on product
}
}
Message Transport
- Without a configured transport, messages are processed immediately (event system)
- AMQP with ext-amqp (rabbitmq, kafka)
- Doctrine (writes to database table and reads from table)
- Redis with ext-redis
- Amazon SQS
- Plug your own, e.g. to call a custom API
Retry
- In a large system, things will go wrong
- Software crashes, maschine reboots
- Network issues
- Service or system not available
- Actual logic bugs
- Message goes back to queue
- Exponential backoff
- Failure transport
Keep workers running
- Long-running PHP processes
- supervisord or similar to restart
- Rejuvenation: Message limit, or memory-limit / time-limit
- Restart workers on deployment
Autoscaler
- Own service to start and stop workers
- Application registers its workers with autoscaler
- Autoscaler monitors queue sizes
- Spawns more workers up to the limit when queue size goes up
What is going on?
- Archive queue with a TTL
- Import state log table, track all steps
Thank you!
@dbu
Middleware: Transaction ID
- Central service with current transaction id
- Used in logging, to trace requests across applications
- Receive message: record id in central service
- Send message: add transaction id as stamp
Middleware: Transaction ID
public function handle(Envelope $env, StackInterface $stack): Envelope {
$stamp = $env->last(IdStamp::class);
if (null === $stamp) {
// sending message, set the transaction id
$id = $this->transService->getId();
$env = $env->with(new IdStamp($id));
} else {
// receiving message, record transaction id
$this->transService->recordId($stamp->getId());
}
return $stack->next()->handle($env, $stack);
}
Transport Decorator: Message Priority
- Record priority from AMQP metadata
- Propagate priority on messages generated while processing a message
- Can be overwritten by explicitly setting the priority in the AmqpStamp
Transport Decorator: Message Priority
public function get(): iterable
{
foreach ($this->wrappedTrans->get() as $env) {
$stamp = $env->last(AmqpReceivedStamp::class);
// error handling if stamp not found...
$this->priority =
$stamp->getAmqpEnvelope()->getPriority();
yield $env;
}
}
Transport Decorator: Message Priority
private function send(Envelope $env): Envelope
{
$stamp = $envelope->last(AmqpStamp::class);
$attr = $stamp ? $stamp->getAttributes() : [];
if (!array_key_exists('priority', $attr)) {
$stamp = AmqpStamp::createWithAttributes(
['priority' => $this->getPriority()],
$stamp
);
$env = $env->with($stamp);
}
return $this->wrappedTrans->send($env);
}
Transport Decorator: Deduplicate
- We might send messages that are already waiting in the queue
- Semaphore system with Redis to know which product ids are currently in the queue
- Check semaphore and don't queue if locked
- Lock semaphore when sending
- Release semaphore when receiving
Transport Decorator: Amqp Routing Key
- Routing keys tell RabbitMQ into which queue to put a message
- Our middleware checks the message and may decide to set the routing key
- We have a separate queue for products with many variants, because they take a long time to process
- Currently needs a custom receiver to get messages from just one queue, Symfony pull request #38973