Task queues

The pattern used by Celery workers is a push-pull tasks queue. One service pushes messages into a specific queue, and some workers pick them up from the other end and perform an action on them. Each task goes to a single worker. Consider the following diagram:

There's no bidirectional communication. The sender just deposits a message in the queue and leaves. The next available worker gets the next message.

This blind, unidirectional message passing is perfect when you want to perform some asynchronous parallel tasks, and that makes it easy to scale.

Plus, once the sender has confirmed that the message was added in the broker, we can have message brokers such as RabbitMQ offer some message persistence. In other words, if all workers go offline, we don't loose the messages that are in the queue.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset