Message Delivery Guarantees

Good developers often associate certain words with particular pitfalls or techniques for avoiding them. Large dataset aggregation suggests the map-reduce pattern; immutability may suggest functional languages. In this section, we’ll build another association. Whenever you consider events, you should also consider message delivery guarantees.

Consider a simple welcome email that you might want to send each time a user creates an account. You could do so asynchronously with the Task module, but if the client fails or the server abruptly terminates, the email won’t be sent. Maybe you are willing to live with the consequences—if the email contained the user confirmation token, they can always request a new one later. In such cases, we provide at-most-once delivery. The email may be sent or not.

Maybe you’re not willing to drop the email, because losing it could impact adoption. This means you need a persistence mechanism. If the node responsible for sending the email catches on fire, another node needs to pick this job up. Once persistence becomes part of the equation, don’t reinvent the wheel. Use a third-party solution designed for the problem, such as RabbitMQ.

Once we add persistence, we can explore other delivery guarantees. In the email case, the best we can do is at-least-once delivery. For example, imagine that when the user creates an account, we store a job on RabbitMQ. A worker in your cluster picks up that job and sends the email, but while sending the email, an error happens. Was the email sent or not really?

Due to how emails work, we can’t quite answer this question. So the best strategy is to consider the job as failed and try again. Maybe the user will receive the email twice but they will hardly lose sleep over it.

At-least-once delivery is not enough in certain cases. For example, when billing a credit card, you surely don’t want to bill it twice. We need exactly-once delivery. It happens that, when communicating over the network, it is quite hard to guarantee exactly-once delivery. Imagine you send a message to an external server and it does not reply in 30 seconds. Does it mean it failed? Or does it mean it is busy and your message will be processed eventually? How long should you wait then?

If you need exactly-once delivery, the safest bet you can make is to guarantee that your messages are idempotent: if messages are idempotent, sending multiple messages won’t further affect the status of the system. In the billing case, this can be done by generating unique numbers for each transaction. When you message the billing service, you can include a unique ID. If the request fails, you send the same request, with the same unique ID. If the server has already seen and processed that unique ID, it can reply back that all is OK. If the server has not seen the ID, then it knows it has work to do.