What Are Messaging Queues And How Do You Use Them

Messaging queues are used to transmit short messages between disconnected services. They can be used for communication in microservices as well as distributed systems that need to be kept in sync. We’ll discuss the fundamentals, as well as how to use them with AWS SQS.

How Do Messaging Queues Work?

You can think of message queues as the “glue” between disconnected services. They provide a link between message producers and message consumers.

For example, say you have a web application deployed on many different servers behind a load balancer. You have users uploading images, and you want to run some processing on the images on a different server. You could try to have each server manually send a POST request to the processing server, which would work, but what if that server failed to receive the request, or the producer wanted to confirm that the request was handled? It becomes hard to scale such a system, especially when you need to deal with many thousands of messages per second.

With a messaging queue, you can have all of those web servers send messages to a central queue. Then, the processing server will regularly check the queue and process each message. After processing is done, the consumer deletes the message from the queue, ensuring it won’t be processed twice by another consumer. If you care about the order, you can use a First-In, First-out queue, where messages must be processed in exactly the order they were received.

Messages can contain anything you’d like, though generally they should be kept fairly small. AWS SQS has a maximum message size of 256kb, which is more than enough for a bit of JSON containing some metadata. If you need to store large objects, you should upload them to a storage service like S3, then reference the URI in the message.

Queues also have many other benefits. They enable redundancy through message persistence; messages placed in the queue must be manually deleted, so you can ensure that even if it’s delayed, they are always processed at some point. If anything fails, the messages can be processed once the system comes back online.

Queues can also be used to batch process items. Rather than sending 100 database requests, you can process the whole queue at the same time and send one big request, which is often more performant.

Plus, most queue services like AWS SQS will have built in monitoring tools, so you can keep track of how many messages are being sent over time.

Getting Started with AWS SQS

Setting up an SQS queue is pretty simple. Head over to the SQS console and create a new queue:

You have two options available: Standard, which ensures messages are delivered at least once and does its best job to order them properly; and First-in, First-out, which ensure messages are processed only once and in exact order. Both have their perks though, as FIFO is generally less performant for workloads generating tons of messages per second.

Next up are the settings for how messages are handled. “Visibility Timeout” defines the length of time that a message needs to process successfully. When the consumer asks SQS for message updates, it will remove that message from the queue temporarily to allow the consumer time to process and delete it manually. This allows you to scale out and have multiple consumers, though if you want to send a message to each consumer, you’ll need to create multiple queues.

Message retention period refers to how long messages that aren’t processed will stay in the queue. Delivery delay can be used to delay messages being sent to producers, and “receive message wait time” can be used to enable long polling, reducing the number of empty message receives, which can reduce costs in high-performance systems.

Lastly, the access policy can be used to lock down this queue to specific IAM users or roles.

To use the queue, you can send messages to SQS programmatically using client SDKs, or access it from the AWS API. We’ll show it with code as it’s easier to understand.

Then, you can construct new SendMessageRequests, give it the message body and queue URL, and send it off with SQSClient.SendMessageAsync, which can be await‘ed and returns an HTTP status code.

To consume messages, you can use RecieveMessageAsync in a similar fashion. This returns an object that contains an array of messages, which you can send to the rest of your code for processing.

Of course, once you’re done, you’ll need to clean up after yourself. For each message that you handle, you’ll need to send a DeleteMessageRequest to SQS to remove it from the queue. You’ll want to do this before the message timeout, or else you may end up with messages being processed twice.