With the continuing growth of microservices and a cloud best practice of designing decoupled systems, it’s important that developers have the ability to utilize a service or system that handles the delivery of messages between components and this is where SQS comes in.
Amazon SQS (Simple Queue Service) is a fully managed service offered by AWS, that works seamlessly with server-less systems, microservices or any other distributed architecture.
Although it’s simply a queuing service for messages between components, it does more than that, it has the capability of sending, storing and receiving these messages at scale without dropping message data as well as utilizing different queue types depending on requirements and includes additional features such as Dead Letter Queue.
It is also possible to configure the service using the AWS management console, the AWS CLI or using the AWS SDKs. Let me focus in some of the components to allow you to understand how the service is put to action.
The service uses three different element queues which are a part of your distributed system; these being the producers and the consumers and the third part is the actual queue which is managed by SQS and is managed across a number of SQS servers for resiliency.
Let, me explain how these components work together.
The Producer is responsible for sending messages to your queue. At this point, the SQS service stores the message across a number of SQS servers for resiliency within the specified queue. This ensures that the message remains in the queue if any failure occurs with one of the SQS servers.
The Consumers are responsible for processing the messages within your queue as a result when the consumer is ready to process a message from the queue; the message is retrieved and is then marked as processed by activating the visibility timeout on the message. This timeout ensures that the same message will not be read and processed by another consumer. When the message has been processed the consumer then deletes the message from the queue.
Before moving on I just want to point out a little more about the Visibility Timeout. As I said when a message is retrieved by consumer the visibility timeout gets started. The default time is 30 seconds but it can be set up to as long as 12 hours. During this period the consumer processes the message. If it fails to process a message perhaps due to a communication error the consumer will not send a delete message request back to SQS. As a result, if the visibility timeout expires and it doesn’t receive the requested message, the message will become available again in the queue for other consumers to process. This message will then appear as a new message to the queue. The value of your visibility timeout should be longer than it takes for your consumer to process your messages.
SQS Queue Types
I mentioned earlier that there are different types of queues. They are:
- Standard Queues
- FIFO Queues
- Dead Letter Queues
Standard Queues which are the default queue type upon configuration. It supports “at least once delivery guarantee” of messages. This means that the message might be delivered to the queue more than once which would make the message appear out of its original order of delivery. As a result, the standard queue will only offer the best effort on trying to preserve the message ordering from when the messages are sent by the producers. Standard queues also offer an unlimited number of transactions per second (TPS) making this queue highly scalable. If message ordering is critical to your solution then standard queues might not be the right choice for you. Instead, you would need to use FIFO Queues.
FIFO (First-In-First-Out) Queue is able to ensure the order of messages is maintained and that there is no duplication of messages within the queue. Unlike standard queues, FIFO queues do have a limited number of transactions per second. It defaults to 300 transactions per seconds (TPS) for all send, receive and delete operations.
Dead Letter Queue
A Dead Letter Queue differs to the Standard and FIFO queue as this Dead Letter queue is not used as a source queue to hold messages submitted by producers. Instead, it is used by the source queue to send messages that fail processing for one reason or another. This could be the result of code within your application, corruption within the message or simply missing information. Other way, if the message can’t be processed by a consumer after a maximum number of tries specified, the queue will send the message to a dead letter queue. A dead letter queue must be configured as the same queue type as the source is used against.
i. SQS Official Page: https://aws.amazon.com/sqs/
ii. SQS Developer Guide: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-how-it-works.html