/ AWS

AWS SQS

SQS is one of several AWS services that allows communication between your AWS resources. Today I am learning about it in my Udemy course. Here are some notes.

SQS (Simple Queue Service) accepts messages from one microservice and delivers them to another. Your microservices do not communicate directly - this is asynchronous communication. SQS is able to scale up or down depending on traffic.

A “producer” sends messages into an SQS queue, and “consumers” poll the queue for messages, essentially asking if the queue has any messages for it. Consumers are applications that run on an EC2, Lambda, or other servers, and they process the messages. In a standard queue you can send as many messages to your queue as you want, and you can keep unlimited messages in the queue as well. SQS is very quick, but messages need to be small (256 kb or smaller).

Messages stay in the queue until the consumer reads and deletes it, up to 14 days.

Standard SQS message delivery is “at least once” delivery, meaning your consumers may receive a message more than once. Order is also not guaranteed, as SQS uses a “best effort ordering” policy.

This works nicely with auto-scaling groups/EC2 and Cloudwatch. You could set an alarm that watches the “queue length” metric, and when the alarm goes off, trigger your auto-scaling groups to increase the number of EC2 instances. In this case, your EC2s are your consumers, and you will have more of them to poll your SQS queue.

Message visibility timeout: there is a 30 second window in which the consumer that got the message has to process it, before it becomes visible again to other consumers. During this 30 seconds the message is invisible to others. If the first consumer can’t process it in that window, other consumers will also be able to process it, so you may have the same message processed multiple times. You can have your consumer call the Change Message Visibility API to get more time for processing. It’s also possible to set visibility to something other than 30 seconds when setting up the queue.

Dead letter queue: contains messages that have gone back into the queue multiple times. You can set a threshold for the maximum number of times a message goes back into the queue. Once the threshold is reached, the messages go to the dead letter queue. Another app can analyze these messages for debugging. Set the message retention period high, like to the 14 day maximum, for dead-letter queues.

Delay queue: can delay messages for up to 15 minutes so consumer can’t see them right away

Long polling: consumers can wait for messages if none are currently in the queue. Means fewer API calls. Lower latency because a consumer can act on a message as soon as it comes into the queue. Preferable over short-polling since it reduces cost. In the console, add any number from 1 - 20 in the Receive Message Wait Time field while editing your queue.

Extended client: a Java library that uses an S3 bucket for large files, greater than the SQS 256kb limit. A small meta-data message goes into the queue, and it tells the consumer where to go in S3 to get the files.

First In First Out: your other option instead of the standard queue. This queue preserves the order in which the messages are received by the queue. The consumer will receive them in order. It also has “exactly once” capability, so messages are only sent once.

  • deduplication: do not allow messages into the queue if a duplicate is sent by the producer
    • content-based: SHA-265 algorithm creates a hash based on the content, and if that hash is seen by the queue again, it is not allowed
    • message duplication ID: an actual id that is provided with the message
  • message grouping: a group of messages has the same message group id, and only one consumer can accept the group of messages

Using SQS with serverless

Since I work with serverless it’s important for me to understand how to create SQS quess with code, not just in the AWS console. Sometimes at work we use SQS with Lambda functions. The youtuber FooBarServerless thankfully has really great videos on serverless, and she happened to have a tutorial on using SQS with Lambda. Just what I need! Here is the video.

Also, here is the reference on serverless.com

It’s best not to call a lambda from another lambda function. If you make changes to your lambda you could be in trouble. Putting something in between, like SQS in this case is helpful. You would have a lambda as the producer, and another as a consumer.

When working with lambdas, you would first set up your SQS Queue as an AWS resources at the bottom of your file:

resources:
  Resources:
    MyQueue:
      Type: "AWS::SQS::Queue"
      Properties:
        QueueName: "DemoQueue"

And you connect your lambdas to the queue under the “events” section in your lambda code:

mySqsFunction:
  handler: handler.mySqsFunction
  events:
    - sqs:
        arn:
          Fn::GetAtt:
            - MyDemoQueue
            - Arn
        batchSize: 1

Your actual function code is in a separate handler file.