I have thousands of messages in an SQS queue, and ...
# help
j
I have thousands of messages in an SQS queue, and well under the concurrency limit, but the consumer (a lambda function) is not increasing its concurrency to handle the messages. Anyone know why? The function has not been throlled at all according to the dashboard
j
Do you know how long the messages are sitting in the queue? quite a while it would seem? Check your
maxRecordAge
setting
Not sure that will default to 1 minute or not with CDK. After that, I would check your parallelizationFactor,
Most settings you will want to tune are in the
lambda.EventSourceMapping
config
another thing to consider is if there is a failure that is holding up the batch. Do you have a onFailure flow and DLQ?
f
Just expanding on Jarod’s point, a couple of parameters you can try tweaking and see if they make a difference. Here’s a code snippet for v1 construct.
Copy code
new Queue(stack, "Queue", {
  cdk: {
    queue: {
      receiveMessageWaitTime: Duration.seconds(0),
    }
  },
  consumer: {
    function: "src/function.handler",
    cdk: {
      eventSource: {
        batchSize: 10,
        maxBatchingWindow: Duration.minutes(0),
      }
    }
  }
});
j
Thanks for the input folks - I have all of the above settings on already actually. One thing is that the number of messages not visible was 120000. Do messages not visible count as "in flight"? If so I think that might be it because the max allowed messages in flight is 120k
But actually that would still be weird because the number of messages that WERE visible was also very high, so surely it should have scaled up to handle those
Speaking in the past tense because i purged the queue now