I’m building a feature which is going to be writing events to SQS from a lambda. The incoming throughput can go very high ~ 2000 req/s and these would be processed in realtime . Should I be using lambda and SQS for this or something else. Anyone processing data at such scale, how are you approaching it, should I be looking into MQTT or Kafka. I’m worried about back-pressure and throttling.

What does this come in from - apig?

yep, apigv2.

apigv2 -> lambda -> sqs -> lambda?

yep.

If your messages sizes are small and you only have a single consumer (not pub sub) then your setup is probably what's best afaik

yep, single consumer. the consumer lambda just validates the event payload and writes to dynamodb. How do I decide my batchsize, supposing that the throughput is low, say 100 req / s but the events need to be processed immediately and my batchsize is set for 1000?

no aws should invoke your lambda in parallel

oh okay.

so it's

maxBatchingWindow: time

and

batchSize

the default is to poll for 20 seconds in an attempt to reach the batchSize, and if not, process whatever there is.

Basically aws will trigger a lambda when either max batching window is hit or batchSize

max batching window adds polls to this, for instance if 40 seconds, it would poll twice

So if you set it to 10 seconds + 1000 items, worst case your events will be processed with a 10 second delay

Ah! good catch, perfect, this solves my dilemma. Great. 🙏

once you have this going you'll probably feel like "wow this is magic" because it'll perfectly scale

thank you @thdxr and @Derek Kershner. I appreciate it.

A company I advise made a mistake last week and accidentally 100x'd the events they were receiving. Nothing was effected, sqs scaled, lambdas scaled and dynamo scaled

My company still doesn’t believe that we serve ~ 10 M API requests in $85. 😂

just in case its important, the 20 second poll time setting is

receiveMessageWaitTime: Duration.seconds(20)

, and it is on the Queue itself (Dax’s are on the event source)

oh, good eye!

on ordering, it just orders based on time received to my knowledge. duplication is the stronger use case, and they provide an id param.

Ah! de-duplication would be lovely, thanks I’ll venture into more depth on the fifo based features. Thank you once again for all your assistance @thdxr and @Derek Kershner. 😁🙏

FIFO is 20% more expensive, if you can make it idempotent, you should

If fifo saves me time, I’ll prefer fifo as the feature will make me more money than what fifo might cost. I really don’t want my frontend logic to become convoluted to ensure idempotency. I’ll think on it, thank you for the heads-up.

makes sense to me

Yeah, on-demand is dirt cheap. 😂

Lower bandwidth on FIFO queues too, so watch out for that. The place I'd use Kinesis for is very high bandwidth and sharding data to help (but not guarantee) order. It's popular for IoT data ingestion, sharding on the device ID so that the same Lambda processes all messages from the same device.

I am going to take a day and do some in-depth reading but from what I understand AWS MSK is managed Kafka as a service and almost all large scale enterprises that rely on realtime event processing recommend it over Kinesis / SQS as it allows you to customise how you handle incoming throughput. For async event processing, EventBridge is the preferred choice as it can integrate with almost all AWS services and handle processing as you wish.

Eventbridge is an outlier in the comparison, it is primarily about fanout, not throttling/batching, and it itself has pretty low throughput compared to this group. SNS is a closer comparable.

Yep, I agree but when async processing comes into play, throughput is usually not the concern. async processing (in AWS context) usually tries to use other AWS services which EventBridge integrates with very well.

Makes sense, I was only mentioning due to your first message:

The incoming throughput can go very high ~ 2000 req/s

I am really not worried about the cost, regarding the setup, I believe AWS MSK will definitely make life easier, I got in touch with a couple of people at an analytics company who process at a scale of billions and they say MSK is a breeze compared to a K8s Kafka operator. Oh right, I need realtime processing and so EventBridge is a no go.

MSK is a breeze compared to a K8s Kafka operator.

With this, I can fully agree.

I believe AWS MSK will definitely make life easier.

With this, I think you need to add words:

because our use case is very complex, and we are thinking about the long run.

My use cases would never overgrow that of an analytics company. We’re currently using a third-party services for analytics, ads, marketing, etc. My use case is to make something custom in-house so that I can save cost. They’re very pricey like 10 to 50 x of my API costs and plus I am unable to audit their stats because when I aggregate cloudwatch logs data the numbers differ by 30 - 35% and that’s huge. A custom solution could also integrate better and rely on events directly from the backend and make my frontend apps lighter. So, that’s what I am planning to do.

I see, with that information, I think you are on the right track, but I wouldn’t necessarily discount kinesis just because some folks choose MSK. I’ve never been in your particular shoes (using Kafka for throughput reasons), but I was part of a decision on centralized eventing (where we chose Eventbridge over MSK for simplicity and serverless pricing). Your decision is quite different, though, and good luck!

Thank you, yep, can’t hurry with such impacting decisions, I am going to take my time and ingest TBs worth of info and feedback before I choose my way ahead. Obviously, I’ll keep this community informed or even write a blog post. 😅

Sounds good to me, please personally @ me if you do either of those last ones.

Sure, I will.

Kafka is really powerful, but overkill for point to point queuing. Kafka is an event bus. You can replicate it with AWS core services by combining SNS with a combination of SQS queues and Kinesis data streams in various configurations; which is more complex than just using Kafka. That complexity can be managed by making your own CDK contstructs. Your original description was just buffering requests from one lambda to another though. For that, SQS.

Yes, for the feature that I plan to implement in the coming weeks I’m going to go with SQS. I believe the FIFO feature would even reduce the duplicate events that might come and save me lot of money. The Kafka comes into play for near-realtime non-user impacting services where the throughput would not be dependent by the user but by my configuration. That’s where I might resort to MSK instead of EventBridge. There are many services which actually don’t need to be realtime but I know my stakeholders and their sporadic feature requests and so I’d rather make them near-realtime from the beginning. Also, thank you for your assistance @Adam Fanello. I appreciate it. 😁🙏