Hi all, I’m currently running into SNS message

Invalid parameter: Message too long

error. My payloads are variable in size and can exceed 256kb. I’m just wondering what is considered best practice for transfering data between services when payload size is too big for sns/sqs? I’ve been reading that S3 bucket and passing signature link via sns message is one method. Does anyone have any other options to investigate?

it depends so much on what type of data you are sending and at what rate 🙂 kinesis supports larger messages at higher throughputs than sns/sqs which is useful. Your s3 solution sounds good for low throughput applications but if you have a lot of data coming through you could see the cost of all the s3 gets and puts be pretty prohibitive

Yeah, the S3 solution is definitely the easiest. If you are already using FIFO queues, then breaking a message up into multiple parts works too.

I think a combination of EventBridge and DynamoDB is a better option. You write your message to DynamoDB and listen to the write event in your consumer functions, also EventBridge is like the CloudFormation of logging and monitoring and so you can mostly connect almost any AWS Service to EventBridge. Another Advantage is pricing, from what I remember, S3 is charged per hour while DynamoDB is charged for individual read writes. DynamoDB also offers you better latency than reading and writing to files. The simplest solution is using elasticache and ec2 and this would work extremely well but isn’t for serverless architectures. The serverless equivalent of elasticache with EC2 is the DynamoDB with EventBridge and Lambda architecture and would scale infinitely without reduced throughput.

Hey thanks guys, some interesting options to investigate!