I am creating a discussion forum type app for my l...
# guidem
Muhammad Ali
09/09/2021, 6:05 PMI am creating a discussion forum type app for my learning. There will be a table called "threads" and then "threads_comments". One of the objective i have is to keep a count of total "comments" from "thread_comments" into "threads". How would i do that? I can think of few approaches
1. Whenever a thread_comment is added, the same lambda can update the thread.total_replies count. The problem is that if multiple lambdas are doing the same thing, they may end up writing a wrong count
2. Use dynamodb triggers (? not sure whats the technical term here). Whenever a thread_comment is added, a message is posted in a queue and that queue can be used to update the thread table. would prefer if i can batch up the queue processing based on thread_id .
3. Probably use a cron job. which would run every 5 minutes, scan the tables which have been changed in last 5 minutes and do a scan of them and count the replies.
Is there any better approach?
r
Ross Coundon
09/09/2021, 6:07 PMA DynamoDB stream can act as a trigger, a lambda consumer will receive all changes to the table and perform whatever logic it needs
m
Muhammad Ali
09/09/2021, 6:14 PMThe problem in this approach is that it will be triggered for every comment. which could be expensive. i would like to "batch it" so that all the comments posted to a specific thread in last 5 min, should only trigger 1 lambda, once in 5 min. this is just a cost optimization approach.
r
Ross Coundon
09/09/2021, 6:14 PMYou can set a batch size for it
f
Frank
09/09/2021, 6:39 PM
Agreed with @Ross Coundon.
Frank
09/09/2021, 6:40 PM
@Muhammad Ali if you want to ensure the count is correct, you could use a cron job to recompute the comment counts, ie. nightly. But it seem irrelevant to ur questions 😁
m
Muhammad Ali
09/09/2021, 6:51 PM@Frank This is the 3 option that i have. but instead of nightly, i may be doing nit in after every 10ish mins or so
Muhammad Ali
09/09/2021, 6:51 PM@Ross Coundon how would i batch it up? I mean i want that kinesis stream for thread_1 should trigger only 1 lambda in 5 mins.
r
Ross Coundon
09/09/2021, 7:10 PMWell, if you truly wanted to only process every x mins, then cron's really your only option using something like an eventbridge rule. Doing it this way you're going to incur the cost of reading the data and then deciding what to update.
However, if I was building it I'd set the batchSize of the stream to be a sufficiently high number so more records are sent to the lambda.
Lambda invocations are cheap and all the changes are included in the event so usually no reads are necessary.
Ross Coundon
09/09/2021, 7:16 PMFrom docs
Batch size – The number of records to send to the function in each batch, up to 10,000. Lambda passes all of the records in the batch to the function in a single call, as long as the total size of the events doesn't exceed the payload limit for synchronous invocation (6 MB).
Batch window – Specify the maximum amount of time to gather records before invoking the function, in seconds.
m
Muhammad Ali
09/14/2021, 1:09 AM@Ross Coundon thanks for help. I have a follow up question. Does the SST has some configuration file that can be used to create this kinesis stream and then assign a lambda to it? or should it be done via aws ui?
Muhammad Ali
09/14/2021, 1:09 AMI want to make sure that my whole stack is replicateable in different regions
f
Frank
09/14/2021, 2:31 AM
Hey @Muhammad Ali yes you can. Here are some examples https://docs.serverless-stack.com/constructs/KinesisStream