https://serverless-stack.com/ logo
#random
Title
# random
s

Sam Hulick

08/25/2021, 7:28 PM
welp, that’s a new error from CloudFormation:
Received response status [FAILED] from custom resource. Message returned: Rate exceeded
😄
a

Ashishkumar Pandey

08/25/2021, 8:09 PM
You’re DDOSing AWS. 😂🙏
s

Sam Hulick

08/25/2021, 8:27 PM
AWS is DDOS’ing itself! 😄
AWS seems a bit stumped. but it looks like it’s probably something to do with all those
Custom::LogRetention
resources
@Frank @Jay is there any reason you didn’t just have SST create log groups via
AWS::Logs::LogGroup
instead of creating a bunch of custom resources to set the log retention?
or this a CDK thing
YEP
I went with this, to be super safe:
Copy code
logRetentionRetryOptions: {
      base: Duration.millis(500),
      maxRetries: 15,
    },
@thdxr @Frank I’d suggest maybe baking this into SST as a default, so no one has to suffer like I did 😄 and waste hours trying to figure it out
I can’t see any advantage to the defaults of 100ms and 3 retries. especially if it doesn’t do the trick
f

Frank

08/25/2021, 9:11 PM
@Sam Hulick yeah CDK doesn’t create the log groups for Lambda functions in CloudFormation. Lambda functions will automatically create the log group when they get first get invoked.
So if you are configuring the log retention, CDK has to do that in a custom resource.
Btw, how many functions did you add?
s

Sam Hulick

08/25/2021, 9:14 PM
will check in a sec. is there some quick way to tell? my code is pretty spread out
oh I can just look at the # of lines for “Building Lambda func..” in the CLI
I’ll let you know in a bit
f

Frank

08/25/2021, 9:16 PM
Oh are you deploying ur app to a fresh account?
s

Sam Hulick

08/25/2021, 9:16 PM
55 functions. yeah, deploying to stage prod in my primary account now
I got lucky in the other accounts for some reason.. didn’t hit that retry rate limiting. it’s just pure chance I guess
sorry, 42 functions in the API stack, actually. 55 across all stacks
f

Frank

08/25/2021, 9:24 PM
ah I see.. that makes sense
s

Sam Hulick

08/25/2021, 9:28 PM
thanks!
what an obscure issue to run into 🙂
t

thdxr

08/25/2021, 10:03 PM
Woah I think I just got this too for the first time ever
I'm not even doing anything crazy
Copy code
Received response status [FAILED] from custom resource. Message returned: Waiter InvalidationCompleted failed: Max attempts exceeded (RequestId: ded6e1ea-8b11-44ae-ae1e-4a4b8d24c1a3)
s

Sam Hulick

08/25/2021, 10:04 PM
damn it, AWS! 😄
t

thdxr

08/25/2021, 10:04 PM
This was on deploying a StaticSite which has a CRD for replacing values
s

Sam Hulick

08/25/2021, 10:04 PM
let me know if the log retention retry values above work for you
if it’s a log retention resource that failed, of course
f

Frank

08/25/2021, 10:22 PM
@thdxr the custom resource triggers a CF invalidation and waits for invalidation to finish. I think that’s causing the error.
t

thdxr

08/25/2021, 10:23 PM
Yeah it worked on the second try
f

Frank

08/25/2021, 10:23 PM
I see
t

thdxr

08/25/2021, 10:24 PM
I noticed this was in python the other day, was there a specific reason for it?
f

Frank

08/25/2021, 10:24 PM
b/c I want to show off my python skillz
the code is based on CDK’s custom resource in
@aws-cdk/aws-s3-deployment
And we rewrote some of it to get around 500MB Lambda disk space, better S3 cache policy, and made it faster
t

thdxr

08/25/2021, 10:27 PM
Ah ok cool
s

Sam Hulick

08/25/2021, 10:28 PM
thanks for your help today, guys 💪 I managed to get our new SST back end deployed to prod in our primary account
j

Jay

08/26/2021, 12:41 AM
That’s awesome!
3 Views