Anyone else using CDK w/ Secrets Manager + RDS + Lambda? We just clicked the "rotate secret" button, and discovered that our Lambdas did not see the new secret. I'm stunned. We redeployed and it still didn't update. Today, we use environment variables to give the secrets to our lambas. Does anyone use the AWS SDK to directly fetch secrets from Secrets manager?

Hi @Drew, what do the deploy log say?

I use the sdk to fetch secrets directly in lambda. I actually even cache them and setInterval to update them every minute or so. Also did some funky stuff with

deasync

to prevent lambda from being invoked before secrets are fetched. I don't think passing through env vars is recommended as it negates some of the security reasons to use secrets manager in the first place

We do the same as @thdxr except we cache as long as lambda is warm. If we need to force an update we would redeploy

@Frank ooh, i'll look

Are you using secrets manager in lambda’s backing API calls? I’ve heard latency on getting secrets can be pretty bad? I guess caching is the only solution

Yeah, if we switch to fetching secrets in our Lambdas, then we'll have longer "cold starts" since we'll need to get the secrets before we'll be ready

Worth benchmarking before you declare it a problem.

Awesome, that's not too bad. Our cold starts are annoying, but that's a problem for later. 😄

I'm actually thinking about using SSM for this instead. More free but need to investigate the exact difference. Problem with secrets manager is you can't fetch multiple in one call afaik. Right now putting everything in one secret

I should have said, for clarity, this is for Parameter Store encrypted parameters

I’m using SSM but the parameters are fetched during compile/deployment time, so if those changes then I need to re-deploy the lambdas.

That’s pretty cool @Adrián Mouly — is there a moment that the service is unavailable while the lambdas are updated?

@Drew no it shouldn’t due this is hot swap done by AWS.

Yeah, it probably becomes invalid.

I would recommend to do this during maintenance window anyway.

The old existing connections would stay open

Depending which value are you storing… I use this to rotate SQL user/pass.. maybe you can create a new user/pass every time, and delete old one once upgraded.

Fetching parameters during runtime won't cause your avg response time to be slower, only cold starts. Remember not every invocation results in a cold start. Would recommend fetching on start and caching

I agree with @thdxr, this feels like premature optimisation. For us it's an added 90-100ms on a cold start. Also, you say you need to do this in a maintenance window so why add all that complexity when you could just redeploy?