https://serverless-stack.com/ logo
#help
Title
# help
j

Jeff Hanna

03/09/2022, 10:13 PM
hey guys, I’m getting an error that only shows when I am NOT live debugging I made a change to my auth trigger lambda to update my DB during sign in when I live debug it works perfectly, the DB is updated and a token is given to the client however when I deploy and test again, the DB query hangs and the lambda times out after 10s any ideas? thanks so much for your help! 🙏
t

thdxr

03/09/2022, 10:14 PM
What DB are you using?
j

Jeff Hanna

03/09/2022, 10:16 PM
the issue is on my dev/staging envs which is RDS postgres I am not seeing the issue on uat/prod envs which use aurora. even after I deploy (not live debugging) these envs are working well
t

thdxr

03/09/2022, 10:20 PM
When you say RDS postgres do you mean our new sst.RDS construct?
j

Jeff Hanna

03/09/2022, 10:52 PM
looks like it’s standard cdk rds.DatabaseInstance
t

thdxr

03/09/2022, 11:05 PM
and when running locally what db does it connect to?
j

Jeff Hanna

03/09/2022, 11:06 PM
same one, rds.DatabaseInstance, and I can see the query is successful and the auth flow succeeds
t

thdxr

03/09/2022, 11:21 PM
This database is public and exposed to the world?
is it possible this isn't the case when deployed in non-local mode?
j

Jeff Hanna

03/09/2022, 11:25 PM
yes, exposed to world, and I don’t have any conditional logic to change my stack if I am local
so I assumed when I local debug I am hopefully identical to when I deploy
I’m looking through CloudWatch to see if there’s any clues, also trying to get sentry working in case it’s easier to see anything there
I don’t know how to troubleshoot since it’s working on local
ok I found an Xray Trace ID and I can see successful select statements but then a start transaction and commit then nothing until 10s later when it is killed
t

thdxr

03/09/2022, 11:48 PM
are you missing an await anywhere?
j

Jeff Hanna

03/09/2022, 11:57 PM
I don’t think so, would that behavior be different with local debug vs deployed?
s

Seth Geoghegan

03/09/2022, 11:59 PM
10s sounds like a default lambda timeout. I suspect something is awry with how you're connecting to the DB and that connection is timing out
Is the request making it to your lambda (verifiable via cloudwatch?)
j

Jeff Hanna

03/09/2022, 11:59 PM
definitely, but why success in local debug and failure after deploying and how to troubleshoot since it works on local debug
ya the xray trace id shows the lambda logging that it is running sql queries
s

Seth Geoghegan

03/10/2022, 12:00 AM
oh, interesting. So it's actually connected to the DB and executing queries. That is odd
j

Jeff Hanna

03/10/2022, 12:01 AM
i see the sql logged, but not responses, but I am assuming they are successful since it moves onto the next query instead of throwing
s

Seth Geoghegan

03/10/2022, 12:01 AM
can you share the handler logic?
j

Jeff Hanna

03/10/2022, 12:01 AM
it is curious that the last query is an update
yes I will try to share a simplified failing example
t

thdxr

03/10/2022, 12:06 AM
can you add more logs to see where it's stuck?
j

Jeff Hanna

03/10/2022, 12:06 AM
my rds db has a vpc, and vpcSubnets set to vpc.publicSubnets, publicallyAccessible: true but I don’t suspect it’s an issue because my normal lambdas all succeed, it’s only the auth lambda updating the db that hangs
@thdxr yep, that’s what I’m doing now
s

Seth Geoghegan

03/10/2022, 12:08 AM
Yup, the VPC was my first suspicion
Your local dev env probably has some sort of VPN running that is allowing you access to the DB when running locally
but your lambda is running in a different environment when deployed
I ran into this a lot when connecting to RDS for Postgres (non-serverless, non-aurora) within Lambda. Huge PITA to get that configured properly
j

Jeff Hanna

03/10/2022, 12:12 AM
my api stack is working well though, so must be something different about the auth stack or auth trigger lambdas?
f

Frank

03/10/2022, 6:03 AM
I can think think of a couple of things worth check, ie. is this set?
Copy code
context.callbackWaitsForEmptyEventLoop = false;
But none of what I can think of explains why some of ur Lambdas work, but other don’t.
If you can’t get an answer from Sentry or Xray, maybe try running the queries that are timing out in the authorizer inside ur Api lambdas. And see if they succeed there. This would help ruling out if it’s the problem w/ the Lambda or w/ the queries.
Either way, I think maybe try finding out why the authorizer’s behavior is different.
j

Jeff Hanna

03/10/2022, 1:06 PM
hey all! thank you everyone for your help, I figured it out! The db instance was set up with only vpc.publicSubnets, changing it to public and private subnets fixed the problem, so I guess that means: 1. my API lambdas access the db through public subnets 2. when local debugging my auth lambdas also access the db through public subnets 3. when deployed, my auth lambdas access the db through private subnets