https://serverless-stack.com/ logo
Join Slack
Powered by
I’m using API GW v2. construct, but after having a...
# sst
a
I’m using API GW v2. construct, but after having a 500 error, my service doesn’t work anymore, and start to get 503 “service unavailable”. Looks like something is blocking my API after having a 500, and from so on starting to get 503. Anybody had the same problem?
f
Hey @Adrián Mouly, are u using a Lambda authorizer?
a
Yes sir.
f
the Lambda authorizer response by default cache for 5min
a
Yeah I removed that.
Looks like is still caching.
Because after waiting, I get 500 again.
I don’t get it 😞
f
So if you go into ur AWS Gateway console, it shows the route you are hitting is not using a Lambda authorizer right?
a
This is my authorizer:
Copy code
const authorizer = new HttpLambdaAuthorizer({
      authorizerName: 'authorizer',
      handler: new Function(this, 'Authorizer', {
        functionName: ,
        handler: ,
        tracing: Tracing.DISABLED,
        timeout: 30,
        permissions: ['ssm'],
      }),
      resultsCacheTtl: Duration.seconds(0),
      responseTypes: [HttpLambdaResponseType.SIMPLE],
    });
I mean, I’ve removed the TTL.
Putting it in 0.
But the authorizer is still there in my endpoint.
f
hmm… when you get 503, does the Lambda function handling the route get invoked?
a
No, looks like it never hits the authorizer and neither the final lambda.
I can only see the log in the API GW.
f
Did this start happening after making some changes?
a
Not sure when, it happens suddenly, and sometimes is fixed after deleting my stack and deploying again.
@Frank question… are HTTP APIs always Regional?
It’s so strange, having so many issues with the authorizer, even running with 
local
.
f
Yup, HTTP APIs are always Regional
lemme do a quick test
@Adrián Mouly I just created a Lambda authorizer example with simple response format https://github.com/serverless-stack/serverless-stack/tree/master/examples/api-auth-lambda-authorizer-simple-response
I took this example, and in the Lambda handler for the 
/private
route, I added 
throw new Error("manual")
So the first request hits the Lambda authorizer (authorized successfully), and then hits the 
/private
route handler, and returned 500
Hitting 
/private
again, 500 response again
Now I remove the 
throw
, hit 
/private
again, 200 response
a
Mmm.
So doing local development?
f
Yeah
Thought the setup in the example is very minimal… if you can tweak it to reproduce the 503 error, I can take a look at that.
a
Ok yeah, also on my authorizer Y removed the TTL.
I mean:
Copy code
resultsCacheTtl: Duration.seconds(0),
      responseTypes: [HttpLambdaResponseType.SIMPLE],
f
hmm.. btw there’s a recent service disruption with EventBridge https://status.aws.amazon.com
Not sure if it’s related if this started happening to u all of a sudden
a
Yeah no.
d
I also ran into this exact problem months ago when I tried to use the authorizer. Unfortunate to see that you hit the same scenario. My authorizer lambda, if it hits an error, just poops out, and stayed poops up until a few mins. By which time I assume AWS recycled it.
a
@Dennis Dang yeah I think that’s the problem, the lambda keeps failing and is not killed in a while.
Did you get 503 error? “service unavailable”?
d
exactly that
a
KIDDING?
d
?
a
I’m not feeling alone anymore, haha.
Did you get 503 too?
d
yeah i remember this experience very vividly. i did indeed get a 503 literally from my auth lambda after the first exception.
got a branch out there still hahahah, but i have no time to get back to this anytime soon.
a
Yeah.
Not sure why this happens thought.
d
i've resigned to keeping our auth layer within our middleware layer. was hoping we can move off it too. could be a local thing. i didn't test it on a deployed env.
a
😞
My authorizer worked fine with API GW v1.
Something is wrong with GW v2.
d
you're moving from the REST to HTTP one? sorry to hear :[
a
Never got 503 before, I’m getting it now that moved from v1 to v2 and rewrote it.
Yes.
Moving from REST to HTTP, you don’t recommend it?
d
I highly recommend it, but sucks to find out late that the auth lambda is unstable for local dev. Have you tried deploying your stack live in a test env? I'm curious what the behavior is there like
I didn't try that, so perhaps that can hint at the root cause.
a
I’m not having issues locally, only when it’s deployed.
I mean locally it works most of the time fine.
But once is deployed, sometimes get 503, and sometimes work fine after a while.
So difficult to know what is wrong.
f
Hey @Dennis Dang, was the 503 Service Unavailable only happened in local dev, and they worked well after u deploy them to live?
Open in Slack
PreviousNext
Powered by