https://serverless-stack.com/ logo
Join Slack
Powered by
whoa! SST/CDK just tried to do something really me...
# help
s
whoa! SST/CDK just tried to do something really messed up. I have a route that has a JWT authorizer. I modified it to be:
Copy code
'GET /media': {
    function: 'get-media.main',
    authorizationType: sst.ApiAuthorizationType.NONE,
  },
so I can test the endpoint without needing authorization. SST said “Press ENTER to redeploy infrastructure” and then this happened:
Copy code
Deploying stacks
Checking deploy status...
dev-microservices-core | UPDATE_IN_PROGRESS | AWS::CloudFormation::Stack | dev-microservices-core
dev-microservices-core | UPDATE_ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | dev-microservices-core Export dev-microservices-core:ExportsOutputRefUserPoolClient2F5918F753847A55 cannot be deleted as it is in use by dev-microservices-api
Checking deploy status...
Checking deploy status...
dev-microservices-core | UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS | AWS::CloudFormation::Stack | dev-microservices-core
dev-microservices-core | UPDATE_ROLLBACK_COMPLETE | AWS::CloudFormation::Stack | dev-microservices-core
it tried to delete my Cognito user pool client??
I’m in a stuck state 😕 I can’t even remove the API endpoint without it trying to delete my user pool client
lemme make sure my real stack & debug stack weren’t out of sync. I ran 
yarn deploy
now I’m running 
yarn start
again.
nope, that didn’t fix it either. I’ll try stopping the debug process, making API changes and just straight-up deploying them and see if that works
a
Have you set a removal policy on the cognito user pool?
Actually you don’t need to, the cdk sets it to retain by default. Even If sst would’ve attempted it, it would’ve failed.
The worst case scenario here would be that your stack which was being updated would dereference the user pool, I don’t think anything worse would have happened.
s
I set it to DESTROY so when the stack is deleted, it cleans up. crap.. even a regular deploy doesn’t work. 
yarn diff
doesn’t show anything happening regarding the user pool. this is pretty bad if we can’t modify API routes without taking down the user pool
yarn diff
shows this as the only change that will happen:
Copy code
Stack dev-microservices-core
Resources
[~] AWS::Lambda::Function CognitoEmailSender CognitoEmailSenderB4B6EAD9
 └─ [~] Environment
     └─ [~] .Variables:
         └─ [-] Removed: .SENTRY_RELEASE
I’ll try 
yarn deploy:api
which just deploys the API stack. hopefully that will work. I have no clue why it wants to delete the user pool client. @thdxr @Frank
a
Ah! That’s a problem. I also think that the pool shouldn’t be affected unless it’s in the same stack, which it isn’t in this case, right?
s
correct, the user pool is in the 
core
stack
so if I do 
yarn deploy
to do all stacks, that’s when changes in the API stack try to delete the user pool client in the Core stack 🤔
a
Is your user pool still available?
s
deploy:api
worked. it didn’t touch anything in the core stack, including the user pool
a
you could try removing the removal policy.
t
Hm
Did the naming of anything change?
Stack name, user pool name
s
nope
the extremely odd thing is, 
yarn diff
showed no changes being made to the Core stack. and then when I deployed all stacks, it tried to delete the user pool client
a
My suggestion would be to always create critical resources such as HostedZones, UserPools, CloudfrontDistributions via the console and import them. While I trust the CDK and open-source community, you never know what could go wrong.
s
unfortunately, manual creation is not an option
I think I’ll remove those 
REMOVAL
flags in my stacks, now that I don’t fully trust what CDK is doing
t
So running 
yarn sst build
And then 
yarn sst diff
shows no difference?
a
Oh! hard luck, I definitely don’t want some pissed dev running 
npx sst remove
on my prod stack. Thankfully seed is the abstraction that helps me sleep peacefully.
s
devs will be in their own isolated AWS accounts
@thdxr correct.. 
yarn diff
is clean, I modify the API route, then 
yarn diff
again and it doesn’t show it’ll change the Core stack. but upon doing 
yarn deploy
, it does
a
yeah, I’m doing the same but we have a central server at our office that has access to the prod stage as well.
t
That's bizarre. Afaik sst deploy isn't doing much but sending those synthesized stacks over. Hm
Think I need @Frank's eyes on this one
a
@Sam Hulick could you change the removal policy and see what happens then.
s
@Ashishkumar Pandey yeah, I changed that setting. I’ll try this again, just out of curiosity
t
Can you actually go to the AWS console and check the stacks events
Sometimes it has more step by step detail about what went down
s
nope, no extra detail. just boom, let’s update your stack.. ok, deleting your user pool client 😄
I could cp the raw JSON templates out of 
.build/cdk.out
between the route changes, then diff those and see what’s actually going on
ok, hang on.. maybe I missed this before. but here’s the diff:
Copy code
Stack dev-microservices-core
Outputs
[+] Output Exports/Output{"Ref":"UserPoolClient2F5918F7"} ExportsOutputRefUserPoolClient2F5918F753847A55: {"Value":{"Ref":"UserPoolClient2F5918F7"},"Export":{"Name":"dev-microservices-core:ExportsOutputRefUserPoolClient2F5918F753847A55"}}
I can’t tell what this is doing
a
I think it’s a cross stack reference.
s
whoa.. is it trying to change the name?
here’s the raw diff:
Copy code
--- before	2021-08-16 21:30:17.000000000 -0500
+++ after	2021-08-16 21:33:20.000000000 -0500
@@ -981,7 +981,6 @@
         },
         "Environment": {
           "Variables": {
-            "SENTRY_RELEASE": "414f847",
             "SENTRY_DSN": "<https://ba8217b648fb41f0b6ec7e70564e95d3@o75564.ingest.sentry.io/5904948>",
             "STAGE": "dev",
             "AWS_NODEJS_CONNECTION_REUSE_ENABLED": "1"
@@ -1295,6 +1294,22 @@
         "Ref": "AuthIdentityPool12DFB5E1"
       }
     },
+    "ExportsOutputRefUserPoolClient2F5918F753847A55": {
+      "Value": {
+        "Ref": "UserPoolClient2F5918F7"
+      },
+      "Export": {
+        "Name": "dev-microservices-core:ExportsOutputRefUserPoolClient2F5918F753847A55"
+      }
+    },
+    "ExportsOutputRefUserPool6BA7E5F296FD7236": {
+      "Value": {
+        "Ref": "UserPool6BA7E5F2"
+      },
+      "Export": {
+        "Name": "dev-microservices-core:ExportsOutputRefUserPool6BA7E5F296FD7236"
+      }
+    },
     "ExportsOutputRefRDSDatabaseA0AC51B29C8ABB11": {
       "Value": {
         "Ref": "RDSDatabaseA0AC51B2"
@@ -1319,14 +1334,6 @@
         "Name": "dev-microservices-core:ExportsOutputRefStorageBucket89AAB28650E5C65B"
       }
     },
-    "ExportsOutputRefUserPool6BA7E5F296FD7236": {
-      "Value": {
-        "Ref": "UserPool6BA7E5F2"
-      },
-      "Export": {
-        "Name": "dev-microservices-core:ExportsOutputRefUserPool6BA7E5F296FD7236"
-      }
-    },
     "ExportsOutputFnGetAttStorageBucket89AAB286Arn1F924F6C": {
       "Value": {
         "Fn::GetAtt": [
a
I don’t think so, the second refers the first.
s
ExportsOutputRefUserPoolClient2F5918F753847A55
doesn’t exist in the original template
a
Did you also remove the auth prop passed to the media stack when you switched to no auth?
s
nope
a
okay, was the auth prop being used anywhere else in the same stack?
s
would HTTP API authorizers have anything to do with the user pool?
as in.. if I made all routes not use JWT authorizers, would it try to alter the user pool?
that doesn’t seem likely
a
No, it shouldn’t, let me check.
s
AHA
t
I'm dying to know
s
@thdxr so when I change the route’s auth type from JWT to NONE, this suddenly appears in my outputs:
Copy code
ExportsOutputRefUserPoolClient2F5918F753847A55
and that was not there before
er sorry.. backwards. lemme go back & explain
my default auth type for all routes is JWT. I previously had this:
Copy code
'GET /media': {
    function: 'get-media.main',
    authorizationType: sst.ApiAuthorizationType.NONE,
  },
  'GET /tags': {
    function: 'get-tags.main',
    authorizationType: sst.ApiAuthorizationType.NONE,
  },
I modified it to this:
Copy code
'GET /media': 'get-media.main',
  'GET /tags': {
    function: 'get-tags.main',
    authorizationType: sst.ApiAuthorizationType.NONE,
  },
and then the user pool client output appeared
f
Sorry to jump into this thread late
a
Huh, are you using cognito as your jwt issuer? 🤔
s
yeah
a
Damn! That’s why. 😂
s
but I’m just disabling the authorizer for a single API route
f
From the original message, I don’t see anything about the Cognito User Pool was being removed
a
Yeah, The ref was being removed. Check this out about cross stack references - https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_core.CfnOutput.html
f
It seems only a stack export was being removed
Copy code
dev-microservices-core Export dev-microservices-core:ExportsOutputRefUserPoolClient2F5918F753847A55 cannot be deleted as it is in use by dev-microservices-api
s
@Frank see the most recent screenshot above. it tried to remove the user pool client
a
It says ExportsOutputRef, That’s a reference to a cross stack output.
f
It’s only trying to remove a stack export that contains the User Pool Client’s ref
The user pool client itself isn’t being removed… at least the screenshot doesn’t show that
a
Yep. Ditto. 👆
s
oh.. crap. sorry, this is where my CDK knowledge ends
t
I swear the screenshots looked different before...
f
I’m guessing the cognito user pool and the user pool client are being created in another stack?
a
Haha! Well, that’s how we learn, in high stress and shocking situations. 😂
s
yes, the Core stack. and when I change an authorizer in the API stack, this is happening
I still don’t understand why a change to an API Gateway route would cause this problem 🤔
f
Right, and by setting the route’s authorization to JWT, behind the scene, CDK knows you are trying to use a value from the Core stack (ie. User Pool Client’s ref), so it automatically create a stack output and exports it in the Core stack.
And then it imports it to the Api stack.
a
The CfnOutput is a resource that is created when you reference a construct across stacks. When you removed the jwt auth, the user pool reference isn’t being used anymore. Yep, like that. 👆
s
I assume it’s because none of my API routes use the JWT authorizer anymore. if I had 100 routes, and I modified some of them to not use the JWT authorizer, but many still did, this wouldn’t happen
f
When you remove the JWT authorizer, the User Pool Client’s ref is no long needed in this Api stack. So CDK removes the stack output from the Core stack. Hence you see 
core:ExportsOutputRefUserPoolClient2F5918F753847A55
being removed.
as long as 1 route still uses it, this won’t happen.
a
Yep, exactly.
s
ok.. and so the removal failed because something else in the API stack still needs that ref?
a
no, I think, it tried to remove because of the removal policy and failed because other stacks were using the pool.
s
I got rid of the removal policies so they’re defaulting to RETAIN now.
a
yep, that’s right.
s
hmm.. well. at any rate, I know how to work around it. I just have to deploy the stacks individually for now. thanks, all three of you, for your help! 😄
a
yep, that should help, my pleasure. 🙏
the only problem I see is that you’d have to think what you need to deploy, could get tricky.
s
oh this is only temporary. once I have a decent amount of routes migrated from my Serverless Framework stack, it won’t be an issue
a
oh, okay. sounds good.
Open in Slack
PreviousNext
Powered by