is there any way to tell why a stack was modified I added on Serverless Stack #help

is there any way to tell why a stack was modified?...

Sam Hulick

08/20/2021, 8:55 PM

is there any way to tell why a stack was modified? I added one new API function and a route for it, and a whole bunch of policies changed, and I’m not sure why.

Sam Hulick

08/20/2021, 8:56 PM

to clarify, this is while in live dev mode

Frank

08/20/2021, 8:59 PM

Yeah, a new IAM role and policy should get created for the function.

Frank

08/20/2021, 8:59 PM

Or are you seeing more than that?

Sam Hulick

08/20/2021, 8:59 PM

why would that affect other functions in the same stack, as well as functions in another stack?

Frank

08/20/2021, 9:01 PM

if you revert the changes, and run

sst diff

, that shows the changes in more details

Sam Hulick

08/20/2021, 9:01 PM

can I run

sst diff

in another shell while

sst start

is running?

Frank

08/20/2021, 9:03 PM

running

sst diff

would changes the

.build

folder.. it might or might not cause an issue. Give it a try..

Sam Hulick

08/20/2021, 9:03 PM

will do. I’ll also just quit the live process and try a regular deploy & see if the same thing happens too

Sam Hulick

08/20/2021, 9:03 PM

to see if it’s an issue isolated to the live dev mode or not

Sam Hulick

08/20/2021, 9:05 PM

removed the API route, then ran diff.

Copy code

[~] AWS::Lambda::Function CopyOriginalFunc CopyOriginalFunc4D19A178
 ├─ [~] Code
 │   ├─ [~] .S3Bucket:
 │   │   └─ [~] .Ref:
 │   │       ├─ [-] AssetParametersc45dfba91f772913ab34aafb293943b690f422f11de2c7f698ce907430ea3286S3Bucket0CF0A5AA
 │   │       └─ [+] AssetParameters52b4fb516b56e477a5fd6a668dc58a4c11551a75ff70b9a4acb5573c55da99e2S3Bucket5488F6DE
 │   └─ [~] .S3Key:
 │       └─ [~] .Fn::Join:
 │           └─ @@ -8,7 +8,7 @@
 │              [ ]   "Fn::Split": [
 │              [ ]     "||",
 │              [ ]     {
 │              [-]       "Ref": "AssetParametersc45dfba91f772913ab34aafb293943b690f422f11de2c7f698ce907430ea3286S3VersionKey87C54803"
 │              [+]       "Ref": "AssetParameters52b4fb516b56e477a5fd6a668dc58a4c11551a75ff70b9a4acb5573c55da99e2S3VersionKeyDBA1C0E6"
 │              [ ]     }
 │              [ ]   ]
 │              [ ] }
 │              @@ -21,7 +21,7 @@
 │              [ ]   "Fn::Split": [
 │              [ ]     "||",
 │              [ ]     {
 │              [-]       "Ref": "AssetParametersc45dfba91f772913ab34aafb293943b690f422f11de2c7f698ce907430ea3286S3VersionKey87C54803"
 │              [+]       "Ref": "AssetParameters52b4fb516b56e477a5fd6a668dc58a4c11551a75ff70b9a4acb5573c55da99e2S3VersionKeyDBA1C0E6"
 │              [ ]     }
 │              [ ]   ]
 │              [ ] }

Sam Hulick

08/20/2021, 9:05 PM

and I didn’t modify the above func at all

Sam Hulick

08/20/2021, 9:06 PM

also:

Copy code

[~] AWS::IAM::Policy CopyOriginalFunc/ServiceRole/DefaultPolicy CopyOriginalFuncServiceRoleDefaultPolicy08109FE7
 └─ [~] PolicyDocument
     └─ [~] .Statement:
         └─ @@ -8,12 +8,9 @@
            [ ]   "Resource": "*"
            [ ] },
            [ ] {
            [-]   "Action": "s3:*",
            [+]   "Action": "ssm:*",
            [ ]   "Effect": "Allow",
            [-]   "Resource": [
            [-]     "arn:aws:s3:::dev-microservices-debug-stack-bucket83908e77-cvtpsun5ili4",
            [-]     "arn:aws:s3:::dev-microservices-debug-stack-bucket83908e77-cvtpsun5ili4/*"
            [-]   ]
            [+]   "Resource": "*"
            [ ] },
            [ ] {
            [ ]   "Action": "sts:AssumeRole",
            @@ -42,11 +39,6 @@
            [ ]   }
            [ ] },
            [ ] {
            [-]   "Action": "ssm:*",
            [-]   "Effect": "Allow",
            [-]   "Resource": "*"
            [-] },
            [-] {
            [ ]   "Action": [
            [ ]     "s3:GetObject*",
            [ ]     "s3:GetBucket*",

Sam Hulick

08/20/2021, 9:08 PM

seeing if this happens w/ regular deploys

Frank

08/20/2021, 9:09 PM

Yeah,

sst start

deploys the stub function code, so if you were to run

sst deploy

, the real function code will get deploy. That’s why

sst diff

shows the code change.

Sam Hulick

08/20/2021, 9:10 PM

well, the above is removing an API func while running

sst start

, not deploy

Sam Hulick

08/20/2021, 9:11 PM

sst start -> add a single API route/func -> all kinds of functions & policies get updated

Sam Hulick

08/20/2021, 9:11 PM

let me double check a few things on my end to make sure I’m not doing something weird like depending on some env var that’s changing

Frank

08/20/2021, 9:18 PM

Yeah, this could help: 1. start up `sst start`; 2. backup the CFN files in `.build/cdk.out`; 3. add a single API route/function; 4. hit enter to deploy the infrastructure changes; 5. backup the CFN files again in `.build/cdk.out`; Now we can diff the CFN files and see what changes, and if the changes matches what was deploy.

Sam Hulick

08/20/2021, 9:19 PM

ok, so with no start mode running, I deployed.. then diff’d, and it says no changes. I add the API route back in, and run a diff, and it shows all kinds of stuff going on. VERY weird stuff like:

Sam Hulick

08/20/2021, 9:20 PM

screenshot is easier to read

Frank

08/20/2021, 9:21 PM

it looks like that blocked just got moved around within the same IAM policy? nothing is really changed i think

Sam Hulick

08/20/2021, 9:22 PM

strange it’s doing that. but it shouldn’t trigger CF to update the resource, right?

Sam Hulick

08/20/2021, 9:22 PM

good idea on the output folder diff. I’ll try

Frank

08/20/2021, 9:22 PM

yeah it shouldn’t. If you run

sst deploy

again, does it show (no changes)?

Sam Hulick

08/20/2021, 9:23 PM

trying that now

Sam Hulick

08/20/2021, 9:23 PM

all it should do is add the one function & related policies

Sam Hulick

08/20/2021, 9:27 PM

Core stack unchanged. API stack, modifying IAM policies for every function. MediaProcessor stack, updating policies for all functions ok, I’m gonna diff the output folders to see what’s up. this is super weird.

Frank

08/20/2021, 9:28 PM

Is this when you run

sst tart

Sam Hulick

08/20/2021, 9:28 PM

this is deploy

Sam Hulick

08/20/2021, 9:29 PM

ok. running

sst start

now, and then I’ll back up the

.build/cdk.out

folder, remove the API func, then compare the two output folders

Sam Hulick

08/20/2021, 9:29 PM

I’m sure it’s probably me doing something dumb 😄 just not sure what it would be. nothing else is changing between deploys as far as I can tell

Frank

08/20/2021, 9:29 PM

it doens’t seem to be an

sst start

specific issue now since it’s happening to

sst deploy

as well.

Frank

08/20/2021, 9:30 PM

But yeah, diffing the CFN templates should help.

Sam Hulick

08/20/2021, 9:46 PM

so weird. the diff shows that it’s just a bunch of removing something & adding the exact same thing back in

Sam Hulick

08/20/2021, 9:46 PM

and it’s triggering an update of all those resources, needlessly

Sam Hulick

08/20/2021, 9:48 PM

is it because, in the main app level, I have

app.setDefaultFunctionProps

with:

Copy code

permissions: ['ssm'],

and then later in the same code block, I set default props again:

Copy code

permissions: [
      new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        actions: ['sts:AssumeRole'],
        resources: [process.env.SSM_ROLE_ARN],
      }),
      [coreStack.db, 'grantDataApiAccess'],
    ],

Sam Hulick

08/21/2021, 12:20 AM

sure enough, it’s that ssm permission. if I comment that out, so that the first call to

app.setDefaultFunctionProps

has no permissions set, then adding/removing a single route behaves as you’d expect when you deploy

Sam Hulick

08/21/2021, 12:22 AM

I think having various calls to

app.setDefaultFunctionProps()

is becoming problematic

Frank

08/21/2021, 4:53 AM

Hey @Sam Hulick, I’m not able reproduce the issue. Here’s the project I created to triage https://github.com/fwang/sst-triage-set-default-function-props

Frank

08/21/2021, 4:54 AM

I’m doing an

app.setDefaultFunctionProps()

index.js

, and then again inside a stack between 2 Function definitions.

Frank

08/21/2021, 4:56 AM

First

npm run deploy

takes a while to finish b/c it needs to deploy the 2 stacks. Then I ran

npm run diff

, and it showed no changes. Running

npm run deploy

again also showed no changes.

Frank

08/21/2021, 4:57 AM

Maybe take a look at the repo, and see what might be the difference?

Sam Hulick

08/21/2021, 11:41 PM

hey @Frank! Man, I dunno.. like, everything is different 😄 I’ve got multiple stacks, and an

sst.Api

instead of a SNS topic. I can duplicate my repo and try to cut it down to as simple as possible and see if I can replicate it.

Sam Hulick

08/22/2021, 12:18 AM

boom! replicated. Frank, I’ll create a private repo & add you to it

Sam Hulick

08/22/2021, 12:19 AM

deploy, then open up

infra/api/stacks.ts

and comment out the

/test1

route. do a diff, and you’ll see functions test2 & test3 are affected

Sam Hulick

08/22/2021, 5:49 PM

I just added @thdxr to the repo too, in case he wants to take a look

Frank

08/22/2021, 8:01 PM

I see. Let me take a look.

Sam Hulick

08/22/2021, 8:11 PM

thanks! it’s been bugging the hell out of me 😄

Sam Hulick

08/23/2021, 7:16 PM

Any luck on this?

Sam Hulick

08/24/2021, 3:25 PM

@Frank @thdxr hey guys, sorry to be a pain! 😬 this is about to become a “semi-blocker” for us (it doesn’t really prevent work, but it drastically slows us down) sorry, I know you’ve got your hands full as it is!

thdxr

08/24/2021, 3:27 PM

No worries! I just moved this issue yesterday to our roadmap. I'm going to address it either today or tomorrow

thdxr

08/24/2021, 3:27 PM

Should be a quick fix might be able to knock it out today

Sam Hulick

08/24/2021, 3:29 PM

awesome, thank you! 🙏

thdxr

08/24/2021, 3:33 PM

I believe it's this issue: https://github.com/serverless-stack/serverless-stack/issues/737

Sam Hulick

08/24/2021, 3:39 PM

weird. all I could tell that was off was that the ssm params stayed the same in the CF template, but it moved up or down. and so I assumed that’s what was causing it to update the whole function. but you’d think CloudFormation would be smarter than that.. 😄

thdxr

08/24/2021, 3:43 PM

@Frank can you confirm this was Sam's issue?

Sam Hulick

08/24/2021, 3:44 PM

I granted access to you & Frank (https://github.com/ffxsam/repro-sst-extraneous-update) so it should be super easy to test

Sam Hulick

08/24/2021, 3:45 PM

commenting/uncommenting out a single API func makes all of them update upon deploy

Frank

08/25/2021, 8:34 AM

Got it! Yup I promise to take a look at this tmr 😁

thdxr

08/25/2021, 12:02 PM

I fixed this already frank

Open in Slack

Previous Next