https://serverless-stack.com/ logo
#sst
Title
# sst
d

Dan Suceava

05/26/2022, 10:46 PM
My backend mono-repo deployments are getting ridiculously slow, approaching 30 minutes. While I could live with that (not happily) for the CI/CD pipeline, for local development it's obviously a problem. I'm looking for some recommendations on addressing this. Is there a way to still work with a mono-repo but split the stacks in groups somehow and just run certain ones during development? Or do I need to split my mono-repo all-together?
Oh, and Live debug is not an option anymore (unless I'm missing something), it does not work with private APIs - I have several API gateways on a private VPC, also several lambdas running on the VPC, and those never work with live debug
t

thdxr

05/26/2022, 11:05 PM
Hm it's 30min every time you run SST start?
d

Derek Kershner

05/27/2022, 1:10 AM
we have monorepos with SST, and the deploy times for each individual app in the monorepo have no noticeable increase in deploy time…are you sure you dont just mean you have a gigantic app with tons of stacks?
d

Dan Suceava

05/27/2022, 1:27 AM
@thdxr i'd say 15-20 minutes each time I run
npm run deploy
... the build times on CodeBuild are 20-30 mins, so that includes a couple of mins for provisioning and package installation @Derek Kershner yes, that's what I have ... all my services are in one repo, but the SST stacks are all deployed together ... at least that was my understanding of the recommendation of setting up the repo ... how are you splitting up your services? do you have separate /stacks folder for each service? So you deploy each service independently?
d

Derek Kershner

05/27/2022, 1:33 AM
we run fat-ish microservices, and each service has between 3 and 10 apps (which have between 1 and 15 stacks). I think the longest deploy is a couple minutes if starting from scratch, possibly up to 5. Generally our APIs are a stack all by themselves though, I think, cant see a reason to combine it with anything else… Not sure what the recommendation is persay, but its possible you have just outgrown it.
confirmed we have a separate app per API
f

Frank

05/27/2022, 5:54 AM
Hey @Dan Suceava i’m curious which step in
npm run deploy
is taking the most time. Here’s a high level breakdown of the steps: 1.
npm install
2. the build step in
npm run deploy
3. the uploading assets step in
npm run deploy
4. the generate CFN stack changesets step in
npm run deploy
5. the updating CFN stacks step in
npm run deploy
Few things to note: Step 1 happens only in CI i’m guessing. Step 2 can be slow if: • u have a lot of functions; or • u r building frontend, and that take long • u r building docker images Step 3 can be slow if u r uploading docker images to ECR Step 4 can be slow if you have a lot of resources (ie. 200+) in a stack Step 5 this step depends on what type of resources you are creating in ur stack. (S3 bucket gets created instantaneously, RDS cluster can take minutes to spin up)
It might be worth finding out the bottle neck here. Let me know if you need some help tracking it down 🙂
d

Dan Suceava

05/27/2022, 1:17 PM
@Frank I do have a couple of services that are building docker images, and that is where it seems to be taking most of the time. I guess my biggest problem is because we have all the services in one repo, and only one place for all the stacks, we currently deploy all stacks all together. I was hoping there was a way to split them up so we can deploy services separately, and it sounds like @Derek Kershner is doing just that. So to put it in terms of folder structure, what I have is:
Copy code
/src
  /service1
  /service2
  /service3
/stacks
and where I should be going is
Copy code
/src
  /service1
    /stacks
  /service2
    /stacks
  /service3
    /stacks
does that sound like a better solution?
d

Derek Kershner

05/27/2022, 1:46 PM
We are not using containers at the moment, but that is essentially what we do (although we have more than one
/src
as well). Not only does this help with time, organization, and potentially provisioning capacity, but it can also help when you get the circular dependency issues in stacks, as you can have an order to the app deployments, whereas a single app `synth`s all at once. This is mostly applicable to things named
lookup
.
t

thdxr

05/27/2022, 1:51 PM
That is one option but it's a shame to be forced to split up into multiple apps just for performance reasons if you don't feel compelled to for other reasons. The issue here is a CDK problem, all code runs synchronously because it's constructor based. So if you have a lot of functions to build or a lot of docker images to build, it can't happen in parallel
We have ideas on how to work around this to enable parallel builds but it's unlikely we can do so for native cdk constructs
I'm curious to know how long just
sst build
takes
another (potentially stupid) way would be to keep things as app and conditionally deploy stacks based off an env variable.
SERVICE=foo sst deploy
Then only initialize the foo related stacks.
d

Derek Kershner

05/27/2022, 1:54 PM
For me, its the same question as most architectural decisions, similar to a monolith vs microservices discussion: scale. I also don’t find any difference in moving a stack between apps, though I know we decouple significantly more than most.
Then only initialize the foo related stacks.
Does
sst deploy
not take a bunch of stacknames as args like
cdk deploy
?
Looks like just one, I see.
t

thdxr

05/27/2022, 2:09 PM
yeah the issue is sst build doesn't take stack names
since an app might need all stacks built even to just deploy 1
d

Dan Suceava

05/27/2022, 3:08 PM
@thdxr just ran
sst build
, it took just over 4 minutes
t

thdxr

05/27/2022, 3:09 PM
that's not good but it doesn't seem like that's th ebulk of the time hmm
yeah think we'd need to understand better what it's waiting on during sst deploy, we've been talking about printing out a better format so you can see exactly how long each resource is taking to deploy
d

Dan Suceava

05/27/2022, 3:11 PM
I think as Derek said, splitting them by service will provide some other benefits like better organization, with the number of stacks growing I can see that starting to be messy ... I might give that a try