My backend mono-repo deployments are getting ridiculously slow, approaching 30 minutes. While I could live with that (not happily) for the CI/CD pipeline, for local development it's obviously a problem. I'm looking for some recommendations on addressing this. Is there a way to still work with a mono-repo but split the stacks in groups somehow and just run certain ones during development? Or do I need to split my mono-repo all-together?

Hm it's 30min every time you run SST start?

we have monorepos with SST, and the deploy times for each individual app in the monorepo have no noticeable increase in deploy time…are you sure you dont just mean you have a gigantic app with tons of stacks?

@thdxr i'd say 15-20 minutes each time I run

npm run deploy

... the build times on CodeBuild are 20-30 mins, so that includes a couple of mins for provisioning and package installation @Derek Kershner yes, that's what I have ... all my services are in one repo, but the SST stacks are all deployed together ... at least that was my understanding of the recommendation of setting up the repo ... how are you splitting up your services? do you have separate /stacks folder for each service? So you deploy each service independently?

we run fat-ish microservices, and each service has between 3 and 10 apps (which have between 1 and 15 stacks). I think the longest deploy is a couple minutes if starting from scratch, possibly up to 5. Generally our APIs are a stack all by themselves though, I think, cant see a reason to combine it with anything else… Not sure what the recommendation is persay, but its possible you have just outgrown it.

Hey @Dan Suceava i’m curious which step in

npm run deploy

is taking the most time. Here’s a high level breakdown of the steps: 1.

npm install

2. the build step in

npm run deploy

3. the uploading assets step in

npm run deploy

4. the generate CFN stack changesets step in

npm run deploy

5. the updating CFN stacks step in

npm run deploy

@Frank I do have a couple of services that are building docker images, and that is where it seems to be taking most of the time. I guess my biggest problem is because we have all the services in one repo, and only one place for all the stacks, we currently deploy all stacks all together. I was hoping there was a way to split them up so we can deploy services separately, and it sounds like @Derek Kershner is doing just that. So to put it in terms of folder structure, what I have is:

/src
  /service1
  /service2
  /service3
/stacks

and where I should be going is

/src
  /service1
    /stacks
  /service2
    /stacks
  /service3
    /stacks

does that sound like a better solution?

We are not using containers at the moment, but that is essentially what we do (although we have more than one

/src

as well). Not only does this help with time, organization, and potentially provisioning capacity, but it can also help when you get the circular dependency issues in stacks, as you can have an order to the app deployments, whereas a single app `synth`s all at once. This is mostly applicable to things named

lookup

.

That is one option but it's a shame to be forced to split up into multiple apps just for performance reasons if you don't feel compelled to for other reasons. The issue here is a CDK problem, all code runs synchronously because it's constructor based. So if you have a lot of functions to build or a lot of docker images to build, it can't happen in parallel

For me, its the same question as most architectural decisions, similar to a monolith vs microservices discussion: scale. I also don’t find any difference in moving a stack between apps, though I know we decouple significantly more than most.

yeah the issue is sst build doesn't take stack names

@thdxr just ran

sst build

, it took just over 4 minutes

that's not good but it doesn't seem like that's th ebulk of the time hmm

I think as Derek said, splitting them by service will provide some other benefits like better organization, with the number of stacks growing I can see that starting to be messy ... I might give that a try