Any tips to debug a long running stack creation/deletion? I am using the sst.Script construct to manage DB migrations. I've written a simple lambda that creates a database

onCreate

and removes a database

onDelete

. I've tested by putting the lambda behind the sst.API construct and it works fine. However, when I define the sst.Script construct with the same lambda,

sst start

and

sst remove

take very long (20+ minutes). Without the Script construct, these actions take around 1-2 minutes to deploy from scratch. Any tips to debug this, or is this the expected behavior with custom resources?

if you make the script do nothing does it still take 20min?

Funny you should ask that. I tried that the other day and it was faster. I will try again shortly (after the stack finally removes) to get a concrete example

I remember seeing this when my script errored out and cloudformation would get stuck

hmm.. when it took 20+ minutes to run, did the script fail in the end? Or did it successfully run? @Seth Geoghegan

Sounds like a VPC-bound function. Sometimes stack removal can hang on deleting ENIs for ~20 minutes.

@thdxr I commented out the lambda to a bare-bones implementation and it still took 21 minutes to remove. However, I think @Matt Morgan is onto something with his comment, as I did put this lambda function inside of a VPN. I assume this is necessary because it's running migrations against an RDS database.

ahh

Another strike against RDS in my book 🙂

Yup, pretty much. If I can't get the best devexp, I'm not that interested.

Agreed. Perhaps I'll create a script to run migrations locally or trigger the lambdas some other way. Although I prefer the custom resource approach,

yarn run db:migrate

or similar doesn't seem awful.

Why do you remove the function after running it? Just to ensure it doesn't run again?

Oh, I'm just testing the entire workflow to ensure it operates properly. I'm creating a long-lived RDS "dev" instance for a team of developers. I want each team member to check out the app, run

sst start

and have a database created for them within the development RDS instance. Using this approach, I hope to eliminate the need to run a local DB in docker (what they do today) and push all development for this particular service to the cloud. I'm mostly trying to ensure that

sst start

and

sst remove

deliver a good experience. I suppose it's not the end of the world if

sst remove

is slow, since it shouldn't happen terribly often. Just exploring options 🙂

Makes sense. I know there are some workaround/hacks to this problem. My org has just sucked it up when necessary and generally avoided VPCs if we can.

Yeah, this could very well be a "suck it up" moment.

Interesting.. I think we could manually remove the ENI similar to what this plugin does mentioned in the issue above https://github.com/medikoo/serverless-plugin-vpc-eni-cleanup

@Frank That is my assumption, since it only happened when I introduced the SST Script construct. Removing the Script construct eliminates the timeout. However, I have not seen a specific error that confirms the two are related. If it helps, this is how I'm defining my Script

new sst.Script(this, "DbMigrate", {
          defaultFunctionProps: {
            environment: { 
              SECRET_ARN: props.databaseSecretArn,
              DB_NAME: scope.stage,
              DEFAULT_DB_NAME: props.defaultDatabaseName
            },
            vpc: Vpc.fromLookup(this, 'VPC', { vpcName: props.vpcName }),
            permissions:["secretsManager"]
        },
        onCreate: "src/script.dbMigrate",
        onDelete: "src/script.dbMigrate",
    });

I see. And the error only happens sometimes?

I think a few things are happening here. 1) If my sst.Script has an error (e.g. timeout while trying to connect to RDS), I can get a

TCPConnectWrap.afterConnect

timeout. This timeout is a result of my crappy code 🙂 2) Setting up and tearing down the stack can be slow, particularly tearing down the stack. I'm consistently seeing 21 minutes for

sst remove

to completely remove a fairly basic REST API. I haven't seen this slowness create a timeout yet. The tricky part is this is painful to diagnose since each create/remove takes a loooong time 🙂

@Seth Geoghegan sorry for the late follow up. Without the

sst.Construct

does it take ~20min to

sst remove

?