I m having trouble starting the local environment It was wor Serverless Stack #help

I’m having trouble starting the local environment....

Ömer Toraman

04/27/2022, 8:05 PM

I’m having trouble starting the local environment. It was working all fine till today. I’m not really sure what the problem is. I removed all of the stacks through the CloudFormation Dashboard, and tried to deploy again, but no good. So I tried to create a brand new SST app, but the same problem persists. I’m not able to deploy the debug stack on a fresh serverless stack app.

Copy code

$ npx create-serverless-stack@latest demo --language typescript --use-yarn

$ cd demo

$ aws-vault exec sg-omer -- npx sst start --stage dev --verbose

I don’t see any CloudFormation Stack in progress:

Ömer Toraman

04/27/2022, 8:34 PM

I even deleted the

CDKToolKit

stack. It is interesting that it does progress creating the

CDKToolKit

stack. But after that, it seems to be stuck. I don’t know if it is relevant, but it also states that “the ongoing operation is not stable”

Ömer Toraman

04/27/2022, 8:36 PM

The CDKToolKit Stack is created, but I don’t see any progress on the terminal output

Ömer Toraman

04/27/2022, 8:56 PM

Still…

Ömer Toraman

04/27/2022, 9:00 PM

Sorry for pinging but.. @Frank @thdxr

Frank

04/27/2022, 9:02 PM

Hey @Ömer Toraman, can you send me ur

.build/sst-debug.json

Ömer Toraman

04/27/2022, 9:02 PM

Sure

Frank

04/27/2022, 9:05 PM

can u check what state is the

development-softgiving-debug-stack

stack in?

Ömer Toraman

04/27/2022, 9:06 PM

None… It does not exist at all

Frank

04/27/2022, 9:07 PM

yeah, from the debug log, it seems it’s taking forever to upload the Lambda zip files to S3

Ömer Toraman

04/27/2022, 9:07 PM

Checked all of the statuses from the dropdown:

Ömer Toraman

04/27/2022, 9:08 PM

The S3 bucket list seems to be buggy as well, it is showing me previously deleted buckets:

Frank

04/27/2022, 9:09 PM

If u look at ur debug log:

Copy code

[2022-04-27T23:45:58.256] [TRACE] core - deploy stack: run cdk deploy: stderr: [0%] start: Publishing bdb08b9984816ecf43301a0c6d8913e270ee2448940b6b95e430dd222a4d90be:609221238233-us-east-1

Copy code

[2022-04-27T23:56:10.948] [TRACE] core - deploy stack: run cdk deploy: stderr: [25%] success: Published bdb08b9984816ecf43301a0c6d8913e270ee2448940b6b95e430dd222a4d90be:609221238233-us-east-1

It took 11 minutes to upload a file that’s likely a few KB in size

Ömer Toraman

04/27/2022, 9:10 PM

So do you think it is about the internet speed?

Frank

04/27/2022, 9:10 PM

can u check ur aws health page https://health.aws.amazon.com/health/status

Frank

04/27/2022, 9:11 PM

Can u try manually creating a bucket on S3 and upload a 1 MB file in there and see if it’s quick?

Ömer Toraman

04/27/2022, 9:12 PM

Can’t see any issue on aws health

Ömer Toraman

04/27/2022, 9:16 PM

It uploaded quickly:

Ömer Toraman

04/27/2022, 9:27 PM

it does create the CDKToolKit pretty fast

Frank

04/27/2022, 9:38 PM

I see.. looking at the debug log again, yeah, it seems the uploading was actually fast. This was slow:

Copy code

[2022-04-27T23:45:58.872] [TRACE] core - deploy stack: run cdk deploy: stderr: [90m[0%] check: Check <s3://cdk-hnb659fds-assets-609221238233-us-east-1/bdb08b9984816ecf43301a0c6d8913e270ee2448940b6b95e430dd222a4d90be.zip>[39m

and then

Copy code

[2022-04-27T23:56:10.957] [TRACE] core - deploy stack: run cdk deploy: stderr: [90m[25%] check: Check <s3://cdk-hnb659fds-assets-609221238233-us-east-1/0d89df84ea307d9cb0d70fbd76481e4e86f449d075a2a082cc2d323ee69a98e4.zip>[39m

It seems CDK is make some call to S3 to check certain things, and that’s taking long.

Frank

04/27/2022, 9:41 PM

Looking at CDK’s source code, the above line is coming from here https://github.com/aws/aws-cdk/blob/2415a802a4fb59c4c00cf014f736a46b2acd718b/packages/cdk-assets/lib/private/handlers/files.ts#L34

Frank

04/27/2022, 9:42 PM

So it’s taking along to get from the above line to this line https://github.com/aws/aws-cdk/blob/2415a802a4fb59c4c00cf014f736a46b2acd718b/packages/cdk-assets/lib/private/handlers/files.ts#L139

Ömer Toraman

04/27/2022, 9:56 PM

It did create some of the stacks

Ömer Toraman

04/27/2022, 9:56 PM

but it is taking really too long

Ömer Toraman

04/27/2022, 9:56 PM

the process is still going on

Frank

04/27/2022, 10:12 PM

hmm.. can you try running

yarn cdk deploy

and see if deploying using CDK directly without going through SST is fast?

Frank

04/27/2022, 10:12 PM

Can you also send me ur latest sst-debug.json?

Ömer Toraman

04/27/2022, 10:38 PM

it finally deployed

Ömer Toraman

04/27/2022, 10:39 PM

I guess it took more than an hour lol

Ömer Toraman

04/27/2022, 10:39 PM

Sure, DM’ing you

Ömer Toraman

04/28/2022, 12:13 AM

It successfully deployed in just a couple of minutes, after my SSO login was expired and I relogged in

Frank

04/28/2022, 12:22 AM

Oh do u think it’s related to SSO?

Ömer Toraman

04/28/2022, 12:40 AM

I have no idea, the only explanation I have… Didn’t change anything at all

Ömer Toraman

04/28/2022, 12:40 AM

Idk how it could relate to aws-vault or sso

Frank

04/28/2022, 3:19 AM

hmm one more thing we can check. If you go into AWS CloudTrail, you can see if there are failed requests during the time when sst deploy was slow.

Frank

04/28/2022, 3:21 AM

If there were anything wrong with the previous SSO session, you might see a series of Rate Exceeded errors. That’d cause the AWS SDK calls to keep retrying, which could lead to the slowness.

Frank

04/28/2022, 3:21 AM

Just a guess.

Open in Slack

Previous Next