https://serverless-stack.com/ logo
#help
Title
# help
e

Erik Robertson

12/21/2021, 9:24 PM
I have a very weird error. I had a functional stack. I decided to reduce the AWS permissions on the IAM user that deploys/drops SST stacks and have actually switched to using a service role for CloudFront using the --role-arn flag. That involved a lot of steps including re-bootstrapping CDK and lot of trial and error to get the policies right. Now it deploys and works again EXCEPT for one specific API call that with certain parameters and not others generates a runtime error in the live dev mode. I've stepped thought it and the error occurs after "my" lambda code ends and returns the result. The error seen on the server is a 403 access denied on S3
Copy code
a625b40c-04d9-4cbe-90b7-8324e904055a REQUEST DEV-ERIK-dwam-back-my-sta-ApiLambdaGETassets0D68A0-FzosTE9J9tP7 [src/routes/assets_get.main] invoked by API GET /assets
a625b40c-04d9-4cbe-90b7-8324e904055a RESPONSE {"headers":{"Content-Type":"application/json"},"statusCode":200,"body":"{\n  \"csvHeader\": \"latitude,longitude\",\n  \"csvData\": \"37.65728378,-122.4156265\\r\\n37.65734863,-122.41... 177051 more characters"}
Failed to upload payload to S3. AccessDenied: Access Denied
    at Request.extractError (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/services/s3.js:714:35)
    at Request.callListeners (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'AccessDenied',
  region: null,
  time: 2021-12-21T21:20:09.867Z,
  requestId: 'R0H1MF37YG9JSGTQ',
  extendedRequestId: 'SXswySS3eGF3DkYb/YjNO8YIw2T911TZViSQZbivYh6VHH71nfQP33L1lTVTA6oixDM5oFvwEuc=',
  cfId: undefined,
  statusCode: 403,
  retryable: false,
  retryDelay: 43.50154802428319
}
and when I check the Cloudwatch logs I see this 404 on S3 :
Copy code
2021-12-21T22:20:05.498+01:00	START RequestId: a625b40c-04d9-4cbe-90b7-8324e904055a Version: $LATEST

2021-12-21T22:20:05.517+01:00	2021-12-21T21:20:05.516Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO connectAndSendMessage()

2021-12-21T22:20:05.641+01:00	2021-12-21T21:20:05.641Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO ws.onopen

2021-12-21T22:20:05.641+01:00	2021-12-21T21:20:05.641Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO sendMessage() - send request

2021-12-21T22:20:05.643+01:00	2021-12-21T21:20:05.642Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO sendMessage() - sending request via WebSocket

2021-12-21T22:20:05.644+01:00	2021-12-21T21:20:05.643Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO sendMessage() - start keep alive timer

2021-12-21T22:20:10.189+01:00	2021-12-21T21:20:10.188Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO ws.onmessage {"action":"client.lambdaResponse","debugRequestId":"a625b40c-04d9-4cbe-90b7-8324e904055a-1640121605516","stubConnectionId":"Kt-45fnPPHcCGBQ=","payloadS3Key":"payloads/a625b40c-04d9-4cbe-90b7-8324e904055a-1640121605516-response"}

2021-12-21T22:20:10.189+01:00	2021-12-21T21:20:10.189Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO receiveMessage()

2021-12-21T22:20:10.189+01:00	2021-12-21T21:20:10.189Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO receiveMessage() - received payloadS3Key

2021-12-21T22:20:10.256+01:00	2021-12-21T21:20:10.256Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO [AWS s3 404 0.065s 0 retries] getObject({ Bucket: 'dev-erik-dwam-back-debug-stack-bucket83908e77-1awrh01i15tnq', Key: 'payloads/a625b40c-04d9-4cbe-90b7-8324e904055a-1640121605516-response' })

2021-12-21T22:20:10.259+01:00	2021-12-21T21:20:10.259Z a625b40c-04d9-4cbe-90b7-8324e904055a ERROR Unhandled Promise Rejection {"errorType":"Runtime.UnhandledPromiseRejection","errorMessage":"NoSuchKey: The specified key does not exist.","reason":{"errorType":"NoSuchKey","errorMessage":"The specified key does not exist.","code":"NoSuchKey","message":"The specified key does not exist.","region":null,"time":"2021-12-21T21:20:10.253Z","requestId":"0G7QZ64M4EVDDPV5","extendedRequestId":"Zk5DMRznR+DMrIMUD0jYqUlGn2aejxk6E+Gl0WupDfZCynNyUKOzjWUJPK10Pseeu80tzGB6t4o=","statusCode":404,"retryable":false,"retryDelay":8.646154170322662,"stack":["NoSuchKey: The specified key does not exist."," at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/services/s3.js:699:35)"," at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)"," at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)"," at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:688:14)"," at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)"," at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)"," at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10"," at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)"," at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:690:12)"," at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)"]},"promise":{},"stack":["Runtime.UnhandledPromiseRejection: NoSuchKey: The specified key does not exist."," at process.<anonymous> (/var/runtime/index.js:35:15)"," at process.emit (events.js:314:20)"," at process.EventEmitter.emit (domain.js:483:12)"," at processPromiseRejections (internal/process/promises.js:209:33)"," at processTicksAndRejections (internal/process/task_queues.js:98:32)"]}

2021-12-21T22:20:10.275+01:00	END RequestId: a625b40c-04d9-4cbe-90b7-8324e904055a

2021-12-21T22:20:10.275+01:00	REPORT RequestId: a625b40c-04d9-4cbe-90b7-8324e904055a Duration: 4775.65 ms Billed Duration: 4776 ms Memory Size: 1024 MB Max Memory Used: 28 MB XRAY TraceId: 1-61c24505-337b01bf10f9f70f44866cac SegmentId: 247605f67fd8dd26 Sampled: true

2021-12-21T22:20:10.275+01:00	Unknown application error occurred
I'm eager to see if any of you have any idea ?
BTW I have no S3 related code in my lambda, this is environmental stuff outside the lambda
I think that the other API calls (or calls to this one with other params) for some reason don't dump the output payload to S3 which is why I'm not getting an error for them...
Also the deployed version works fine. This is only in live dev mode
f

Frank

12/21/2021, 11:00 PM
Hey @Erik Robertson,
sst start
uses a websocket to send/receive data between the real Lambda function and ur local machine. Websocket has certain packet size limit 64KB if i call correctly. So for larger payload, SST would first upload the payload to S3, send the S3 file path via websocket. And then download from S3.
The S3 bucket is created as part of the debug stack.
When you run
sst start
, you can see the bucket arn printed out in the console. Soemthing like this:
Copy code
=======================
 Deploying debug stack
=======================

Deploying stacks

 ✅  dev-playground-debug-stack (no changes)


Stack dev-playground-debug-stack
  Status: no changes
  Outputs:
    BucketArn: arn:aws:s3:::dev-playground-debug-stack-bucket83908e77-1c7me5gq0zocj
    BucketName: dev-playground-debug-stack-bucket83908e77-1c7me5gq0zocj
    Endpoint: <wss://bdw2j75j26.execute-api.us-east-1.amazonaws.com/dev>
SST just needs s3 permissions to this bucket.
e

Erik Robertson

12/21/2021, 11:04 PM
Thanks for the explanation @Frank. That bucket exists alright. How come SST doesn't set the proper permissions when it creates it ? What should I change ?
The role already has full access to it
I just realized it can't be the role since it's not cloud formation acting here anymore. I gave the user access to that bucket and all looks good now.
f

Frank

12/22/2021, 4:51 AM
Ah nice! Do you have a list of permissions you granted the user? I’d love to add that to the doc.
e

Erik Robertson

12/22/2021, 10:11 AM
Here goes : I've put {} placeholders for what needs to be customized (and removed stuff specific to my stack such as SecretManager access). Glad to give a little back to the SST team if it helps ;-)
Copy code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudformation:DescribeStacks",
        "cloudformation:GetTemplate",
        "cloudformation:DeleteChangeSet",
        "cloudformation:CreateChangeSet",
        "cloudformation:ExecuteChangeSet",
        "cloudformation:DescribeChangeSet",
        "cloudformation:DescribeStackEvents",
        "cloudformation:DeleteStack"
      ],
      "Resource": [
        "arn:aws:cloudformation:us-west-2:{AWS_ACCOUNT}:stack/CDK*",
        "arn:aws:cloudformation:us-west-2:{AWS_ACCOUNT}:stack/{STAGE-PREFIX}-*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:PassRole"
      ],
      "Resource": "arn:aws:iam::{AWS_ACCOUNT}:role/{CLOUDFORMATION-ROLE}"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:*Object",
        "s3:ListBucket",
        "s3:getBucketLocation"
      ],
      "Resource": "arn:aws:s3:::cdktoolkit-stagingbucket-*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:*Object"
      ],
      "Resource": "arn:aws:s3:::{LOWERCASE-STAGE-PREFIX}-*"
    }
  ]
}
f

Frank

12/22/2021, 10:19 AM
This is great! Will add this to the doc!
Thanks Erik!
e

Erik Robertson

12/22/2021, 12:14 PM
@Frank when I deployed to another env I realized the cdk bucket name is not consistent so you will probably have to replace "cdktoolkit-stagingbucket-" with just "cdk"
45 Views