I have a very weird error. I had a functional stack. I decided to reduce the AWS permissions on the IAM user that deploys/drops SST stacks and have actually switched to using a service role for CloudFront using the --role-arn flag. That involved a lot of steps including re-bootstrapping CDK and lot of trial and error to get the policies right. Now it deploys and works again EXCEPT for one specific API call that with certain parameters and not others generates a runtime error in the live dev mode. I've stepped thought it and the error occurs after "my" lambda code ends and returns the result. The error seen on the server is a 403 access denied on S3

a625b40c-04d9-4cbe-90b7-8324e904055a REQUEST DEV-ERIK-dwam-back-my-sta-ApiLambdaGETassets0D68A0-FzosTE9J9tP7 [src/routes/assets_get.main] invoked by API GET /assets
a625b40c-04d9-4cbe-90b7-8324e904055a RESPONSE {"headers":{"Content-Type":"application/json"},"statusCode":200,"body":"{\n  \"csvHeader\": \"latitude,longitude\",\n  \"csvData\": \"37.65728378,-122.4156265\\r\\n37.65734863,-122.41... 177051 more characters"}
Failed to upload payload to S3. AccessDenied: Access Denied
    at Request.extractError (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/services/s3.js:714:35)
    at Request.callListeners (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/Users/erik/src/dataworks/dwam-back/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'AccessDenied',
  region: null,
  time: 2021-12-21T21:20:09.867Z,
  requestId: 'R0H1MF37YG9JSGTQ',
  extendedRequestId: 'SXswySS3eGF3DkYb/YjNO8YIw2T911TZViSQZbivYh6VHH71nfQP33L1lTVTA6oixDM5oFvwEuc=',
  cfId: undefined,
  statusCode: 403,
  retryable: false,
  retryDelay: 43.50154802428319
}

and when I check the Cloudwatch logs I see this 404 on S3 :

2021-12-21T22:20:05.498+01:00	START RequestId: a625b40c-04d9-4cbe-90b7-8324e904055a Version: $LATEST

2021-12-21T22:20:05.517+01:00	2021-12-21T21:20:05.516Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO connectAndSendMessage()

2021-12-21T22:20:05.641+01:00	2021-12-21T21:20:05.641Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO ws.onopen

2021-12-21T22:20:05.641+01:00	2021-12-21T21:20:05.641Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO sendMessage() - send request

2021-12-21T22:20:05.643+01:00	2021-12-21T21:20:05.642Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO sendMessage() - sending request via WebSocket

2021-12-21T22:20:05.644+01:00	2021-12-21T21:20:05.643Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO sendMessage() - start keep alive timer

2021-12-21T22:20:10.189+01:00	2021-12-21T21:20:10.188Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO ws.onmessage {"action":"client.lambdaResponse","debugRequestId":"a625b40c-04d9-4cbe-90b7-8324e904055a-1640121605516","stubConnectionId":"Kt-45fnPPHcCGBQ=","payloadS3Key":"payloads/a625b40c-04d9-4cbe-90b7-8324e904055a-1640121605516-response"}

2021-12-21T22:20:10.189+01:00	2021-12-21T21:20:10.189Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO receiveMessage()

2021-12-21T22:20:10.189+01:00	2021-12-21T21:20:10.189Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO receiveMessage() - received payloadS3Key

2021-12-21T22:20:10.256+01:00	2021-12-21T21:20:10.256Z a625b40c-04d9-4cbe-90b7-8324e904055a INFO [AWS s3 404 0.065s 0 retries] getObject({ Bucket: 'dev-erik-dwam-back-debug-stack-bucket83908e77-1awrh01i15tnq', Key: 'payloads/a625b40c-04d9-4cbe-90b7-8324e904055a-1640121605516-response' })

2021-12-21T22:20:10.259+01:00	2021-12-21T21:20:10.259Z a625b40c-04d9-4cbe-90b7-8324e904055a ERROR Unhandled Promise Rejection {"errorType":"Runtime.UnhandledPromiseRejection","errorMessage":"NoSuchKey: The specified key does not exist.","reason":{"errorType":"NoSuchKey","errorMessage":"The specified key does not exist.","code":"NoSuchKey","message":"The specified key does not exist.","region":null,"time":"2021-12-21T21:20:10.253Z","requestId":"0G7QZ64M4EVDDPV5","extendedRequestId":"Zk5DMRznR+DMrIMUD0jYqUlGn2aejxk6E+Gl0WupDfZCynNyUKOzjWUJPK10Pseeu80tzGB6t4o=","statusCode":404,"retryable":false,"retryDelay":8.646154170322662,"stack":["NoSuchKey: The specified key does not exist."," at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/services/s3.js:699:35)"," at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)"," at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)"," at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:688:14)"," at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)"," at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)"," at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10"," at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)"," at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:690:12)"," at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)"]},"promise":{},"stack":["Runtime.UnhandledPromiseRejection: NoSuchKey: The specified key does not exist."," at process.<anonymous> (/var/runtime/index.js:35:15)"," at process.emit (events.js:314:20)"," at process.EventEmitter.emit (domain.js:483:12)"," at processPromiseRejections (internal/process/promises.js:209:33)"," at processTicksAndRejections (internal/process/task_queues.js:98:32)"]}

2021-12-21T22:20:10.275+01:00	END RequestId: a625b40c-04d9-4cbe-90b7-8324e904055a

2021-12-21T22:20:10.275+01:00	REPORT RequestId: a625b40c-04d9-4cbe-90b7-8324e904055a Duration: 4775.65 ms Billed Duration: 4776 ms Memory Size: 1024 MB Max Memory Used: 28 MB XRAY TraceId: 1-61c24505-337b01bf10f9f70f44866cac SegmentId: 247605f67fd8dd26 Sampled: true

2021-12-21T22:20:10.275+01:00	Unknown application error occurred

I'm eager to see if any of you have any idea ?

Hey @Erik Robertson,

sst start

uses a websocket to send/receive data between the real Lambda function and ur local machine. Websocket has certain packet size limit 64KB if i call correctly. So for larger payload, SST would first upload the payload to S3, send the S3 file path via websocket. And then download from S3.

Thanks for the explanation @Frank. That bucket exists alright. How come SST doesn't set the proper permissions when it creates it ? What should I change ?

Ah nice! Do you have a list of permissions you granted the user? I’d love to add that to the doc.

Here goes : I've put {} placeholders for what needs to be customized (and removed stuff specific to my stack such as SecretManager access). Glad to give a little back to the SST team if it helps ;-)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudformation:DescribeStacks",
        "cloudformation:GetTemplate",
        "cloudformation:DeleteChangeSet",
        "cloudformation:CreateChangeSet",
        "cloudformation:ExecuteChangeSet",
        "cloudformation:DescribeChangeSet",
        "cloudformation:DescribeStackEvents",
        "cloudformation:DeleteStack"
      ],
      "Resource": [
        "arn:aws:cloudformation:us-west-2:{AWS_ACCOUNT}:stack/CDK*",
        "arn:aws:cloudformation:us-west-2:{AWS_ACCOUNT}:stack/{STAGE-PREFIX}-*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:PassRole"
      ],
      "Resource": "arn:aws:iam::{AWS_ACCOUNT}:role/{CLOUDFORMATION-ROLE}"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:*Object",
        "s3:ListBucket",
        "s3:getBucketLocation"
      ],
      "Resource": "arn:aws:s3:::cdktoolkit-stagingbucket-*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:*Object"
      ],
      "Resource": "arn:aws:s3:::{LOWERCASE-STAGE-PREFIX}-*"
    }
  ]
}

This is great! Will add this to the doc!

@Frank when I deployed to another env I realized the cdk bucket name is not consistent so you will probably have to replace "cdktoolkit-stagingbucket-" with just "cdk"