https://serverless-stack.com/ logo
Join SlackCommunities
Powered by
I'm getting an error when doing a `npx sst deploy ...
# sst
l
I'm getting an error when doing a 
npx sst deploy --stage test
of my monorepo app. CDK is in Typescript, Lambdas are in Python. Things had been pretty smooth with deploying/removing through this morning, and I haven't changed the CDK code in a couple days. Here's the error:
Copy code
test-karaoke-auth | CREATE_FAILED | AWS::Lambda::Function | AuthpostConfirmation9638BE34 Resource handler returned message: "Could not unzip uploaded file. Please check your file, then try to upload again. (Service: Lambda, Status Code: 400, Request ID: 62093ef6-5fc4-4c34-9fc9-41f040129049, Extended Request ID: null)" (RequestToken: 7ff6ef6c-3053-e0bd-a668-dd1d985361bc, HandlerErrorCode: InvalidRequest)
Putting other failed Lambda here to preserve:
Copy code
test-karaoke-singers | CREATE_FAILED | AWS::Lambda::Function | LambdaPUTkaraokesingerssingerId0B097659 Resource handler returned message: "Could not unzip uploaded file. Please check your file, then try to upload again. (Service: Lambda, Status Code: 400, Request ID: 78b004a4-77ed-4336-bc7e-4ce5c5eac1c4, Extended Request ID: null)" (RequestToken: 4529ae3a-b7be-a06a-e162-7a16afd056ee, HandlerErrorCode: InvalidRequest)
I did a complete deploy before my last git push and deploy and integration tests worked great. The only changes I've made this go around, are to add a non-infrastructure-related domain object with isolated unit tests. It might be not related to sst, since I seem to also have a Python issue where modules in the same folder are not being recognized on imports, but that is happening throughout the project, and only a couple of Lambdas are being rejected on deploy.
f
Hey @Luke Wyman, hmm.. that’s weird error.
l
It's incredibly weird. I'm still testing some things here to see what's me and what's sst.
There are two lambdas that are bombing. Also, I know you're more node than python, but my 
pip install -e .
was working great last night and this morning, and all of a sudden, my .py files can't find modules in that python environment.
So, I switched envs so that my unit tests don't need to find my application code. Just trying to get the deploy to work, and those two lambdas still bomb.
f
hmm.. when you run 
sst deploy
and gets this error, try this: 1. go into 
.build/cdk.out
2. open up the CF template for 
test-karaoke-auth
3. look up the asset file for the Post Confirmation lambda 4. find that asset zip 5. try manually unzip
Let’s see if the zips are fine
l
k, checking...
Okay, this is interesting/confusing - when I watch the build in the terminal I see docker images being built. When I looked in the CF template, I see `"ZipFile": "placeholder"`:
Copy code
"AuthpostConfirmation9638BE34": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Code": {
          "ZipFile": "placeholder"
        },
f
Oh.. lemme check
we do the “placeholder” trick on 
sst remove
so your Lambdas are not getting built on remove. And this is happening from running 
sst deploy
?
l
oh, no. I did a remove after. Let me redeploy and check...
f
yup yup
l
Okay, so now that I've done the deploy (and it failed as "expected"), searching 
test-karaoke-auth.template.json
for the term "ZipFile" finds nothing.
Copy code
"AuthpostConfirmation9638BE34": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Code": {
          "S3Bucket": {
            "Ref": "AssetParametersac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91eS3Bucket0FBA01FB"
          },
          "S3Key": {
            "Fn::Join": [
              "",
              [
                {
                  "Fn::Select": [
                    0,
                    {
                      "Fn::Split": [
                        "||",
                        {
                          "Ref": "AssetParametersac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91eS3VersionKey771BDE70"
                        }
                      ]
                    }
                  ]
                },
                {
                  "Fn::Select": [
                    1,
                    {
                      "Fn::Split": [
                        "||",
                        {
                          "Ref": "AssetParametersac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91eS3VersionKey771BDE70"
                        }
                      ]
                    }
                  ]
                }
              ]
            ]
          }
        },
that's the entry for the Lambda
f
Can you search for 
AssetParametersac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91eS3Bucket0FBA01FB
in ur CFN template?
l
Copy code
"AssetParametersac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91eS3Bucket0FBA01FB": {
      "Type": "String",
      "Description": "S3 bucket for asset \"ac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91e\""
    }
f
Can you see a folder or a zip file named 
ac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91e
in ur 
.build/cdk.out
?
l
No. There are the 
.template.json
files in cdk.out, and then 3 sub folders prefixed with 
asset.[some long string]
which contain copies of my 
.py
files.
f
Is there an asset.ac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91e?
l
there is. It has all the lambdas for the singers microservice. The two lambdas that are bombing are both in there.
The reason that one of them is Auth, is because Auth uses it as a post confirmation trigger to create a new singer in the singers dynamodb table using the 
UserAttributes
from Cognito.
The rest are used by the API Gateway endpoint for put, get, etc.
(will be back in about 30 min)
back. Can also check in tomorrow if you'd like more info or for me to try other things.
f
Can you check how many other functions in the CFN template are also pointing to the 
AssetParametersac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91eS3Bucket0FBA01FB
S3 asset?
I’m curious if there are other functions that uses the same asset and was able to deploy.
l
Hey @Frank - I did some testing and looking around and here's what I got. First, to your question: 
AuthStack
and 
SingersStack
are the two stacks that fail to deploy due to the zip error complaint. They both point to the same folder for their Lambdas. I found the S3 asset, 
AssetParametersac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91eS3Bucket0FBA01FB
, in both stacks - once in Auth for the function it takes, and five times in Singers for the five functions it uses. I checked the other stacks to be sure, and those don't use that S3 asset. Second, I did a little experiment, where I commented out the API routes in the Singers stack one by one:
Copy code
singersApi.addRoutes(this, {
      'POST   /karaoke/singers': 'KAR_SNG_test_create_singer.handler',
      'GET    /karaoke/singers/{singerId}': 'KAR_SNG_get_singer_by_id.handler',
      'GET    /karaoke/singers': 'KAR_SNG_get_singers.handler',
      'PUT    /karaoke/singers/{singerId}': 'KAR_SNG_update_singer.handler',
      'DELETE /karaoke/singers/{singerId}': 'KAR_SNG_delete_singer.handler',
    });
...and then tried the deploy again after each comment-out. Each Lambda failed, one after the other:
Copy code
AuthpostConfirmation9638BE34
LambdaPOSTkaraokesingers87AC2C24
LambdaGETkaraokesingerssingerId784287B5
LambdaGETkaraokesingersFED899AE
LambdaPUTkaraokesingerssingerId0B097659
LambdaDELETEkaraokesingerssingerIdF1924E8A
To summarize - that S3 asset only appears in the Auth and Singers stacks. They all point to the same folder, 
src/karaoke/singers
in my application code. So, every Lambda in that folder is failing.
f
@Luke Wyman That makes this a bit easier to debug. At least the failure behavior is consistent.
Let’s try this: • go to CloudFormation console and look for the Auth stack; • go to the Parameters tab (it should look something like the attached screenshot); • note the parameter value that starts with 
cdk-
, that’s the name of the S3 bucket; • also note the parameter value for 
AssetParametersac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91eS3Bucket0FBA01FB
, that’s the S3 path to the Lambda zip • now go to the S3 console and locate the file • download it • try unzip it Let’s see if u can unzip the zip file; and what’s in it.
l
Okay, @Frank - seems I might have some useful data. To clarify, I believe 
AssetParametersac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91eS3Bucket0FBA01FB
is the Key to the S3 bucket, and 
AssetParametersac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91eS3VersionKey771BDE70
is the Key to the zip file, which is named 
assets/||ac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91e.zip
. I downloaded the zip file and tried to unzip it locally with 
unzip ac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91e.zip
and got the following error message:
Copy code
$ unzip ac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91e.zip 
Archive:  ac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91e.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of ac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91e.zip or
        ac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91e.zip.zip, and cannot find ac4e6005b437cb3ac058af34adb98d4147d60091b897ce896f12457b6a7cc91e.zip.ZIP, period.
In other news, I was doing a little re-org in my project that I had meant to do. I renamed some app code folders with underscores instead of hyphens to pythonize, and I moved the microservice READMEs out of the code folders into a 
site
folder outside of 
src
. Did a full deploy as a lark, and now everything works just fine. Thanks for your help on this all the same. If there's still any information I can throw your way because this bug is important to sst, let me know.
f
Oh nice! Glad u got it to work.
Is ur repo public? (I remember it was in the beginning lol).. If it is, I want to check out the commit that had the issue and give it a try on my machine.
l
Yeah, my repo is public and you're welcome to try your luck with it. It would be on the 
rotations_impl
branch, and I believe it was commit 
67a7de9
"Pythonized app folder names with underscores" that unintentionally resolved the issue, maaaaaybe commit 
1da2ce2
right before it. A word of caution: there are issues around my Python packaging that may make it a bit of hit and miss to isolate the problem. I've been trying to create a package with a 
setup.py
, 
___init___.py
etc so that my unit tests can "see" my domain objects they're testing. This has resulted in problems where Lambda can't see a package called 
karaoke.song_library.songs_db
, for example. So I'm deep in the weeds figuring out how to get unit tests, packaging and deployment to all play nicely together. And here's the repo: https://github.com/lukewyman/karaoke-backend-sst
s
Having this exact error message myself this morning. Updated a small file in my python app and re-deployed only to get
stage-kas-KafkaIngest failed: Resource handler returned message: "Could not unzip uploaded file. Please check your file, then try to upload again. (Service: Lambda, Status Code: 400, Request ID: ac5a9bd2-16f3-4af3-ae44-381b1096f583, Extended Request ID: null)" (RequestToken: dbf205dc-7fe2-2ad4-ae90-21e999d258fe, HandlerErrorCode: InvalidRequest)
I updated a file that includes string constants, which each of my Lambdas uses. Now I'm getting that error when deploying any of the lambdas. 🤔
Welp, I got past the error by changing one of my lambdas (added a print statement) and re-deployed. Did the trick 🤷
Open in Slack
PreviousNext
Powered by