Hi, has anyone been able to publish an API/Lambda that depends on a private Python module? My module is published in AWS CodeArtifact, but during the Docker-based build process, it’s looking in public module repositories and fails

#13 [9/9] RUN if [ -f 'requirements.txt' ]; then pip install -r requirements.txt -t .; else echo "requirements.txt not found"; fi
#13 0.723 ERROR: Could not find a version that satisfies the requirement myprivatemodule (from versions: none)
#13 0.723 ERROR: No matching distribution found for myprivatemodule
#13 ERROR: executor failed running [/bin/sh -c if [ -f 'requirements.txt' ]; then pip install -r requirements.txt -t .; else echo "requirements.txt not found"; fi]: exit code: 1
------
 > [9/9] RUN if [ -f 'requirements.txt' ]; then pip install -r requirements.txt -t .; else echo "requirements.txt not found"; fi:
------
executor failed running [/bin/sh -c if [ -f 'requirements.txt' ]; then pip install -r requirements.txt -t .; else echo "requirements.txt not found"; fi]: exit code: 1

Error: docker exited with status 1
    at dockerExec (/src/myapp/node_modules/@aws-cdk/core/lib/bundling.ts:450:11)
    at Function.fromBuild (/src/myapp/node_modules/@aws-cdk/core/lib/bundling.ts:266:5)
    at Function.fromAsset (/src/myapp/node_modules/@aws-cdk/core/lib/bundling.ts:156:24)
    at Object.bundle (/src/myapp/node_modules/@serverless-stack/resources/src/util/python/bundling.ts:107:41)
    at Object.builder (/src/myapp/node_modules/@serverless-stack/resources/src/util/pythonBuilder.ts:42:15)
    at new Function (/src/myapp/node_modules/@serverless-stack/resources/src/Function.ts:245:21)
    at Function.fromDefinition (/src/myapp/node_modules/@serverless-stack/resources/src/Function.ts:326:14)
    at Api.addRoute (/src/myapp/node_modules/@serverless-stack/resources/src/Api.ts:339:23)
    at /src/myapp/node_modules/@serverless-stack/resources/src/Api.ts:198:14
    at Array.forEach (<anonymous>)

There was an error synthesizing your app.

FWIW, when building my own docker containers I use the following script with environment AWS_* credentials to pull dependencies defined in

requirements.txt

from both AWS CodeArtifact and PyPi.org

#!/bin/sh
CODEARTIFACT_AUTH_TOKEN=`/usr/local/bin/aws codeartifact get-authorization-token --domain mydomain --domain-owner 1111111111 --query authorizationToken --output text`
pip3 install -v d  \
  --index-url "<https://aws>:$CODEARTIFACT_AUTH_TOKEN@mydomain-1111111111.d.codeartifact.us-east-1.amazonaws.com/pypi/myprivatemodule/simple/" \
  --extra-index-url "<https://pypi.org/simple>" "myprivatemodule>=0.0.16"

One workaround would be to publish a lambda Layer of my private module, but I’m not sure that solution would work for local debugging.

Hey @Guy Shechter, thanks for the details. It should pretty straightforward to support for private modules.

The latter also installs the non-private modules because it looks in the --extra-index-url for those. so you don’t need both (in fact if you have both, then the first will fail on the private repos in the requirements.txt)

I see.. 2 more questions: 1. what does

-v d

do? 2. why is “myprivatemodule>=0.0.16” explicitly specified in the cli?

The -v is for debugging; i think the d is a typo. More generally speaking, is it possible for the developer to provide a dockerfile that could be used in the build process to pull in whatever is needed?

Ah that makes sense.

In which case you would need to keep the requirements.txt for the public modules

Yeah, that’d offer full customization. But for this use case, if

sst.Function

lets you pass in a build command and a docker environment, ie:

new Function(this, "MyLambda", {
  runtime: "python3.7",
  handler: "src/index.main",
  bundle: {
    command: "pip3 install -v --index-url ...",
    environment: {
      CODEARTIFACT_AUTH_TOKEN: process.env.CODEARTIFACT_AUTH_TOKEN,
    }
  },
});

You can run the AWS CLI to get the TOKEN and pass that in to the

sst deploy

command. ie:

CODEARTIFACT_AUTH_TOKEN=`/usr/local/bin/aws codeartifact get-authorization-token ...` sst deploy

That could work. Could that also be configured to work through Seed ?

Yeah, that will work on Seed. I just opened an issue to track the progress https://github.com/serverless-stack/serverless-stack/issues/506

Thanks @Frank will do!