https://serverless-stack.com/ logo
Join Slack
Powered by
I'm in the process of automating the deployment of...
# sst
s
I'm in the process of automating the deployment of my python SST app using CircleCI and am coming up against an issue around how SST is managing python packages within docker during the build process.
During the 
sst deploy
step, I'm getting file permission errors within the Docker container that is installing the dependencies for my python environment (the Dockerfile in question)
looks like it gets all the way to the last step in the build before failing due to file permission issues:
Copy code
Step 11/11 : CMD [ "python" ]
 ---> Running in 24550ed43f69
Removing intermediate container 24550ed43f69
 ---> 989fd7f9d339
Successfully built 989fd7f9d339
Successfully tagged cdk-6871cfca97ab8f0b1447c6d5d1503633f66d7d9f6fd7ac4053a87a2705ec5c06:latest
Bundling asset stage-kas-api-stack/pingQueue/Consumer/Code/Stage...
rsync: recv_generator: mkdir "/asset-output/./__pycache__" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./bin" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./boto3-1.19.5.dist-info" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./boto3" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./botocore-1.22.5.dist-info" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./botocore" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: mkstemp "/asset-output/./.poetry.lock.0NTBUX" failed: Permission denied (13)
rsync: mkstemp "/asset-output/./.pyproject.toml.8O3QZw" failed: Permission denied (13)
rsync: mkstemp "/asset-output/./.requirements.txt.2LN945" failed: Permission denied (13)
rsync: mkstemp "/asset-output/./.six.py.Wo9saF" failed: Permission denied (13)
rsync: recv_generator: mkdir "/asset-output/./certifi-2021.10.8.dist-info" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./certifi" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./charset_normalizer-2.0.7.dist-info" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./charset_normalizer" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./dateutil" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./idna-3.3.dist-info" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./idna" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./jmespath-0.10.0.dist-info" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./jmespath" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./psycopg2" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./psycopg2_binary-2.9.1.dist-info" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./psycopg2_binary.libs" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./python_dateutil-2.8.2.dist-info" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./pytz-2018.7.dist-info" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./pytz" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./requests-2.26.0.dist-info" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./requests" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./s3transfer-0.5.0.dist-info" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./s3transfer" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./six-1.16.0.dist-info" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./urllib3-1.26.7.dist-info" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/asset-output/./urllib3" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]

Error: Failed to bundle asset stage-kas-api-stack/pingQueue/Consumer/Code/Stage, bundle output is located at /home/circleci/project/.build/cdk.out/asset.e6b98195dcdf885b13e5b23d0a38a55551fad0ed45f31727b3ba2cf1f2f60a81-error: Error: docker exited with status 23
    at AssetStaging.bundle (/home/circleci/project/node_modules/@aws-cdk/core/lib/asset-staging.ts:398:13)
    at AssetStaging.stageByBundling (/home/circleci/project/node_modules/@aws-cdk/core/lib/asset-staging.ts:246:10)
    at stageThisAsset (/home/circleci/project/node_modules/@aws-cdk/core/lib/asset-staging.ts:137:35)
    at Cache.obtain (/home/circleci/project/node_modules/@aws-cdk/core/lib/private/cache.ts:24:13)
    at new AssetStaging (/home/circleci/project/node_modules/@aws-cdk/core/lib/asset-staging.ts:162:44)
    at new Asset (/home/circleci/project/node_modules/@aws-cdk/aws-s3-assets/lib/asset.ts:68:21)
    at AssetCode.bind (/home/circleci/project/node_modules/@aws-cdk/aws-lambda/lib/code.ts:183:20)
    at new Function (/home/circleci/project/node_modules/@aws-cdk/aws-lambda/lib/function.ts:335:29)
    at new Function (/home/circleci/project/node_modules/@serverless-stack/resources/src/Function.ts:338:7)
    at Function.fromDefinition (/home/circleci/project/node_modules/@serverless-stack/resources/src/Function.ts:479:14)

There was an error synthesizing your app.
error Command failed with exit code 1.
info Visit <https://yarnpkg.com/en/docs/cli/run> for documentation about this command.

Exited with code exit status 1
CircleCI received exit code 1
my CircleCI config.yml is fairly basic
Copy code
version: '2.1'
orbs:
  aws-cli: circleci/aws-cli@2.0
jobs:
  aws-cli-example:
    executor: aws-cli/default
    steps:
      - checkout
      - aws-cli/setup:
        profile-name: CC-Non-Prod.CC_Engineering_ReadWrite
      - setup_remote_docker:
          version: 19.03.13
      - run: yarn install
      - run: cd src && poetry install
      - run: cd src && poetry run yarn run deploy --stage stage
workflows:
  aws-cli:
    jobs:
      - aws-cli-example:
          context: aws
I'm pretty new to both Python and Docker, so this Docker-In-Docker setup is making my 🤯. I'm not even sure what to google at this point, but it looks like the Docker container that is packaging the lambdas is trying to write a file on the host (CircleCI) and doesn't have the permission to do so.
After a day of effort, I'm waving the white flag on this one for now. Would love to see an example of a working deploy of a python SST app in any CI tool at this point (Github Actions, Circle CI, Jenkins, etc). The packaging of the python dependencies inside docker is tripping me up and it's difficult to find any examples of this anywhere.
s
I was packaging my python dependencies for one website into docker, maybe that helps Dockerfile
s
I have deployed python lambdas with SST using github actions. This is minimal setup. Docker is available already and I don't believe you have to run 
poetry install
explicitly, maybe why the file permission issue. SST will detect dep manager and run it in the docker according to the doc here: https://docs.serverless-stack.com/constructs/Function#bundle. For mine, I used Pipenv. Let me know if you have more questions.
Copy code
name: Deploy app

on:
  push:
    branches:
      - master

jobs:
  deploy:
    name: Deploy app
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [14.x]
    steps:
      - name: Check out source repository
        uses: actions/checkout@v2
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v2
        with:
          node-version: ${{ matrix.node-version }}
      - name: Install dependencies
        run: yarn install
      - name: Configure production AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.PROD_AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.PROD_AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.PROD_AWS_REGION }}
      - name: Deploy app
        run: yarn deploy --stage production
s
@Sione This worked for me, I can't thank you enough!!! This community saves my butt once again. You all are amazing!
s
@Seth Geoghegan yaay. I'm glad it worked. Agreed, this community is amazing.
Open in Slack
PreviousNext
Powered by