Fun question for the evening: If sst creates zip files for Lambdas and pushes them up to AWS, then why can I see Docker images being built in the terminal output during the deployment?

when building Python functions, SST will build the dependencies in

requirements.txt

inside a docker container, and then zip them up and pushes to AWS.

I'm not using any OS-specific dependencies that I'm aware of (perhaps feeling a bit ignorant - example of OS-specific dependency?) What I am working on though, is a lambda deployment spikes project, where I'm learning all the ways of deploying python lambdas. My first few are "raw" lambdas, no serverless framework of any kind, and then I want to play with the custom

installCommands

on this guy (sounds like a lot of fun):

export interface FunctionBundlePythonProps {
    readonly installCommands?: string[];
}

The motivation is as follows: 1. The design pattern style documentation on serverless apps is pretty scant (best way to separate out non-lambda-related code, how to unit test it, etc), so I'm trying to come up with something clean and composable. 2. I like to make my dynamodb CRUD code in it's own module so that it can be unit tested. Or I'll have a nice domain object like, like my Rotation class that does well with heavy unit test coverage 3. The issue is that I don't like my test files to hang out in the same directory as application code, so then I need to do a

setup.py

thing something like this, so that my unit tests can see the app code they're testing (integration tests aren't a problem, since they use

requests

to talk to the REST API.)

── karaoke
│   ├── performance_history
│   ├── rotations
│   ├── singers
│   └── song_library
├── seed-data
│   ├── load_seed_data.py
│   ├── seed-singers.json
│   └── seed-song-library.json
├── setup.py
└── test
    ├── integration
    └── unit

4. That works really well for the unit tests, but then the app code doesn't like it. I either have to deal with vs code squawking about not being able to find something like

import rotation

, but then lambda itself will error if I do something like

from karaoke.rotations.rotation import Rotation

because it can't find the package

karaoke

. 5. But wait a second - Lambdas are never unit tested, so why would I want them in a

setup.py

package situation anyway?? 6. So, what I want to do, is put all the unit-testable code like domain classes and dynamodb access functions, in a Python package, maybe a

.whl

file, and then install that into a docker image. (I think docker images is better than zip files for this, from my current understanding, since you can just do a

pip install

on the

.whl

)

numpy

is an OS-specific package I’ve come across way too often in all the hello world tutorials out there 🤣

I hit enter too soon, haha. But the above is what I've been thinking about the last few days.

Yeah there is a Dockerfile behind the scene https://github.com/serverless-stack/serverless-stack/blob/master/packages/resources/assets/python/Dockerfile.dependencies

Hey @Frank - okay, I've done some experimenting outside of sst (for now at least, will probably want to play with

installCommands

soon). But I think this shows what I'm trying to ultimately accomplish in the context of a serverless framework: 1. The git repo/folder is https://github.com/lukewyman/lambda-deploy-spikes/tree/master/raw-lambda-whl-package-image if you wanna take a look. I'll fix up the READMEs and make the documentation pretty tomorrow, but the thing is done and works. 2. Looking at the slimmed file tree:

.
├── dist
│             ├── rest_helper-0.0.1-py3-none-any.whl
│             └── rest-helper-0.0.1.tar.gz
├── Dockerfile
├── ecr_repository_policy.json
├── functions
│             ├── getter.py
│             └── poster.py
├── LICENSE
├── packaged
│             ├── rest_helper
│             │             ├── __init__.py
│             │             └── rest_helper.py
├── pyproject.toml
├── setup.cfg
└── test
    └── test_rest_helper.py

everything inside

packaged

is code that the Lambdas use.

pyproject.toml

,

setup.cfg

, etc is to create the Python package 3. Running

pip install -e .

in

src

installs an editable dependency in my virtual environment with a sim link so that dependent python code in my tests and lambdas always has the latest in real time. 5. Running

python3 -m build

in

src

creates the wheel file in

dist

6. The Dockerfile contains commands to copy the

.whl

from

dist

into the image and installs it (this avoids uploading to pypi and allows it to be part of the flow). I tried parameterizing the Dockerfile with `ARG`s, but that flopped, so I need to tinker with that some more to make it extensible for any Lambda. 7. This accomplishes (where I was a bit stumped in my karaoke project): • the tests in

test

can access the package locally (with the sim link) • the lambda import statements work in vs code with dot notation as if using a pip installed package (with the sim link) • the lambdas themselves have access to the dependencies when running in their cloud environment (with the pip installed

.whl

file in the container). 8. I did this the image way because I like how all the commands make sense from the 30,000 ft view. I'm sure the zip file approach would also work - it would just take unpacking the

.whl

into the

.zip

before deploying.

DO NOT, I reapeat - DO NOT try to deal with python without docker - trying to make it work on windows / linux various distros / macos consistently - is an absolute nightmare 😄