<@U01J5Q8HV5Z> How is the way to go about using Py...
# sst
d
@Jay How is the way to go about using Python with SST? Is that possible? Or is SST more for JS?
f
Hi @Dillon Peterson Yeah we’ve been getting a lot of requests for Go and Python. The work will start next week.. I don’t have a hard date at the moment.
j
Yup, we are going to start working on it soon.
d
Hi @Frank and @Jay, thank y'all: we'll be following this! Also, we are VERY happy Seed customers 😊🤠.
j
Oh that's super cool to hear!
j
hey guys! Do we have any updates on Python support for SST? I’m just beginning a new project that unfortunately needs to use python because of a data science requirement. I’d also love to contribute if there’s any way to do that! I really like the experience of developing using SST using JS/TS 😄
j
Yeah we are going to start work on it this week. Now that we added Go support, it should be a bit more straightforward to add other languages. @Frank might have some questions for you if that's okay?
d
Very good @Jay, we're excited to hear that!! Would be happy to answer any questions you have for me as well. We do a lot of data science projects. 🤠
f
@José Ribeiro @Dillon Peterson hey guys! I’m planning to get started on this tmr. There are two parts to this: 1. supporting Python runtime (so u can write Lambda functions in Python); and 2. supporting writing CDK code in Python Does it make sense if we made #1 possible first? So you still write your CDK code in JS/TS, and Lambda code in Python.
Or does it make sense to have both 1 and 2?
d
Hi @Frank, Option 1 is what we need. Option 2 is a nice-to-have. Would much rather Option 1 be implemented first, as the Python lambdas are what we require.
j
that’s great news! Thanks for the updates guys. Regarding your question Frank: I think it does make sense to ship #1 first. Quite honestly, as soon as we have #1 I’m completely ok with writing CDK code in JS/TS 🙂
d
Same as @José Ribeiro 👍👏
f
Sounds good! I will keep y’all posted on the progress 💪
Btw, what packager r u guys using for Python? ie. pipenv vs pip with requirements.txt? And what are you using to deploy ur Python Lambda functions right now?
j
I’m using
poetry
, but
requirements.txt
works good as well as it’s easy to export from poetry
In fact that’s what I think
serverless-python-requirements
does
f
I see. Yeah I came across
poetry
when looking into packagers. Is that what most folks use?
I’m think if we need to support all of
poetry
pipenv
and
pip
, or if there’s something most ppl use, maybe I can start with that.
j
@Frank from a product point of view I think I’d add support for pip/requirements.txt at first and leave support for the actual package managers to the future. Everyone can translate to pip/requirements, so that’s a common API
f
Yeah that makes sense. I think that’s what
serverless-python-requirements
does, it generates the
requirements.txt
file from
pyproject.toml
or
Pipefile
. Install all dependencies locally in a directory, and zips up the directory with the Lambda code all together.
j
just to illustrate, in terms of usage, to me support for poetry/pipenv would rank lower than building in docker. I’m very worried about the build environment not getting the functions broken due to cross-os lib implementations
f
oh I see.. let me put together a v1 asap without docker build, and if some libs break for u, i’ll put that in right after.
j
sorry, I was in a call but can elaborate a bit more there. We’re using opencv/numpy for a lot of our data science applications, and back when I was actively developing these applications they seemed to be particularly sensitive to different platforms, to the point that we just started building everything in docker. Dealing with postgres and mssql missing/disconnected drivers has been a bit of pain in the past as well using python (using psycopg2 and pyodbc respectively), but I don’t have a sense whether that’s got any better now. I’m open to trying it out again if that helps!
f
Got it! Yeah let me put something together so you can start playing around with it.
j
ok @Frank! Let me know if there’s anything I can help with
f
hey @José Ribeiro is there a way to check for syntax errors in a python handler? I tried the
python -m py_compile handler.py
command, but also saw linters like flake8 and pylint.
I’m working on the python watcher for
sst start
, so when you save a file, the console will complaint any errors in the code. For Node, on file save,
sst start
transpiles the code, lints it, and type checks it if it’s a Typescript file. Transpile and type check are likely irrelevant for Python, I’m trying to figure out the linting part.
I’d appreciate some pointers.
j
@Frank if you merely want to check for syntax errors I think
py_compile
works well. I’m used to using flake8, but that’s more on the style enforcement/static code analysis side of things (also, it’s not shipped with the default python install). I feel like adding flake8/pylint would be adding some opinionatedness to serverless-stack without really adding any value. Does that make sense?
I’m guessing we could only check for syntax errors and add optional linting through an api where you could run your custom command. Would that make sense?
hey @Frank, is there anything in particular you want me to try out or help develop? I’m quite interested to start playing with python there 😄
f
Hey @José Ribeiro, I just cut a new release with Python runtime support. To create a new sample app, run
Copy code
npx create-serverless-stack@latest my-sst-app --language python
Point the
srcPath
for
sst.Function
to the directory containing the
poetry.lock
. And on build/deploy, SST will generate a
requirements.txt
and install the modules inside a docker container.
Here are some details on what
srcPath
and
handle
takes for Python functions - https://docs.serverless-stack.com/constructs/Function#functionprops
I will get @Jay to add a couple more Python examples in the coming days.
Give it a try🎉
j
nice, that sounds awesome! will try it out today
f
Hey @José Ribeiro, just curious if u had a chance to give Python runtime a try.
j
hey Frank! In fact I did — I created a small app to try it out with an Api and then was trying to transform one app we have that uses S3/SQS triggers. Do you know whether sqs triggers will also work for an offline function, or do we pretty much support only functions that have an api gateway endpoint attached to them for local dev?
f
Are the S3/SQS triggers created through
sst.Bucket
and
sst.Queue
constructs?
As long as the Lambda functions are created using
sst.Function
(triggers created by
sst.Api
,
sst.Bucket
,
sst.Queue
, etc are all using
sst.Function
) they will be invoked locally when running
sst start
Are you seeing otherwise?
j
so I’m using the following setting, but i might as well be wrong in how I’m handling that. Should I use
addEventSource
instead?
Copy code
new Function(this, "Processor", {
      handler: "handler.process",
      srcPath: "src",
    });

    new Queue(this, "ProcessorQueue", {
      handler: "handler.process",
      srcPath: "src",
      consumerProps: {
        batchSize: 1,
      },
    });
also — is there a way to use existing s3 buckets as a source right now?
f
Change the
Queue
definition to this:
Copy code
new Queue(this, "ProcessorQueue", {
  consumer: {
    function: {
      handler: "handler.process",
      srcPath: "src",
    },
    consumerProps: {
      batchSize: 1,
    },
  },
});
As for using existing S3 buckets, CDK is adding support for this. I opened an issue here to track the progress. https://github.com/serverless-stack/serverless-stack/issues/338
j
awesome, that worked great!