https://serverless-stack.com/ logo
#sst
Title
# sst
t

Tyler Flint

04/02/2021, 2:42 PM
ok now I’m running into another issue. It appears that setting
bundle: false
doesn’t actually stop esbuild from building even for deploy.
I have 4 functions:
Copy code
const lambda = {
      graphql: new sst.Function(this, "graphql", {
        handler: 'index.main',
        bundle: false,
        srcPath: 'dist/graphql',
        environment: env,
        memorySize: 256,
        timeout: 30,
      }),
      
      web: new sst.Function(this, "web", {
        handler: 'index.main',
        bundle: false,
        srcPath: 'dist/web',
        environment: env,
        memorySize: 256,
        timeout: 30,
      }),
      
      worker: new sst.Function(this, "worker", {
        handler: 'index.main',
        bundle: false,
        srcPath: 'dist/worker',
        environment: env,
        memorySize: 256,
        timeout: 500,
      }),
      
      migrate: new sst.Function(this, "migrate", {
        handler: 'index.main',
        bundle: false,
        srcPath: 'dist/migrate',
        environment: env,
        memorySize: 512,
        timeout: 500,
      })
    };
and I would assume that
bundle: false
would just zip/upload the content of
srcPath
and inform lambda of the file/handler to call.
however I then noticed that the lambda sizes are much bigger than I would have thought, so I started inspecting the contents of the lambda function and noticed a
.build
directory in addition to my source
which is an esbuild duplicate of my
index.js
but with the sourcemap screwed up.
am I missing something here?
bummer. Alright, that means
bundle: false
isn’t a simple “package up my source and tell lambda to use the handler specified”.
but it’s a esbuild bundle attribute: https://esbuild.github.io/api/#bundle
not at all what I was hoping for, but I guess that’s the system so now to think about how to work within…
hmmm. I suppose it means I will need to feed all of my source through esbuild. So in my case, a legacy project that already has a build pipeline, I’ll either have to do a major rewrite of the build pipeline (definitely no time for that) or I’ll just have to ensure that my current build pipeline doesn’t do any optimizations or minifications that will get screwed up once piped through esbuild.
my lambda functions will be at least 10x bigger, but maybe that’s ok for now.
maybe I can get esbuild to minify
(the rationale behind minification is to keep the lambda zip as small as possible. Source maps have allowed me to minify and still get original stack traces)
ah, looks like the builder would need to be expanded to support a minify attribute: https://github.com/serverless-stack/serverless-stack/blob/master/packages/resources/src/util/builder.ts#L209
which would have to be added in the FunctionBundleProps: https://docs.serverless-stack.com/constructs/Function#functionbundleprops
alright, unfortunately I’m passed my allotted time limit for this experiment right now. Maybe when I come back and have a bit more time I’ll submit a few PRs: 1 - Support minify as a build option 2 - Introduce a raw builder, which doesn’t run through esbuild at all, thus allowing other build pipelines to be used (webpack, rollup, etc).
a quick thought on #2, I don’t think sst should even try to support other loaders. I love that sst is opinionated and is clearly planted in the future with esbuild. It’s a great product and no defense is required for the choice. Rather, I would love to see sst support a raw or non-build option. Legacy build pipelines can generate output which can be zipped up, and for
sst start
could end-run 90% of the functionality and just map a request to a file/handler. A legacy builder like webpack could be run in watch mode and achieve the same result.
For instance, if using
sst start
with an external builder, just run
sst start
and
webpack --watch
concurrently, with the
concurrently
package: https://www.npmjs.com/package/concurrently
like this:
concurrently "webpack --watch" "sst start"
and then sst doesn’t have to support the whole world
which, fwiw is probably how I would think about supporting other languages. In theory as long as “insert language here” knows how to build the source and can be setup to watch -> compile then sst only needs to be a simple forklift for deploys and a simple pipe for development.
you could even simplify the workflow with a
buildCmd
and
devCmd
in the sst.json, and substitute the current esbuild process for those if they exist. With this simple addition you could support anything that lambda currently supports.
because the genius of sst (which it is genius) is 1 - The cdk constructs and building on top of cdk. 2 - The development stack.
the packaging is a mostly solved problem for each language. If sst can provide a first-class experience where possible, but then an escape hatch for everything else… wow.
f

Frank

04/02/2021, 4:47 PM
Hey @Tyler Flint Thanks for looking into and appreciate the thoughts! Yeah, the esbuild’s bundle flag is always enabled currently. Can you share a snippet of how you are building ur functions right now?
@Jay can you take a look at this thread and share your thoughts? I think supporting a
buildCmd
as Tyler suggested makes sense. Even with the Go runtime I’m working on right now I can see a usecase of overriding the default
go build
command.
t

Tyler Flint

04/02/2021, 5:04 PM
@Frank for the time being, I’m using my existing webpack build process to generate a non-optimized javascript file, then I inform sst to start there. It then gets pumped through esbuild which doesn’t do much since it’s already bundled from webpack. It works, but it’s not ideal because I have to turn off all of the optimizations in webpack and currently I don’t have any way to enable optimizations in esbuild.
here’s my function definition:
Copy code
graphql: new sst.Function(this, "graphql", {
        handler: 'dist/graphql.main',
        environment: env,
        memorySize: 256,
        timeout: 30,
      }),
and
dist/graphql.main
is a generated file. Here’s a shot of the top of the file:
f

Frank

04/02/2021, 5:06 PM
Got it. Can I see the webpack script/command u r using?
t

Tyler Flint

04/02/2021, 5:07 PM
Copy code
const fs                     = require('fs');
const path                   = require('path');
const webpack                = require('webpack');
const { CleanWebpackPlugin } = require('clean-webpack-plugin');

const entries = (dir) => {
  return fs.readdirSync(path.resolve(__dirname, dir)).reduce((res, file) => {
    // split the file at the extension
    const [name, extension] = file.split('.');
    
    // only add the entry if it's a coffeescript or javascript file
    if (extension.match(/coffee|js/))
      res[name] = `${path.resolve(__dirname, dir)}/${file}`;
      
    return res;
  }, {});
};

module.exports = {
  entry: entries('src/lambda'),
  output: {
    path: path.resolve(__dirname, 'dist'),
    filename: '[name].js',
    libraryTarget: "umd"
  },
  mode: 'development',
  target: 'node',
  externals: ['aws-sdk', 'bufferutil', 'utf-8-validate'],
  node: false,
  optimization: {
    usedExports: true,
  },
  plugins: [
    new webpack.ContextReplacementPlugin(/any-promise/),
    new CleanWebpackPlugin()
  ],
  devtool: 'inline-cheap-module-source-map',
  module: {
    rules: [
      {
        test: /\.coffee$/,
        loader: 'coffee-loader',
        options: {
          inlineMap: true,
          transpile: {
            presets: ['@babel/env']
          }
        }
      }
    ]
  },
  resolve: {
    alias: {
      '~'                         : path.resolve(__dirname, 'src'),
      'busboy'                    : path.resolve(__dirname, 'src/app/null.coffee'),
      'apollo-reporting-protobuf' : path.resolve(__dirname, 'src/app/null.coffee'),
      'apollo-tools'              : path.resolve(__dirname, 'src/app/null.coffee'),
    },
    extensions: [ '.wasm', '.mjs', '.js', '.coffee', '.json' ]
  }
};
we still use coffeescript extensively, which isn’t supported through esbuild
and we also gut the garbage out of the apollo project that isn’t needed for serverless deploys
to cut literally MBs out of the resulting bundle
f

Frank

04/02/2021, 5:10 PM
I see.. yeah that makes sense
t

Tyler Flint

04/02/2021, 5:11 PM
at any rate, there are probably ways to get it to work with esbuild by first compiling coffeescript to javascript, but then I’d have to figure out how to ensure the sourcemaps make it all the way through.
then I’d have to figure out how to strip cruft out apollo. Unfortunately the tree-shaking stuff doesn’t work there because of the way they’ve tied all of their types together. Even one function import that has nothing to do with file uploads will end up including a huge library that isn’t needed.
anyways, for me the solution right now is to just keep using webpack for compiling the source, then piping that through esbuild. Ideally, I could just dump the result directly, as was described above.
f

Frank

04/02/2021, 5:16 PM
right.. definitely not ideal.. you probably would have to start ur own webpack watcher to output to
/dist
and
sst start
watcher gets then triggered
t

Tyler Flint

04/02/2021, 5:17 PM
yeah, here’s what I’m currently doing:
in my package.json:
Copy code
"scripts": {
    "dev": "concurrently 'webpack --watch' 'sst start'",
it works great
f

Frank

04/02/2021, 5:18 PM
ah i see… interesting!
t

Tyler Flint

04/02/2021, 5:19 PM
I also have a
compile
script that is just
webpack
so for my seed configuration I’ll tell it to run
compile
first before build
that’s what I’m about to do right now
so looking through the source for
sst start
, you could support a ton of languages right off just by conditionally looking for a
devCmd
or similar, and if it exists run that in a pty process instead of starting the watcher/builder.
as far as I can tell, the logic to wire up a request from a wss message into a local handler is not specific to javascript.
and for packaging a function for deployment, the existence of a
buildCmd
or similar could just end-run the build process and zip the contents.
you could literally shove a rails app into it haha. I mean, the cold-start time would probably be horrible but the point is that sst doesn’t have to care.