Anyone know how to implement <top-level await in A...
# help
d
Anyone know how to implement top-level await in AWS Lambda when using
sst.function
? Seems like there are only two options: 1. add
"type": "module",
to
package.json
which don’t think works since we only have a single package.json for all out lambdas 2. save the file as extension
*.cjs
do we have access to this with
sst.Function
?
t
Did you see the new support we added for esm?
Can mark a specific function as esm but I didn't test mixing the two
d
@thdxr I did not see that. What do you mean by “mixing the two”? Going to check the release notes and docs for esm
docs say `_Type_ : “esm” | “cjs”, defaults to 
cjs
so does that mean that all functions are already by default saved with file extension
cjs
?
so should top level await should work by default?
t
This is all extremely confusing haha. By default node treats everything as commonjs unless it has mjs. If you have type: module it will invert the behaviour and treat everything as ESM unless it has cjs
Top level await afaik only works for esm
If you set format: esm we write a package.json setting type: module to the build folder of each function that has it
d
Oh, I guess I was reading that incorrectly… so ESM is what it needs to be and NOT CJS to have top level await. If we set ALL our TS built functions to ESM…. there is no downside is there?
t
Depending on your dependencies you might run into issues. You have to be conservative with deps and only use ones that shit esm
Ship lol sorry on my phone
d
doh! … so its not just a matter of editing the package.json or file extension to get top level await…. your funciton needs to be compiled as ESM as well.
I wonder how hard it is to only use ESM deps. hmmm…. looking at function…
t
Try turning it on and do sst start and see what happens
d
hmmm…. wow…. none of the VERY common libs we use are ESM. 😛 not even axios
r
@thdxr seeing this has already come up a few times, i wonder if its worth exploring this plugin. https://github.com/hyrious/esbuild-plugin-commonjs I know its not great and feels a lot like a hack but maybe worth it until esbuild gets full esm output support? It essentially hoists dynamic requires to top level imports.
t
Worth a try we already support esbuild plugins
r
Actually... can we add plugins using the esbuild config option?
I am guessing we can...
t
Yeah it's a bit weird though you have to point to a file that exports the plugins
I think there's examples in our docs
d
cool… worth looking at. Thanks @Ross Gerbasi
t
ESM is a big step. I'm commiting to it personally but it really limits dependency choices (which I actually prefer)
I was thinking the other day, vite uses esbuild and never has this issue, wonder if they use this plugin
r
I was just thinking about this like 5 mins ago haha
I am not sure the output is ESM though
t
I think it is because it depends on your browsers ability to use esm. I wonder if they build each dependency specifically
When converting CommonJS dependencies, Vite performs smart import analysis so that named imports to CommonJS modules will work as expected even if the exports are dynamically assigned (e.g. React):
r
vite also does some different "bundling"
t
I would love to build on top of vite but need to understand it better
r
its splits things up by usage
you dont just get an
app.js
@thdxr alright so the plugin totally works....
I needed to change my
package.json
to
type: module
, then swap my esbuild format to
Copy code
bundle: {
          format: 'esm',
        },
d
@Ross Gerbasi I think @thdxr said that rolling with format: esm, they put a package.json in all functions with type: module
r
however there seems to be some issue with how plugins are sent into esbuild.
The esbuild docs show we should be able to apss in a EsBuild plugin. https://esbuild.github.io/plugins/#using-plugins
In order to get this working i modified the generated
node.js
file from this https://github.com/serverless-stack/serverless-stack/blob/master/packages/core/src/runtime/handler/node.ts#L156
to look like this
Copy code
const {commonjs} = await import ('@hyrious/esbuild-plugin-commonjs')
                const result = await esbuild.build({
                    ...config,
                    plugins: [commonjs()], //? require(plugins) : undefined,
                    metafile: true,
                    minify: false,
                    incremental: true,
                });
t
It's complicated, cdk forces things to not be async. So thatll work for local dev but not for deploy to prod
r
r
oooo boy. haha
t
Yeah cdks stupid approach of using constructors instead of functions means we have all kinds of hacks to support things that are async
I have an idea of getting away from that
r
What would you think about including
@hyrious/esbuild-plugin-commonjs
with SST and flipping it on as aplugin when using ESM... maybe some other option we can pass in? transformCommonJS or something?
Or if you dont want to trust that random npm package , maybe just implement it into SST itself.. its not a lot of code, transform seems pretty straight forward
This would open up SST ESM to a considerable amount of ESM development
I think far fewer packages do dynamic imports
t
I think because of this hack it's difficult for us to automatically use an esbuild plugin. Let me give it some thought on how to implement, it's a good idea
r
yeah i agree
its dirty for sure
but honesty the ESM option is kind of useless without something
I mean getting anything built with FULL esm packages is going to be a tall order haha
possible.. but unliklely
just for the record, in reference to our conversation earlier about this, after the plugin this is what you get in your transpiled code
Copy code
// node_modules/jwt-simple/lib/jwt.js
import __import_crypto from "crypto";
var require_jwt = __commonJS({
  "node_modules/jwt-simple/lib/jwt.js"(exports, module) {
    var crypto = __import_crypto;
so hoisted the require to an import and linked it back up that way
t
I'm somehow using all esm libraries lol but I keep deps to a minimum
r
Really? not even like
require('path')
or something? what the heck man haha
t
Haha yeah idk haven't run into the issue somehow
Did you actually get that esbuild plugin working?
I tried real quick and it didn't right away
I wonder if
swc
compiles more correctly
r
It was working when i hacked around in the code and force added it
i never tired to add it as a proper plugin. i just went back to cjs for now
t
That plugin doesn't work consistently...but that super hacky script seems to work. I think I'm going to build it into SST, as ugly as it is
r
I still am surprised this is a thing with esbuild. its used everywhere?! how is it not a bigger deal that it can't generate esm compatible code?!
t
Idk how vite is dealing with this
I just added this monstrosity to sst
Copy code
const ESM_HACK_REGEX =
  /\b__require\("(_http_agent|_http_client|_http_common|_http_incoming|_http_outgoing|_http_server|_stream_duplex|_stream_passthrough|_stream_readable|_stream_transform|_stream_wrap|_stream_writable|_tls_common|_tls_wrap|assert|async_hooks|buffer|child_process|cluster|console|constants|crypto|dgram|diagnostics_channel|dns|domain|events|fs|http|http2|https|inspector|module|net|os|path|perf_hooks|process|punycode|querystring|readline|repl|stream|string_decoder|sys|timers|tls|trace_events|tty|url|util|v8|vm|wasi|worker_threads|zlib)"\)/gm;
function esmHack(target: string) {
  const data = fs.readFileSync(target, "utf-8");
  const modules: Record<string, string> = {};
  const out = data.replace(ESM_HACK_REGEX, (_, mod) => {
    const id = "__import_" + mod.toUpperCase();
    modules[mod] = id;
    return id;
  });
  fs.writeFileSync(
    target,
    [
      ...Object.entries(modules).map(
        ([key, val]) => `import ${val} from ${JSON.stringify(key)};`
      ),
      out,
    ].join("\n")
  );
}
This fixes in my test case with the
jsonwebtoken
+
uuid
library
hopefully it works for you too
r
oh boy, haha I will try to take a look next chance I get.. what a mess 🙂 And yeah vite does seem to handle this just fine. maybe rollup is doing the magic here?
t
For vite dev mode rollup isn't involved. Let me see if I can get some answers from them
r
What if you built with vite? 😛 https://vitejs.dev/guide/api-javascript.html#build
t
I've been considering trying to build on top of vite in which case we can maybe do something with vitest
haven't explored it at all though
r
I see so less about the build and more interested in their runtime transform.
r
If i just throw together a blank vanillajs vite app and go back to our sturdy jwt-simple like this in the main.js
Copy code
import jwt from 'jwt-simple'
console.log(jwt)
the browser does attempt to load
<http://localhost:3000/node_modules/.vite/jwt-simple.js?v=1c43ceb5>
t
can you send me the contents of that file
r
and the transformed response does not have require at all, I wonder if it is because the file is transformed for the browser?
t
ahh
r
so esbuild platform is not node? so require cant even exist?
t
I wonder what happens if we built in browser mode...
r
well we would lose node dependencies
so if i try to use
jwt.decode
it throw an error in the browser
Buffer
is not defined
obviously this package isnt meant for the frontend
t
yeah it throws hard errors when node stdlib packages aren't found
r
Here is the transformed file
t
I see
r
I am not sure if vite can build for node...
🤔
t
I started to look into using vite for backend but it didn't seem like a thing
r
oo interesting
sooo
npx vite build --ssr main.js
Copy code
'use strict'
var jwt = require('jwt-simple')
function _interopDefaultLegacy(e) {
  return e && typeof e === 'object' && 'default' in e ? e : { default: e }
}
I figured SSR probably would not strip out node imports as it can use that stuff on the server...
t
So I think what esbuild might do is on first run, take all your dependencies and use the esbuild transform api to transpile them into ESM. And then it imports them as normal esm packages so esbuild won't rewrite them
r
do you mean what vite does on first run?
t
er yeah vite
r
no worries, i was with ya just confirming 🙂
t
idk - either way my current hack as ugly as it is is preferable to trying to do this much. I'm hoping esbuild fixes this issue
r
Yeah sure seems like something esbuild should support.
I wonder what happens if you take a SST project and try to compile a lambda handler as the entry with
vite --ssr
though...
t
it might only output CJS for node code
r
sure, but does it properly handle requires? we may not get ESM but we may get something compatible...
t
But this is only a problem in esm mode right?
a require statement isn't allowed in ESM at all
r
dynamic require
right?
top level require was ok?
t
oh right I see what you're saying
r
The downside to this, is there could be a performance hit here, as if we move all dynamic requires to top level your lambda would need to load them all
t
I guess if I could tell esbuild to not transform all cjs deps then it would work as long as they don't have dynamic requires
r
regardless of if they were needed
so when the invocation starts up you have to load all requires even if you never call functions that needed them in the first place
t
Yeah which I'm probably willing to tradeoff because I don't think any of my deps use dynamic requires
The hack I added to sst only lifts requires of node std libraries
I think ultimately it's just that esbuild does not support cjs -> esm transformation yet
r
oh gosh.. ok so i think i got something
given this starting point in
main.ts
Copy code
import JWT from 'jwt-simple'

console.log(JWT)
const sleep = () => new Promise((resolve) => setTimeout(resolve, 2000))

const g = await sleep()
console.log(g)

export const handler = async (event) => {
  return {
    event,
    statusCode: 200,
    body: 'Hello World!',
  }
}
I then run esbuild,
npx esbuild main.ts --format=esm --platform=node > main-es.js
but not a bundle. simply just the TS transform to JS
then using the following rollup config.
Copy code
import { nodeResolve } from '@rollup/plugin-node-resolve';
import commonjs from '@rollup/plugin-commonjs';


export default {
  input: './main-es.js',
  output: {
    format: 'es',
  },
  plugins: [nodeResolve(), commonjs()]
}
I run
Copy code
import { nodeResolve } from '@rollup/plugin-node-resolve';
import commonjs from '@rollup/plugin-commonjs';


export default {
  input: './main-es.js',
  output: {
    format: 'es',
  },
  plugins: [nodeResolve(), commonjs()]
}
I end up with a ESM compatible, fully bundled, top level await supporting js file...
Like this
t
Yeah rollup doesn't use esbuild. Which is fine for deployment but it's too slow for live development
r
is it? for bundling?
even if the transpile is handed off to esbuild?>
I mean vite is using rollup??
maybe not during dev time.... 🤷‍♂️
t
Yeah vite uses esbuild during dev time for fast reloads and rollup for deployment to bundle "properly"
Which is a path we can go with because we don't need to bundle any deps locally, it's all already there in node_modules. But it means local and prod match less
eg
mysql2
will work locally but not in prod until you add it to
bundle.nodeModules
r
Gotcha, well thats to bad, so we need esbuild bundling because its fast but sadly doesnt support esm
where as rollup produces the right output but is to slow...
javascript man...
t
I started to mess with
swc
as well but it doesn't do bundling well yet 😢
put me to sleep and wake me up in 2 years
r
Exactly, but with JS you will be saying that forever
t
lol
r
Well it was a good fight.. sadly we are stuck with hacking it together.
@thdxr we are back in business here 🙂
You probably got pinged on that github comment as well, but https://github.com/evanw/esbuild/issues/1944#issuecomment-1022886747 seems to work great
given the following
maint.ts
Copy code
import JWT from 'jwt-simple'

console.log(JWT)
const sleep = () => new Promise((resolve) => setTimeout(() => resolve('yup'), 2000))

const h = await sleep()
console.log(h)

const handler = (event: string) => {
  return {
    statusCode: 200,
    body: 'yay',
  }
}
my
package.json
is set to
"type": "module"
and
"main":"main.bundle.js"
build.js
is
Copy code
import esbuild from 'esbuild'

esbuild
  .build({
    entryPoints: ['./main.js'],
    bundle: true,
    platform: 'node',
    format: 'esm',
    outfile: './main.bundle.js',
    banner: {
      js: [
        `import { createRequire as topLevelCreateRequire } from 'module'`,
        `const require = topLevelCreateRequire(import.meta.url)`,
      ].join('\n'),
    },
  })
  .catch(() => process.exit(1))
running
node build.js
then
node .
results in
Copy code
{
  version: '0.5.6',
  decode: [Function: jwt_decode],
  encode: [Function: jwt_encode]
}
yup
with the top level await delay just before the
yup
So I think instead of that hack you can simply add a banner when the format is
esm
and we are good to go.
@thdxr reminder on this this stuff ^^^ just seeing if you think this is a better way to go.
t
woahh totally missed this
This is 100x better
I had that exact issue with
ulid
also
r
Yeah this should be a nicer way to handle this issue. Hopefully you sleep better with this then the hack 🙂
t
esm here we come 🎢
r
heck ya 🥳
Gonna await all the things now 🙂