RE: Lambda ESM Note that I am not looking for help...
# sst
d
RE: Lambda ESM Note that I am not looking for help, just sharing Upon conversion to ESM, every graphql resolver fails with the following error:
{"errorType":"Error","errorMessage":"Dynamic require of \"stream\" is not supported","stack":["Error: Dynamic require of \"stream\" is not supported"," at file:///var/task/src/api/graphql/index.js:1:464"
I am not sure this is even the fault of the SST implementation, as it happens in AWS itself, but worth knowing perhaps. I am not doing anything with
stream
directly myself, so it must be inside apollo or another package.
It also didnt treeshake anything that I could identify. All of our internal packages publish with cjs and esm, and there were some obvious candidates.
t
What version of sst are you on
It turns out esbuild doesn't support esm to the degree something like rollup does and I added some better work around for the cases where esbuild fails
r
Should be v0.60.11+ with this, I believe https://github.com/serverless-stack/serverless-stack/pull/1319. As far as treeshaking did you try the pure comment i mentioned in your previous question about this? I had some struggles as well, just doesn't seem ESBuild is as good at shaking as some others.
d
Ill update to latest, was on 0.60.9
@Ross Gerbasi, I didn't, I simply resorted to splitting up the lib in question, as that is foolproof and it wasnt too hard. I may care more when getting into deeper optimization much later.
t
In my tests esbuild will treeshake at the file level. So if you
import { MyThing } from "./myfile"
it'll include all of
myfile
and all of
myfiles
imports. It will not trace
MyThing
and only import what MyThing references from what I could tell
r
I was specifically having trouble with generated code from our openapi -> TS stuff. marking things pure cleaned it up and kept a bunch out that was not being used.
d
In my case, I have a shared lib
@mycompany/base-models
which has cognito, dynamo, etc in it. I needed it to shake the SDK for cognito. it did not.
t
How are you importing from aws-sdk?
I only have a single package
@acme/core
which my functions import from and I generally see treeshaking happening (just not to the ultimate degree like rollup does)
d
Copy code
import {
    AdminAddUserToGroupCommand,
    AdminDeleteUserCommand,
    CognitoIdentityProviderClient,
} from "@aws-sdk/client-cognito-identity-provider";
t
ah v3
I was using v2 because v3 wasn't being shipped in the lambda env....and then it turns out the v2 version they ship doesn't work with esm. So I should probably go to v3
d
so long as you limit it to one, maybe two, its not too bad. about 0.2MB per.
t
If you use
Copy code
npx esbuild --analyze=verbose --target=esnext --platform=node --target=esnext --format=esm --bundle ./backend/functions/auth/events.ts
it'll tell you why everything is included
r
isnt the SDK externalized from bundles regardless?
d
v3 is not
updating made it work with ESM
the package size is identical to CJS, though
t
@Ross Coundon I had to make an update to not externalize it for esm because it doesn't work
r
wrong ross 🙂 but i am sure he will be happy to join, we get mixed up regularly haha
t
oops
r
so adding
@aws-sdk
to the external modules config for esbuild would be a no-go?
is it simply that AWS doesnt provide v3 like it does for v2?
d
correct
@thdxr, I ran your command, and it seems like it is using the cjs dist for everything
t
does your package.json specify entrypoints
d
for the API, I dont think so, but for all supporting libs it would
t
Copy code
"exports": {
    ".": {
      "import": "./dist/esm/index.js",
      "require": "./dist/cjs/index.js"
    }
  },
Copy code
"module": "dist/esm/index.js",
  "main": "dist/cjs/index.js",
I don't know which of these makes it work but I have them all in there
d
Copy code
"main": "dist/cjs/index.js",
 
Copy code
"module": "dist/esm/index.js",
i dont have the exports one...but i dont think thats the one, this works fine with webpack
I have
"sideEffects": false,
as well
t
Yeah but I think there's some kind of split that happened, I don't remember the contexts but I believe
exports
is the thing to use
d
well, it appears to be using the cjs of the aws-sdk packages, which presumably do handle this well, right?
t
Oh weird I wonder why it's doing that
oh add --main-fields=module,main
d
to the esbuild command?
that worked, and it seems like its grabbing esm
t
The docs are confusing because it says
The default main fields depend on the current platform setting and are essentially browser,module,main for the browser and main,module for node.
Which I understood as it's using main,module by default but I guess it's not?
d
if this command functions the same as the bundler for sst, then maybe just nothing can be tree-shook
t
The bunder for sst does not specify
--main-fields
d
that seemed to make a difference in the output in the console, but I dont really have a way of testing size
t
With that you see it grabbing the esm versions of aws-sdk?
d
correct
and my own libs as well
Copy code
node_modules/@aws-sdk/middleware-endpoint-discovery/dist-es/getCacheKey.js ──────────────────────────────────────────────────────── 16b ──── 0.0%
   │  └ node_modules/@aws-sdk/middleware-endpoint-discovery/dist-es/endpointDiscoveryMiddleware.js
   │     └ node_modules/@aws-sdk/middleware-endpoint-discovery/dist-es/getEndpointDiscoveryPlugin.js
   │        └ node_modules/@aws-sdk/middleware-endpoint-discovery/dist-es/index.js
   │           └ node_modules/@aws-sdk/client-dynamodb/dist-es/DynamoDBClient.js
   │              └ node_modules/@aws-sdk/client-dynamodb/dist-es/index.js
   │                 └ src/lib/dynamo/client.ts
   │                    └ src/lib/graphql/dataLoaders/commonDataLoaderModelProps.ts
r
to test could he just add that to the esbuild config here
node_modules/@serverless-stack/core/dist/runtime/handler/node.js
just hack it in and see what you're getting out?
I have spent to much time in that file haha
t
Ohh I think order matters. Default is
main,module
. I do remember I removed the
main
file for one of my libraries and it started to pickup the esm one
r
you thinking maybe when ESM is being used those should be supplied? with module first?
t
Yeah exactly
d
i...think I see where to put it...
Copy code
const result = await esbuild.build({
                    ...config,
                    plugins: plugins ? require(plugins) : undefined,
                    metafile: true,
                    minify: false,
                    incremental: true,
                });
no
t
I think further up there's a
const config =
d
Copy code
const config = {
        loader: bundle.loader,
        minify: bundle.minify,
        define: (_a = bundle.esbuildConfig) === null || _a === void 0 ? void 0 : _a.define,
        keepNames: (_b = bundle.esbuildConfig) === null || _b === void 0 ? void 0 : _b.keepNames,
        entryPoints: [path_1.default.join(opts.srcPath, file)],
        bundle: opts.bundle !== false,
        external: [
            ...(bundle.format === "esm" ? [] : ["aws-sdk"]),
            ...(bundle.externalModules || []),
            ...(bundle.nodeModules || []),
        ],
        sourcemap: true,
        platform: "node",
        ...(bundle.format === "esm"
            ? {
                target: "esnext",
                format: "esm",
                banner: {
                    js: [
                        `import { createRequire as topLevelCreateRequire } from 'module'`,
                        `const require = topLevelCreateRequire(import.meta.url)`,
                    ].join("\n"),
                },
            }
            : {
                target: "node14",
                format: "cjs",
            }),
        outfile: target,
    };
t
yeah
r
yeah slap it in there haha around line 106/107
d
mainFields: "module"
?
t
I think it has to be an array
mainFields: bundle.format === "esm" ? ["module", "main"] : ["main", "module"],
d
build seemed to work, and it did change...something. file is a little longer
deploying
Before size: 2.3MB After size: 2.4MB
t
why did it get bigger 🤯
d
it also 500s...checking
r
hahaha
d
Copy code
"Cannot use GraphQLSchema
LOTS HERE
from another module or realm.\n\nEnsure that there is only one instance of \"graphql\" in the node_modules\ndirectory. If different versions of \"graphql\" are the dependencies of other\nrelied on modules, use \"resolutions\" to ensure only one version is installed.\n\n<https://yarnpkg.com/en/docs/selective-version-resolutions>\n\nDuplicate \"graphql\" modules cannot be used at the same time since different\nversions may have different capabilities and behavior. The data from one\nversion used in the function from another could produce confusing and\nspurious results.
looks like it somehow shook some of the graphql package loose, and it needed it? and somehow still got bigger?
it also could be another
sst
package clash, I think you guys install the same as me, though
i dont see other versions, everyone is on 15, its something else
BUT, I will say that it appeared to work, and now uses esm versions
I can see in the output that MOST things use graphql
.mjs
files, but SOME things are still using
.js
, that is probably the problem
r
Man... modules are such a mess haha. we are in the worst times too. Is your stuff working now? just bigger?
d
it does not work, error is about 6 above
but I will say that the SST part DOES appear to work, this is just a dependency clash.
although, I think it is SST causing the clash, lol.
(its the only thing besides Apollo installing graphql, every other package is a peer, it could also be the double apollo)
r
oh interesting so maybe two things in your app are using different versions of the package? So when bundled its not going to allow that to work...
d
exactly, but the second thing is SST itself, i believe
it may also be SST => 2nd Apollo => 2nd GraphQL
I already have issues surrounding this, though, and they are working on it. Just looks like I need to wait to go to ESM till then, though. No biggie
t
I've seen this error before too, I don't remember how I resolved
I'm going to clean up all of our dependencies this month
s
I am also facing the same error, I am using this repo as a base to setup but I want to use Apollo to spin up the graphql server and trying to replace with the
serverless-stack/node
but even after trying for many hours wasn't able to get it working
d
I havent tried yet, I was waiting for SST to slim down their dependencies first (which they have now done, but I havent circled back)
it actually looks like they did NOT cut out graphql, though, so this error will still be there for me as well
the only way I have found to get around installed SST dependencies is just to use their version, but you can also post this issue in #help, because if you are running into issues on a stack they produced, they will want to know
s
Just sharing if someone can help. Below is the error I am getting
Copy code
{
  errorType: 'Runtime.UnhandledPromiseRejection',
  errorMessage:
    'Error: Cannot use GraphQLSchema "{ __validationErrors: undefined, description: undefined, extensions: undefined, astNode: undefined, extensionASTNodes: [], _queryType: Query, _mutationType: Mutation, _subscriptionType: undefined, _directives: [@cacheControl, @include, @skip, @deprecated, @specifiedBy], _typeMap: { Query: Query, String: String, Mutation: Mutation, CreateUserInput: CreateUserInput, User: User, CacheControlScope: CacheControlScope, Upload: Upload, Int: Int, Boolean: Boolean, __Schema: __Schema, __Type: __Type, __TypeKind: __TypeKind, __Field: __Field, __InputValue: __InputValue, __EnumValue: __EnumValue, __Directive: __Directive, __DirectiveLocation: __DirectiveLocation }, _subTypeMap: {}, _implementationsMap: {} }" from another module or realm.Ensure that there is only one instance of "graphql" in the node_modules directory. If different versions of "graphql" are the dependencies of other relied on modules, use "resolutions" to ensure only one version is installed.<https://yarnpkg.com/en/docs/selective-version-resolutionsDuplicate> "graphql" modules cannot be used at the same time since different versions may have different capabilities and behavior. The data from one version used in the function from another could produce confusing and spurious results.',
  reason:
    'Error: Cannot use GraphQLSchema "{ __validationErrors: undefined, description: undefined, extensions: undefined, astNode: undefined, extensionASTNodes: [], _queryType: Query, _mutationType: Mutation, _subscriptionType: undefined, _directives: [@cacheControl, @include, @skip, @deprecated, @specifiedBy], _typeMap: { Query: Query, String: String, Mutation: Mutation, CreateUserInput: CreateUserInput, User: User, CacheControlScope: CacheControlScope, Upload: Upload, Int: Int, Boolean: Boolean, __Schema: __Schema, __Type: __Type, __TypeKind: __TypeKind, __Field: __Field, __InputValue: __InputValue, __EnumValue: __EnumValue, __Directive: __Directive, __DirectiveLocation: __DirectiveLocation }, _subTypeMap: {}, _implementationsMap: {} }" from another module or realm.Ensure that there is only one instance of "graphql" in the node_modules directory. If different versions of "graphql" are the dependencies of other relied on modules, use "resolutions" to ensure only one version is installed.<https://yarnpkg.com/en/docs/selective-version-resolutionsDuplicate> "graphql" modules cannot be used at the same time since different versions may have different capabilities and behavior. The data from one version used in the function from another could produce confusing and spurious results.',
  promise: {},
  stack: [
    'Runtime.UnhandledPromiseRejection: Error: Cannot use GraphQLSchema "{ __validationErrors: undefined, description: undefined, extensions: undefined, astNode: undefined, extensionASTNodes: [], _queryType: Query, _mutationType: Mutation, _subscriptionType: undefined, _directives: [@cacheControl, @include, @skip, @deprecated, @specifiedBy], _typeMap: { Query: Query, String: String, Mutation: Mutation, CreateUserInput: CreateUserInput, User: User, CacheControlScope: CacheControlScope, Upload: Upload, Int: Int, Boolean: Boolean, __Schema: __Schema, __Type: __Type, __TypeKind: __TypeKind, __Field: __Field, __InputValue: __InputValue, __EnumValue: __EnumValue, __Directive: __Directive, __DirectiveLocation: __DirectiveLocation }, _subTypeMap: {}, _implementationsMap: {} }" from another module or realm.',
    '',
    'Ensure that there is only one instance of "graphql" in the node_modules',
    'directory. If different versions of "graphql" are the dependencies of other',
    'relied on modules, use "resolutions" to ensure only one version is installed.',
    '',
    '<https://yarnpkg.com/en/docs/selective-version-resolutions>',
    '',
    'Duplicate "graphql" modules cannot be used at the same time since different',
    'versions may have different capabilities and behavior. The data from one',
    'version used in the function from another could produce confusing and',
    'spurious results.',
    '    at process.<anonymous> (file:///home/MYPC/Desktop/Freelancing/backend/node_modules/@serverless-stack/aws-lambda-ric/lib/index.js:34:23)',
    '    at process.emit (events.js:376:20)',
    '    at processPromiseRejections (internal/process/promises.js:245:33)',
    '    at processTicksAndRejections (internal/process/task_queues.js:96:32)',
  ],
}
Thanks @Derek Kershner, I will post in the help channel
t
@Shadab that repo is very tied to not using apollo since Apollo does not support ESM and is not bundled properly. Any reason why you want to use apollo? Also that repo isn't ready yet we're aiming to release it properly in April
s
Well because I have used Apollo in past and have good familiarity with it, With Helix, I am not that confident, as it's relatively new and apollo is widely used. As I think the
apollo-lambada-server
version of graphql is conflicting,
Yeah but the repo is still great and helped in setting up a graphql server with other requirements like RDS, AWS Cognito etc
t
Helix is actually fairly widely used, it's not a graphql server itself it's a way to build graphql servers, eg graphql-yoga uses it. And having review the repos of both Apollo and Helix, even though Apollo is widely used it's poorly implemented
redwoodjs uses helix as well
s
Thanks, @thdxr, for the inputs. We are in the research phase of starting our new project and looking at different things, serverless-stack framework surely the team liked a lot, Just these kind of few things are confusing us, We plan to use RDS, GraphQL
t
let me know what questions you have! I was where you are a year ago and was lucky to be able to spend a year researching - the graphql stack contains 90% of the results of my research but I haven't documented why I chose each thing