Is there a way in SST Function to have it BAKE in environmen Serverless Stack #help

Is there a way in SST Function to have it BAKE in ...

Dan Van Brunt

09/21/2021, 5:17 PM

Is there a way in SST Function to have it BAKE in environment vars? rather than passing them in as Function.environment ? Lambda@Edge does not support environment vars so it would be cool if there was an option to bake them in at build-time?

thdxr

09/21/2021, 5:19 PM

This is sort of what we do with StaticSite, basically do a find replace once everything is deployed

thdxr

09/21/2021, 5:20 PM

Are the variables you want to bake in referencing other resources created by sst?

thdxr

09/21/2021, 5:20 PM

Eg an sst.Api or a queue

Dan Van Brunt

09/21/2021, 5:20 PM

yes…. but should this not be possible just with the native Function esbuild script?

Dan Van Brunt

09/21/2021, 5:21 PM

no, for this right now its just baking in envs from the build systems’ envs

thdxr

09/21/2021, 5:23 PM

Yeah I was asking because if all the values are available you can do this with a dynamic require

thdxr

09/21/2021, 5:24 PM

Copy code

const Config = require(process.env.ENV_CONFIG)

thdxr

09/21/2021, 5:24 PM

esbuild should bundle those inline

thdxr

09/21/2021, 5:24 PM

that can point to a file that exports an object

{ OPT1: "Hello"}

thdxr

09/21/2021, 5:25 PM

Another option is to just bundle all the configs for all environments and at runtime do

Config[my_env].OPT1

Dan Van Brunt

09/21/2021, 5:25 PM

ah!… ya that could work… https://github.com/serverless-stack/serverless-stack/issues/845

Dan Van Brunt

09/21/2021, 5:25 PM

Guess I didn’t need that then 🙂

Camilo Bravo

09/21/2021, 5:41 PM

Copy code

const Config = require(process.env.ENV_CONFIG)

This line is throwing a few typescript issues @thdxr

Camilo Bravo

09/21/2021, 5:42 PM

Require statement not part of import statement.eslint@typescript-eslint/no-var-requires

Dan Van Brunt

09/21/2021, 5:44 PM

Copy code

// eslint-disable-line no-eval

Dan Van Brunt

09/21/2021, 5:44 PM

If you put

/* eslint-disable no-eval */

before any code in a

.js

file, that will disable the

no-eval

rule for the entire file.

thdxr

09/21/2021, 5:45 PM

I haven't actually tested using

require

like that with esbuild so it might not work

Dan Van Brunt

09/21/2021, 6:18 PM

yea, @thdxr, doesn’t seem to work. It compiles, but we see

process.env

in the bundle output.

Dan Van Brunt

09/21/2021, 6:20 PM

Perhaps adding

define

property to the build props? https://github.com/evanw/esbuild/issues/438#issuecomment-705120203

thdxr

09/21/2021, 6:21 PM

Ah I think we do support custom esbuild config

Dan Van Brunt

09/21/2021, 6:36 PM

@thdxr is that an issue though…. with it overriding any defaults you guys have in place now? perhaps some kinda plugin? https://github.com/rw3iss/esbuild-envfile-plugin

thdxr

09/21/2021, 6:38 PM

I think we merge the configs with our standard

Dan Van Brunt

09/21/2021, 6:38 PM

ah … cool

thdxr

09/21/2021, 6:38 PM

I haven't used esbuild plugins yet so I'm not familiar off the top of my head

Camilo Bravo

09/21/2021, 6:52 PM

Copy code

Error: Cannot configure the "define" option in "/esbuild.js". Only the "plugins" option is currently supported.

Looks like there is a check to prevent any customization of the esbuild config beyond plugins

Frank

09/21/2021, 8:20 PM

This approach only works for static value right?

Dan Van Brunt

09/21/2021, 8:20 PM

yes?

Frank

09/21/2021, 8:20 PM

It won’t work if you want to set values that aren’t available at build time, ie.

Copy code

TABLE_NAME: table.tableName

Dan Van Brunt

09/21/2021, 8:20 PM

correct

Dan Van Brunt

09/21/2021, 8:20 PM

not for CDK resource values…. but it should work for things like

this.stackName

no?

Frank

09/21/2021, 8:21 PM

Right. There’s a trick where you use CloudFront origin headers to pass “environment variables” to edge functions https://stackoverflow.com/a/58101487/7667989

Frank

09/21/2021, 8:22 PM

I was thinking to bake it into SST

Dan Van Brunt

09/21/2021, 8:22 PM

hmmm…. ya… I remember that as well.

Frank

09/21/2021, 8:23 PM

table.tableName

should work in this case

Dan Van Brunt

09/21/2021, 8:23 PM

hmmm… maybe this is cleaner for now…. is there any limits to CF headers?

Frank

09/21/2021, 8:24 PM

oh good call

Frank

09/21/2021, 8:24 PM

The maximum length of a request, including the path, the query string (if any), and headers, is 20,480 bytes

Frank

09/21/2021, 8:25 PM

As a reference, Lambda env var limit is 4kb

Dan Van Brunt

09/21/2021, 8:25 PM

that seems like alot

Dan Van Brunt

09/21/2021, 8:26 PM

20,480 chars = 20,480 bytes I think?

Frank

09/21/2021, 8:26 PM

yeah 20kb

Dan Van Brunt

09/21/2021, 8:32 PM

unfortunately the headers method wont work for us because our distro is in another stack that our lambda@edge. 😄

Dan Van Brunt

09/21/2021, 8:33 PM

@Frank if your cool with it, I’m going to see if I can update

Function

here to allow for inline config? Similar to what you did on line 283.

Dan Van Brunt

09/21/2021, 8:34 PM

that sound like it would work? then allow for

esbuildConfig

to be an object OR a string.

Frank

09/21/2021, 9:01 PM

Would it work we allowed the

define

config here? https://github.com/iDVB/serverless-stack/blob/fcc44adcdae3459bf70d84084cbca640e642d879/packages/resources/assets/nodejs/esbuild.js#L27-L29

Frank

09/21/2021, 9:02 PM

Being able to define esbuildConfig inline also work, but we still want some restrictions on what they can override.

Frank

09/21/2021, 9:02 PM

SST relies on configs like

metafile

etc

Dan Van Brunt

09/21/2021, 9:03 PM

but if you allow for a config path …. why not also allow for inline config?

Frank

09/21/2021, 9:04 PM

on the phone be back in 30min 🙂

Frank

09/21/2021, 9:42 PM

back, yeah inline config works! If it only allowed

plugins

and

define

, would it work for you guys?

Dan Van Brunt

09/21/2021, 9:42 PM

yup

Dan Van Brunt

09/21/2021, 9:42 PM

almost there now with PR

Frank

09/21/2021, 9:43 PM

Man u guys move fast.. yeah I will cut a release with it tonite

Camilo Bravo

09/21/2021, 9:47 PM

currently we are just doing what was already there (only allowing

plugins

)

Camilo Bravo

09/21/2021, 9:47 PM

but if you think

defines

is okay that would probably be good to have too

Dan Van Brunt

09/21/2021, 9:50 PM

@Frank running into an issue between

nodeBuilder.ts

esbuild.js

….. we’re allowing esbuildConfig to be a string or a type that allows for

plugins: [Plugin]

which is fine for nodeBuilder but we are base64 encoding it to be sent to

esbuild.js

which you guys are already decoding for

defaultConfigValue

. We’d do the same thing for customConfig but at that point its either a Base64encoded string or a string path. So all I could think of is to again test for

!fs.existsSync(customConfig)

and if YES then its a string path and if NO then

Buffer.from

Dan Van Brunt

09/21/2021, 9:53 PM

that or @Camilo Bravo was thinking to add a

--inline-config

flag to

esbuild.js

and assume its a base64 string of flag exists

Frank

09/21/2021, 9:57 PM

This just reminded me why we didn’t take inline esbuild config, it’s b/c

plugins

isn’t serializable right? ie.

plugin

is an import right?

Copy code

esbuildConfig: {
  plugins: [plugin]
}

Frank

09/21/2021, 9:58 PM

Sorry I was confused earlier.

Dan Van Brunt

09/21/2021, 9:58 PM

is that the case? I’m not sure, I was hoping it would somehow just work 😛

Dan Van Brunt

09/21/2021, 9:59 PM

plugin could indeed be imported about yes

Dan Van Brunt

09/21/2021, 10:00 PM

does that mean it would not work? Guess I’m not sure why that would make it not serializable ?

Frank

09/21/2021, 10:02 PM

Yeah, 2 reasons: 1. a plugin contain callback functions 2.

sst start

also needs to run esbuild to build the functions, if defined inline,

sst start

won’t be able to figure out what to import.

Dan Van Brunt

09/21/2021, 10:04 PM

hmmm. doh… I guess PlanB is just allow for Function.bundle to set

define

Dan Van Brunt

09/21/2021, 10:05 PM

Gonna put a pin in this for the night. Dinner calls. 🙂 Let us know what you think of define option though.

Frank

09/21/2021, 10:07 PM

Yup. Let’s allow the custom config to accept both

define

and

plugins

. Does that work?

Frank

09/21/2021, 10:07 PM

https://github.com/iDVB/serverless-stack/blob/fcc44adcdae3459bf70d84084cbca640e642d879/packages/resources/assets/nodejs/esbuild.js#L27-L29

thdxr

09/21/2021, 11:45 PM

Maybe you're too far along but a dumb work around is writing the env vars to a file in your sst code which is referenced by your js code

Dan Van Brunt

09/22/2021, 12:49 AM

@thdxr Ya, that only works for static values but would not work for

this.stackName

which I think should be avail to be passed into

define

thdxr

09/22/2021, 1:03 AM

You can write that to a file inside your stack

thdxr

09/22/2021, 1:03 AM

pseudocode

Copy code

fs.write("src/handler/config.js", JSON.stringify({ stack: this.stackname }))

Dan Van Brunt

09/22/2021, 1:05 AM

Ya I can see it working but a bit wonky. Also for static values that already come from .env this would be just one more file. Be nice to just have all statics in .env

Dan Van Brunt

09/22/2021, 12:52 PM

@Frank how should we go about adding define support? If we add it to the external config file it still doesn’t let us pass in things like

this.stackName

. Unless we extend FunctionBundleNodejsProps to allow for

define

Frank

09/22/2021, 2:33 PM

@Dan Van Brunt there’s a recent PR for allowing the

keepNames

esbuild config. I was thinking we can add

define

to the allowed config similarly https://github.com/serverless-stack/serverless-stack/pull/807/files

Dan Van Brunt

09/22/2021, 2:35 PM

@Frank but doesn’t that mean that we have to use an external (not inline)

esbuildConfig

Dan Van Brunt

09/22/2021, 2:40 PM

If so, then it would not allow us to pass in

this.stackName

Frank

09/22/2021, 2:41 PM

hmm i see.. I was under the impression that you were going to use the

esbuild-envfile-plugin

, and plugins had to be defined in external file.

Dan Van Brunt

09/22/2021, 2:42 PM

well… that would take care of 1of2 cases

Dan Van Brunt

09/22/2021, 2:43 PM

we store all all statics in envs but we also need to pass in things like

this.stack

which is why we went down the path to try and allow for inline config.

Frank

09/22/2021, 2:56 PM

Right.. just to sum it up, you need to configure both the

define

and

plugins

esbuild config. And we are stuck right now b/c •

define

has to be configured inline to access `this.stackName`; • but

plugins

has to be configured externally That sounds right?

Frank

09/22/2021, 6:30 PM

Let me know if I’m understanding this correctly, and then I’m going to chat with @thdxr and @Jay quickly about it 😁

Camilo Bravo

09/22/2021, 6:31 PM

yes I believe that is correct. Let me just ask Dan real quick!

Dan Van Brunt

09/22/2021, 7:01 PM

@Frank @Camilo Bravo close actually. We don’t even need plugins IF we can have access to define, as it would allow us to do what we’re using a plugin to do.

Frank

09/23/2021, 2:39 AM

We settled on this https://github.com/serverless-stack/serverless-stack/issues/850 In short, you can do this:

Copy code

bundling: {
  esbuildConfig: {
    define: ...,
    keepNames: ...,
    plugins: "path/to/esbuild.js"
  }
}

You can pass in

define

inline, and

plugins

through an external file.

Frank

09/23/2021, 2:40 AM

Can you confirm this works for you?

Camilo Bravo

09/23/2021, 1:04 PM

yes that should work because we will be able to define all derived variables using that define which is what we were using a plugin for previously

Camilo Bravo

09/23/2021, 1:05 PM

@Dan Van Brunt confirm?

Frank

09/26/2021, 7:20 AM

Hey @Camilo Bravo @Dan Van Brunt in v0.34.7, you can now configure

define

inline like this:

Copy code

bundle: {
    esbuildConfig: {
      define: { … },
    },
  },

Camilo Bravo

09/27/2021, 1:18 PM

Thank you for moving so quickly on this! Much appreciated

2 Views

Open in Slack

Previous Next