hey guys, I have a potential problem with Lambda layers & binary executables. I have a version of sharp compiled for Amazon Linux that lives in the layer, and this works fine normally. but when I run in live debug mode, it doesn’t work:

Something went wrong installing the "sharp" module

Cannot find module '../build/Release/sharp-darwin-x64.node'
Require stack:
- /Volumes/SuperData/Sites/reelcrafter/v2-microservices/.build/src/services/media-processor/functions/get-metadata.js
- /Volumes/SuperData/Sites/reelcrafter/v2-microservices/node_modules/@serverless-stack/cli/scripts/util/bootstrap.js

my theory is that the Lambda code is running locally on the Darwin architecture, but it’s using the Lambda layer which is Amazon Linux arch. how can I solve this?

Yeah I ran into this with Prisma

thanks Dax! I guess I could just build on Mac & copy the darwin executable into the

build/Release

folder along w/ the Amazon one, then upload that to the layer

Okay so yeah that's exactly what I do. I had the opposite problem, Prisma by default installs what makes sense on my local machine. I use linux so I thought it would be fine but AWS requires something more specific. I had it install the more specific version when packaging out for aws

this works for me to use w/ layers:

SHARP_IGNORE_GLOBAL_LIBVIPS=1 npm install --arch=x64 --platform=linux sharp

yeah I had to do something similar with prisma, they look for an env variable to override the default

welp, that didn’t work 😞

When executing locally the code isn't bundled, it's just transpiled and executed so it should be using

node_modules

I figured. but I still get

Cannot find module '../build/Release/sharp-darwin-x64.node'

Was this ever working?

no

Can you try disabling the layer when in local mode?

I already did that:

lambdaFunction: new sst.Function(stack, 'GetMetadataFunc', {
      functionName: `${stack.stackName}-audioProcGetMetadata`,
      handler: `${srcPath}/get-metadata.main`,
      ...(process.env.IS_LOCAL ? {} : { layers: [sharpLayer] }),
      permissions: [[props!.uploadBucket.s3Bucket, 'grantRead']],
    }),

and just for good measure, in the global SST app:

bundle: {
      // sharp is a binary, and we have it in a Lambda layer, so exclude it
      externalModules: [...(process.env.IS_LOCAL ? [] : ['sharp']), 'knex'],
      loader: {
        '.node': 'binary',
      },
    },

Let me try to recreate your exact issue. I should be able to by creating a lambda function that references this library right

sure thing. I really appreciate it, as I’m totally blocked right now and it’s been frustrating 😕

😬 it works for me but I'm on linux

gimme a min. it doesn’t work when the app is deployed now, which is not how it was before.. one sec

I was referring to srcPath in function props

it is. using yarn workspaces

I think if it is a monorepo then it's expected to set the function's srcPath to the root of the sub-package the function is housed in. Otherwise it's not going to be using any package specific dependencies installed in

child/node_modules

ok, lemme try a few things

Yes but if there's no package.json there then it's not a subpackage in a monorepo is it?

it’s at the project root

Ah ok then your srcPath should be

services/media-processor

- I only have one subpackage for all my services not one per service

srcPath

defined at the stack level, per function, right? just for the media processor Lambdas

yeah that's right

ok. I’m actually testing this in my

rest-api

service instead, with just a dummy endpoint I can easily call to test.

'GET /sentry-test': {
      function: {
        handler: 'functions/sentry-test.main',
        srcPath: 'src/services/rest-api',
      },
      authorizationType: sst.ApiAuthorizationType.NONE,
    },

So is it working?

dunno yet..

yarn start

is rather slow, and it bombed a few times with errors

I'll be working on allowing that soon. In the mean time you technically can use

app.setDefaultFunctionProps

as long as you do it in each stack

ok.. running locally. deploying updates

Yeah we definitely need to make this clearer. I'd say yes you should always set srcPath to wherever package.json is

ahh man, that trick doesn’t work. redefining the srcPath. I have this in my MediaProcessor stack which is the last one that gets instantiated:

app.setDefaultFunctionProps({
      srcPath: 'src/services/media-processor',
    });

but oddly enough, I get an error:

Error: Cannot find a handler file for "src/services/graphql/functions/transcode-audio.main".

ah ok that one was a gamble

that would be huge

You can have your tsconfig extend from the base one. But also this was my reasoning for not having a subpackage for every service. I'm not fully sure of the benefits from having multiple packages besides enforcing some kind of boundary. I'm mostly concerned about seperating

core

vs

service

and not so much

service1

vs

service2

yeah, this is not practical 🤔

I'm not even fully sold on

core

vs

service

so what do you have set as yarn workspaces at your project root?

I have 2 packages in my mono repo.

src/core

that contains all my domains and functions inside them and then

src/services

which contains lambdas that import from

src/core

so

src/core

is your supporting functions for the CoreStack in your infra, right?

Yeah but the lambdas for it would go in

src/services

what’s in

src/core

then? sorry, I’m running on fumes the past 24 hrs

A more explicit example. In

src/core

I have

billing, bus, user

modules.

user

contains functions like

from_id

from_email

update_profile(...)

which all just talks to cognito.

billing

contains functions like

create

which creates a stripe customer. Then in

src/services

I have lambdas that react to a new Cognito user and publish an event

bus.publish

and another lambda subscribed to that event that calls

billing.create

to create a new stripe customer whenever a new cognito user is created

ahhh ok, so your

src/services

is just Lambda handlers that import & run code from core

yeah exactly

I guess I wanted to have separate stacks for the app, broken up into Core (auth, db, buckets), API (REST API, GraphQL), and MediaProcessor (handling uploads, transcoding media, processing thumbnails, etc)

I still have seperate stacks for each service

right.. yes please, fewer headaches 😄

Yeah roughly

yeah, my MediaProcessor deploys a step function full of Lambdas, and then separate Lambdas outside that

np! I've slowly been figuring out some good patterns over the past few months and will probably write up my opinionated version of project setup once I get my stuff out in production

I ran into this today, and this is what worked for me. 1. Added these envs in my

deploy

script:

"deploy": "SHARP_IGNORE_GLOBAL_LIBVIPS=1 npm_config_arch=x64 npm_config_platform=linux sst deploy",

2. Set my

bundle

config:

defaultFunctionProps: {
      // Include `sharp-linux-x64.node` binary needed in Lambda
      bundle: {
        nodeModules: ['sharp'],
        loader: {
          '.node': 'binary',
        },
      },

thanks Ted! my sharp binary is in a layer, so it works totally fine when invoked in the cloud, and when in live debug mode, it uses the local Mac binary