https://serverless-stack.com/ logo
Join Slack
Powered by
There appear to be limitations around the Live Lam...
# sst
s
There appear to be limitations around the Live Lambda development environment when running in a VPC. For example, I am using MSK (Kafka) as an event source for Lambda. MSK runs within a VPC, so I must configure my Lambda function to work within the VPC to facilitate that communication. The stub lambda that enables the live-lambda environment lives outside of that VPC, which causes the Kafka->Lambda integration to fail when running 
sst start
.
Is this something I can overcome with my VPC configuration, or is this a limitation in the framework?
a
Have you had a look at Connecting to a VPC on the SST live lambda docs?
a
Can you do it with VPN?
s
Well, I do have a VPN set up on my development laptop
so I can connect to resources like RDS from my laptop with no problem
The problem I'm having is something within my VPC (Kafka) is trying to call the SST lambda stub that doesn't live in the same VPC
a
Ohhh I seee.
Yeah I get it now.
Mmmm, but your Kafka should call the lambda which is in the cloud.
I mean, isn’t a lambda deployed and a function running locally?
Should be a way, technically speaking.
s
Yeah, that's what I'm wondering
when I perform an 
sst deploy
, everything works fine since my lambda is deployed in the same VPC as Kafka
a
Yeah might be possible… I had some many issues with VPC/VPN that always get stuck on this stuff.
s
but when I run 
sst start
, my "lambda in the cloud" is the SST stub that is proxying requests back to my laptop
that SST stub lambda isn't configured to be in the same VPC as my lambda
At least I think this is the problem...this is the first time I've developed with resources in an VPC 🤷
a
I see.
Might be a way to put a 
vpc
parameter to that lambda.
Can you edit the lambda in Console? haha.
And assign it a VPC.
s
haha, I wonder
that would be hilarious
t
Are you saying when you run sst start with a vpc specified, we deploy to the wrong vpc?
s
What do you mean by "with a vpc specified"?
MSK is running within a VPC, and I've configured the MSK to Lambda integration. I see the docs I linked say "The Lambda function target in the event source mapping does not need to be running in a VPC to receive messages from Amazon MSK.", so I suspect this could work if I get my VPC setup configured properly (something I know very little about). I'm mostly trying to confirm if this is expected behavior, or if this is possible to get working if I just get the dang VPC configured properly 🙂
This is my stack
Copy code
import * as sst from "@serverless-stack/resources";
import * as lambda from '@aws-cdk/aws-lambda';
import { ManagedKafkaEventSource } from '@aws-cdk/aws-lambda-event-sources';
import * as ec2 from '@aws-cdk/aws-ec2';

export default class MyStack extends sst.Stack {

  constructor(scope, id, props) {
    super(scope, id, props);

    // Your MSK cluster arn
    const clusterArn = "arn:aws:kafka:us-east-1:ACCOUNT_ID:cluster/stage-analytics/XXXXXXXXXXXXXX"
    const vpc = ec2.Vpc.fromLookup(this, 'VPC', {
      vpcName: 'stage_vpc',
    });
    const securityGroup = ec2.SecurityGroup.fromSecurityGroupId(
      this,
      'SG',
      'sg-YYYYYYYYYYYYY'
    );

    const kafkaIngestFunction = new sst.Function(this, "KafkaIngestHandler", {
      handler: "/src/lambda.handler"
    });

    kafkaIngestFunction.addEventSource(new ManagedKafkaEventSource({
      clusterArn,
      topic: 'analytics_cdc.public.profile',
      batchSize: 100, // default
      startingPosition: lambda.StartingPosition.TRIM_HORIZON
    }));

  }
}
and my lambda is currently doing very little
Copy code
export async function handler(event) {
  return {
    statusCode: 200,
    headers: { "Content-Type": "text/plain" },
    body: `Hello, World! Your request was received!.`,
  };
}
To be clear, this works perfectly when I do an 
sst deploy
. It's the 
sst start
live lambda environment that doesn't work for me
t
Ah it's the target. Hmm wonder why this doesn't work on sst start. Should be just a normal invoke with a payload that MSK is doing
s
yeah, that was my expectation
t
can you tell in the aws logs if it's trying to invoke?
Wondering if it's an invoke + a timeout or for some reason it's not invoking at all
s
trying to track it down now (the stack isn't currently running)
Copy code
2021-11-10T23:39:14.491Z	8faab32b-f3ed-4fa9-9848-5f83d0a865a8	INFO	ws.onerror ErrorEvent {
  target: WebSocket {
    _events: [Object: null prototype] {
      open: [Function],
      close: [Function],
      message: [Function],
      error: [Function]
    },
    _eventsCount: 4,
    _maxListeners: undefined,
    _binaryType: 'nodebuffer',
    _closeCode: 1006,
    _closeFrameReceived: false,
    _closeFrameSent: false,
    _closeMessage: '',
    _closeTimer: null,
    _extensions: {},
    _protocol: '',
    _readyState: 2,
    _receiver: null,
    _sender: null,
    _socket: null,
    _bufferedAmount: 0,
    _isServer: false,
    _redirects: 0,
    _url: '<wss://1ilbjbs70k.execute-api.us-east-1.amazonaws.com/sgeoghegan>',
    _req: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 5,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      socket: [TLSSocket],
      connection: [TLSSocket],
      _header: 'GET /sgeoghegan HTTP/1.1\r\n' +
        'Sec-WebSocket-Version: 13\r\n' +
        'Sec-WebSocket-Key: mZTxW3tpUSFllakA5mkJLQ==\r\n' +
        'Connection: Upgrade\r\n' +
        'Upgrade: websocket\r\n' +
        'Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits\r\n' +
        'Host: <http://1ilbjbs70k.execute-api.us-east-1.amazonaws.com|1ilbjbs70k.execute-api.us-east-1.amazonaws.com>\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: noopPendingOutput],
      agent: undefined,
      socketPath: undefined,
      method: 'GET',
      insecureHTTPParser: undefined,
      path: '/sgeoghegan',
      _ended: false,
      res: [IncomingMessage],
      aborted: true,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: [HTTPParser],
      maxHeadersCount: null,
      reusedSocket: false,
      host: '<http://1ilbjbs70k.execute-api.us-east-1.amazonaws.com|1ilbjbs70k.execute-api.us-east-1.amazonaws.com>',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    [Symbol(kCapture)]: false
  },
  type: 'error',
  message: 'Unexpected server response: 403',
  error: Error: Unexpected server response: 403
      at ClientRequest.<anonymous> (/var/task/node_modules/ws/lib/websocket.js:604:7)
      at ClientRequest.emit (events.js:314:20)
      at ClientRequest.EventEmitter.emit (domain.js:483:12)
      at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:601:27)
      at HTTPParser.parserOnHeadersComplete (_http_common.js:122:17)
      at TLSSocket.socketOnData (_http_client.js:474:22)
      at TLSSocket.emit (events.js:314:20)
      at TLSSocket.EventEmitter.emit (domain.js:483:12)
      at addChunk (_stream_readable.js:297:12)
      at readableAddChunk (_stream_readable.js:272:9)
}
t
@Frank is more familiar with the nodejs stub
f
Hey @Seth Geoghegan, it seems the stub lambda is not able to connect to the websocket server that’s doing the messaging sending to ur local.
Can you go into ur CloudFormation console and see if you can spot the debug stack? Should have the name of 
sgeoghegan-xxxx-debug-stack
. Remove the stack. Then restart 
sst start
.
This will re-create the debug stack.
s
Hey Frank, I'll post an update when I'm back in front of my laptop and have a chance to try this
Open in Slack
PreviousNext
Powered by