These lambda log retentions are going to be the death of me Serverless Stack #help

These lambda log retentions are going to be the de...

Garret Harp

10/21/2021, 6:00 PM

These lambda log retentions are going to be the death of me, they cause a lot of issues when trying to deploy. Normally I get some conflict issue but now I am getting rate exceeded errors too. Anyone have any solutions for these?

Garret Harp

10/21/2021, 6:15 PM

currently always getting rate exceeded errors so its impossible to deploy because it always rolls back

Frank

10/21/2021, 6:32 PM

Hey @Garret Harp, can you give

logRetentionRetryOptions

a try? https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-lambda.Function.html#logretentionretryoptions

Frank

10/21/2021, 6:33 PM

Set

maxRetries

to 100 just to test it out and see if it works.

Frank

10/21/2021, 6:33 PM

Default is 3.

Frank

10/21/2021, 6:33 PM

Let me know and we can set the default to something larger.

Garret Harp

10/21/2021, 6:40 PM

Ok will try that out - is there a reason all the log retention things are even created? I feel like when I used serverless framework it did not create all of these resources or maybe they do just have the higher default so I never noticed any errors.

Garret Harp

10/21/2021, 6:54 PM

Looks like that just fixes the rate exceeded error now its just getting a conflict operation error.

Garret Harp

10/21/2021, 8:05 PM

Ok seems the custom log retention is just poorly implemented. I switched over to manually creating log groups using @aws-cdk/aws-logs and it deployed with 0 problems.

Garret Harp

10/21/2021, 8:06 PM

For anyone else having this issue I just add this to the end of my stacks:

Copy code

this.getAllFunctions().forEach(func => {
	new logs.LogGroup(this, `${func.node.id}-LogGroup`, {
		logGroupName: `/aws/lambda/${func.functionName}`,
		retention: logs.RetentionDays.THREE_MONTHS,
		removalPolicy: scope.defaultRemovalPolicy
	})
})

Ömer Toraman

10/21/2021, 9:36 PM

Thanks for sharing, this is very important. Otherwise no log group ever expires. We have tons of log groups that have been there for years

Ömer Toraman

10/21/2021, 9:37 PM

Though I don’t understand what you mean by rate exceeding in the case of log retention

Garret Harp

10/21/2021, 9:42 PM

I was using the AWS default "logRetention" setting on functions before and instead of creating just a log group like I do in my example it creates some custom log retention resource that was exceeding aws rate limits & causing some weird conflict errors on that stack.

Garret Harp

10/21/2021, 9:42 PM

(https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-lambda.FunctionOptions.html#logretention)

Frank

10/21/2021, 9:44 PM

@Garret Harp, can u still grab a screenshot of the conflict operation error you were getting?

Garret Harp

10/21/2021, 9:48 PM

Yeah I can. Once I got to maybe ~30 lambda functions every single deploy would error out with conflict or rate exceeded (though setting the retry count to 100 always fixed rate exceeded but never fixed conflicts) The message:

Received response status [FAILED] from custom resource. Message returned: A conflicting operation is currently in progress against this resource. Please try again. (RequestId: 27daaaf0-f4b5-472d-8cae-643d3664f740)

Frank

10/22/2021, 12:00 AM

Thanks @Garret Harp. I opened an issue here with two possible solutions, one suggested by you, and the other one being figuring out what’s causing the

conflicting operation

error and rewrite the custom resource ourselves. https://github.com/serverless-stack/serverless-stack/issues/953

Ömer Toraman

10/24/2021, 2:31 AM

@Garret Harp Hey, I also did what you suggested however, there is no connection between the created loggroup and the function. It creates an additional loggroup for the function as well

Ömer Toraman

10/24/2021, 2:35 AM

I might be doing something wrong:

Copy code

import * as sst from '@serverless-stack/resources'
import logs from '@aws-cdk/aws-logs'

export class MainStack extends sst.Stack {
      constructor(scope: <http://sst.App|sst.App>, id: string, props?: sst.StackProps) {
            super(scope, id, props);

            const api = new sst.Api(this, "Api", {
                  routes: {
                        "POST        /todo": {
                              srcPath: 'src/handlers/apiGateway/todo/',
                              handler: 'post-todo.handler',
                              functionName: 'postTodo',
                              description: 'Creates a new post'
                        }
                  },
            });

            this.getAllFunctions().forEach(func => {
                  const { stage, name } = scope
                  const logGroupNamePrefix = `${stage}-${name}-${id}`

                  new logs.LogGroup(this, `${func.node.id}-LogGroup`, {
                        logGroupName: `/aws/lambda/${logGroupNamePrefix}/${func.functionName}`,
                        retention: logs.RetentionDays.ONE_MONTH,
                        removalPolicy: scope.defaultRemovalPolicy
                  })
            })

            this.addOutputs({
                  ApiEndpoint: {
                        value: api.url,
                        description: 'The HTTP Endpoint'
                  }
            });
      }
}

And this in return creates two log groups:

/aws/lambda/post*Todo*

and

/aws/lambda/test-app-middy/post*Todo*

And the logs that come from the function are ingested by

/aws/lambda/postTodo

group.

/aws/lambda/test-app-middy/postTodo

, which is the one I’m manually creating has no stream.

Garret Harp

10/24/2021, 5:57 PM

As far as I know its not possible to add a prefix to the log group name it has to be

/aws/lambda/{functionName}

Ömer Toraman

10/24/2021, 10:17 PM

I see

Open in Slack

Previous Next