This message was deleted.

cc @Sergio Ferragut @Gian Merlino

Looks like your endpoint is the issue...

I had no idea you can ship request logs like this. We did it the hard way by writing custom tailer lololol

• Adding a link for future ref: https://druid.apache.org/docs/latest/operations/request-logging.html

I was able to fix the

301: Moved Permanently

error. However, I am still getting this error

Event too large to emit

continuously.

2022-11-11T08:43:43,806 ERROR [DruidSchema-Cache-0] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Event too large to emit (1,559,214 > 1,047,552): {"feed":"druidRequestLogFeed","timestamp":"2022-11-11T08:43:41.200Z","service":"druid-blue/broker","remoteAddr":"","sql":null,"queryStats":{"query/time":2586,"query/bytes":-1,"success":true,"identity":"allowAll"},"query":{"queryType":"segmentMetadata","dataSource":{"type":"table","name":"hello"},"intervals":{"type":"segments","segments":[{"itvl":"2022-01-26T00:00:00.000Z/2022-01-26T01:00:00.000Z","ver":"2022-01-29T15:43:23.133Z","part":0},{"itvl":"2022-01-26T00:00:00.000Z/2022-01-26T01:00:00.000Z","ver":"2022-01-29T15:43:23.133Z","part":1},{"itvl":"2022-01-26T00:00:00.000Z/2022-01-26T01:00:00.000Z","ver":"2022-01-29T15:43:23.133Z","part":2},{"itvl":"2022-01-26T00:00:00.000Z/2022-01-26T01:00:00.000Z","ver":"2022-01-29T15:43:23.133Z","part":3},{"itvl":"2022-01-26T00:00:00.000Z/2022-01-26T01:00:00.000Z","ver":"2022-01-29T15:43:23.133Z","part":4},{"itvl":"2022-01-26T00:00:00.000Z/2022-01-26T01:00:00.000Z","ver":"2022-01-29T15:43:23.133Z","part":5},{"itvl":"2022-01-26T00:00:00.000Z/2022-01-26T01:00:00.000Z","ver":" ...

Any pointers how to solve this issue?

Based on the codebase, it looks like you can control this by setting maxBatchSize. See:

class BaseHttpEmittingConfig

druid.emitter.http.maxBatchSize=2095104

Tried to increase this setting to 2 MB. However, this didn't solve the

Event too large to emit

problem

I think you supposed to lower it so that it fits

@Didip Kerabat

2022-11-11T08:43:43,806 ERROR [DruidSchema-Cache-0] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Event too large to emit (1,559,214 > 1,047,552):

Here, Isn't Druid sending events of size 1,559,214 bytes but max permitted is only 1,047,552. Didn't understand why lower would solve the problem?

Yes. It seems like MAX_EVENT_SIZE is fixed to 1MB making it a hard limit for a single event regardless of the maxBatchSize setting.

Then in that case, is there any way to overcome such

Event too large to emit

issues?

I've been thinking about that, I'm not sure.

Sounds like you have to do it the hard way by writing custom tailer.