☝️ those are the two repetitive exception I'm getting from the logs.

seems not fatel though from the code 🤔

I dont think your consumer is moving forward at all. Do you see any logs from this line at all? https://github.com/apache/incubator-pinot/blob/caef8f6941250f6723c0a6f847c66122876e1ee8/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java#L1163

@User ^^

we did run into such issues in our environment.

@User ^^

@User is that a Kafka bug (that if the msg size is too high it will not consume anything)?

wait, having a small consume buffer size is not a bug. That is decided by the memory budget that the client has (otherwise, we could declare that msg size to be 60G and crash when such a msg is received). The question is, can pinot get any indication from kafka if this condition happens, so that the consumer can throw exception and stop (instead of a silent discard). I don't think so..

so Pinot can retrieve the max msg size and use it?

not sure if we can get that config. I think the PR does a good job for now

it changed the default to a some reasonable value and allowed the used to set this on a per topic basis

trying to get max message from broker and setting it is an overkill, more over coming up with right value automatically is not trivial

It would be great if we can log that the message is not consumed due to too large message size

so that people can be guided to bump up the config above

@User exactly my q. If we can recognize it, we should raise an exception and stop consumption

i think that we should still not stop consumption though

no, we should stop consumption. We will re-consume from the same point once the serers are restarted with a different config

I dont think it says that. it silently fails (atleast from the description by James)

we don't want to miss a msg

@User we don’t want to stop the service too frequently due to one large message

hmm it would be a trade off between data completeness vs system resiliency

or at least give the user an option to choose between

We can give an option to skip the message, of course -- but all this only if there is a mechanism to recognize whether there is a large message out there. I am onto something else at the moment, and don't have the time to investigate whether pinot can even get notified when this happens. @User since you guys are right there, can you check if there is way by which Kafka can provide this notification? If so, we can take actions like stop consumption or skip the message.

Silent dropping is not good.

@User @User the silent failing part is inside the SimpleConsumer

from what I can see it does not provide any notification - but we can check again

that seems like a useful thing to have

but we're still on old version of Kafka, might be useful to upgrade to the latest client and see if this problem exists

It will be good to understand what’s happening underneath