Apache Pinot

I did some test in my env and it would move on if the message cannot be converted to a json object

With avro though the segment gets stuck in offline state

I believe until the corrupt message is purged from kafka logs

I'll look at the bullet points you highlighted in the parent issue and address them accordingly in a separate PR

Subbu's proposal also makes sense for retryable decoding errors

yeah, possible. I dont know what value Pinot is adding there because the decoder has know thats its retryable vs not

and if it knows its retryable, it's better for the decoder to be self contained and just return a generic row or null

we can discuss more when you get to start on that.. for now, we are good

The value is uniform handling of exceptions from all streams. The amount of change in the code is small, I wouild strongly suggest to go with it.  <@UDRJ7G85T> what is the objection here?

I dont have any objection.. we were just brainstorming on long term solution

ok, got it. I think we can start with what Dan has now. I have asked for adding a small comment on the interface so that other streams can use this if they choose to.

Hi Team, How I can parse kafka header(in bytes) and filter based on specific field ? any code samples?

Apologies for taking this thread to this channel. We believe this is a bug and would like to raise a github issue after confirmation

We don't have upsert enabled for these tables

The table definition is in one of the comments

I think it can be reproduced in a unit test, I'll give it a try as well

I've raised this issue in github for the above <https://github.com/apache/pinot/issues/9091> we can always reject it if this is proven not to be a bug

we could also provide a fix or discuss a potential solution for this

this bug is because time filtering and tombstone filtering are conflated, `LLRealtimeSegmentDataManager` should react to time filtered empty batches differently to batches which only contain tombstones

the advance condition (i.e. line 4 in your description) should depend not on the unfiltered count but all messages having been tombstones

it should be easy to fix by just removing `getUnfilteredMessageCount` from `MessageBatch` and replacing it with `boolean allMessagesInvalid` or something to that effect, then track if all the messages are tombstones in the partition level consumer

unit testing `LLRealtimeSegmentDataManager` is going to be quite challenging

I know, I'm still willing to give it a try

there is a junit test for a fake sublcass and that may help with reproducing this issue