A rather noob question about ingesting into Pinot from kafka. If the upstream (kafka) has bad data that’s causing ingestion to fail, does Pinot ignore the record and continue ingesting the remaining records, or does ingestion get into a bad state and stop altogether?

it depends on the severity of the bad data.. if a field is bad, it will use default value but if we fail to decode the entire message payload, it will stop consumption. We recently added a flag to ignore that as well and move forward

Can you point me to the flag please?

If I'm not wrong, this is still work in progress. The PR should be raised shortly though

sorry my bad.. I thought it was merged.. @Dan DC do you need anything from us to push it?

Not at all, no worries. I'll try to push the code tomorrow and I'll ping you and post the PR on the dev channel to try to get it reviewed. Apologies for the delay

@Dan DC we use JSON. And thanks for the update.

For invalid JSON payloads the decoder should just ignore the messages

Great, thank you!

No problem