does pinot handle kafka transactions and ignore uncommitted messages? we're setting

"stream.kafka.consumer.prop.auto.isolation.level": "read_committed",

in our realtime table, but it also seems like it is processing uncommitted messages

we use the open source kafka consumer and pass the properties you specify in table config to the underlying consumer

got it. are you aware of anyone in the community that is using

read_committed

successfully?

no, we did not use it at LinkedIn either

😞

I think the config should be

"stream.kafka.consumer.prop.isolation.level"

thanks Xiang!

no

our config is:

"stream.kafka.consumer.prop.auto.isolation.level": "read_committed",
        "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
        "stream.kafka.consumer.prop.group.id": "enriched_orders_e5ad3940-f312-40ee-882f-4e4c979f6813",
        "stream.kafka.consumer.prop.client.id": "d85b6d0b-c75f-4179-bc70-32be1a7c5a85",

everything after prop. will be chopped and send to consumer config

so yea, looks like we copied

auto.offset.reset

. that was going to be my next question, should we drop

auto

from that, but sounds like no

no

got it

this is exactly the kafka consumer config

thank you so much @Xiang Fu!

I assume you are using llc. In LLC, we don't commit kafka offsets. Instead,it is a pull model where we pull from the offsets as needed within a single partition

this transaction concept is within kafka with kafka producer