👋 Hi team, I'm just trying to ideate around implementing a write-behind log for Pinot realtime ingestion. I wanted to see if there is any standard way of sending rows to WBL that has been ingested into real time table via Kafka to ensure downstreams that the row has been ingested.

Pinot already tracks the latest consumed Kafka offset per partition in segment ZK metadata. Downstream can poll the Pinot controller API (/segments/{table}/metadata) or read from ZK directly to check how far consumption has progressed. If the downstream knows which offset its record landed on, it can confirm ingestion by comparing. The tradeoff is this is partition-granularity, not row-granularity.

Gotcha that is great to know. I'm thinking of creating a separate kafka topic where downstreams can listen to the rows with confirmed ingestion. Maybe I can work with this api to do periodic polling to send rows between the offsets.

The above api is for committed segments. For consuming segments you can use

/tables/{tableName}/consumingSegmentsInfo

where each replica shows where they are in the consuming offset.

Thank you that is also great to know. Our use case is to get as closet to the stack to notify downstreams that the row has been ingested. As long as they are eventually consistent, I think we are 🟢 .