We have a particular use case where we wanted to ingest some data to Pinot in realtime tables using kafka . However we wanted to get the batch id and count from the Kafka and guarantee that the count matches the number of records in the pinot tables If it matches , then we want to updated the kafka with a flag saying the given batch id is processed . Is there a smart way to do this ? We have a lame way of running another flink job to do the count and send message to kafka but if their is a better idea, we will welcome it .

You've found the best way. Kafka doesn't have the ability to count the number of messages, so externally (for example in flink) is the only way.