What happens if I create 2 tables (different scheme/table def) to consume events from the same kafka topic? Will both tables get all events, or will the events be consumed once per pinot deployment, rather than once per table?

both tables will get all evnets

great, that is what i had expected to happen

do you see any errors on the segments? Any messages in the server logs?

The segments report Status = 'Good' (green), if i click the segment some are 'Consuming', some are 'Offline' (although the server itself is running)

I dunno why it would have stopped consuming, but you can try to kickstart it by calling the pause and then resume APIs. https://startree.ai/blog/apache-pinot-0-11-pausing-real-time-ingestion

ok, thats handy to know about however, ill try that next time

When segments are OFFLINE it means the server had some difficulty consuming data. Could be decoding error, could be other exceptions. These will be logged on the server side. Pinot will attempt to restart consumption every once in while (probably that is why you are seeing GREEN).

@Simon J you can also try the

debug

endpoint to see if there are any consumption failures.

@Subbu Subramaniam Perhaps it should be orange if it is having some troubles? Green is a bit misleading as it means that you don't delve deeper to investigate

@Simon J perhaps the color needs to be fixed, but can you first confirm that this is the case? I think you had mentioned that there were no logs in the servers, so I want to first find out what the problem is

@Subbu Subramaniam There were logs on the servers themselves, unfortunately I only have access to the logs in ELK stack however as I do not run the pinot installation. In this instance however unfortunately the file beat (log tailer) had crashed so was not shipping the logs i needed to ELK. When someone looked on the box we learnt that segment had stopped consuming. The debug endpoint (which I learnt about since then) has been helpful as I can see that I have a NumberFormatException for a timestamp string that is trying to parse into a Long. Strange as I don't recall anyone changing the data type of a field, however at this stage it looks that has happened. Is there a way to determine the kafka offset that causes consumption to stop? That way I could easily find the event to see what the cause is

Hmm... Not sure. I think the log message prints the kafka offset that causes it to stop, but I could be wrong. Also, you mention that you dont have a way to look at logs . Is that permanent, or a temporary thing (because of the crash you mention). If temporary, then you can wait until the next time the segment tries to consume, and look for the log i there. Alternatively, you may need a Kafka consumer (in LinkedIN we have a command line tool ) that can hep you here -- but this iwll work only if you have some reasonable number of messages. You can be searching forever if your kafka partitions produce messages at 50k/sec 🙂

I hope that the file beat crashing is not a recurring event... Will the segment auto retry after stopping like this, or could I just force it to re-try the same event by enable/disable the table? (we produce around 1k events/s)

It will keep retrying, and (afaik) if the offset it tries has expired in Kafka, then it will move forward to a more recent one.

Yes, I was wondering about how you go about recovering from the situation where a Pinot cannot process a particular record on the the kafka stream (maybe due to a bug the has caused it to be corrupt), and whether you can tell Pinot to move on an ignore that record.

As of now, we don't have any such mechanism.