I have a realtime table ingesting from Kafka and an application that writes events to the appropriate Kafka topic. My table originally had

realtime.segment.flush.threshold.rows=30

, and I ran my application to the point where I had pushed around 100k rows to a kafka topic before realizing that this was much too small a segment size. I stopped my app, deleted the table, changed

realtime.segment.flush.threshold.rows=100000

, and recreated it. Then, I ran my app to push 3mil rows to the kafka topic. At some point in Pinot's ingestion process, the status of my table changed to

BAD

. I looked in the controller logs and noticed this error:

2021/09/02 16:02:48.585 ERROR [SegmentCompletionFSM_simplejson__0__632__20210902T1602Z] [grizzly-http-server-21] Caught exception while committing segment metadata for segment: simplejson__0__632__20210902T1602Z

java.lang.IllegalStateException: Failed to find IdealState for table: simplejson_REALTIME

In the web UI for the Pinot Controller, under the Cluster Manager for the affected table, I sorted the

SEGMENTS

list by Status and noticed that I had two

Bad

segments. Inspecting the bad segments, I noticed that each had a total of 30 documents. I checked a handful of

Good

segments, and each had 100k documents. I'm not sure how to bring these segments into a

Good

state, or why they entered into a

Bad

state in the first place. I was unable to find anything in Pinot's documentation on what causes this error or how to resolve it.

The exception posted here seems a transient ZK issue.

How can I do that?

If it is okay, can you try deleting the table again and let's check if the segments are cleaned up properly

I deleted the table and

PinotCluster/PropertyStore/Segments

is reported empty from the zookeeper browser

How about the IdealState? It should also be empty

PinotCluster/IdealStates has two entries: • brokerResource • leadControllerResource I'm not sure what the entries should have inside them. Forgive my ignorance 😅

That is normal. The entry for the table

simplejson__REALTIME

is removed

Yes it is

After recreating the table, the idealstate should show up with the initial consuming segments

That is right. I see that now.

In the table config, did you set the

"stream.kafka.consumer.prop.auto.offset.reset"

to be

"smallest"

?

Ah ha, I did not. I assume the default is

largest