This message was deleted.

Hi @sriramdas sivasai, This is a tricky one to debug as there is not much info in the log. A few ideas: • Are you able to see preview data for this stream in the UI? (I would use the UI flow to see the preview and verify that it looks good) • What is different about these streams and the ones that are successful? ◦ Do these couple streams that are failing have a different data format from the others? (Parquet instead of CSV for ex.) ◦ Are you using special datatypes? such as sketches in these streams and not in the ones that complete? ◦ Is the ingestion spec correctly calling the right files in the file structure in the source system? You may have to investigate the data in the streams as well as the file structure in your source.

thanks @Vijeth Sagar . we observed this is happening for all the streams that we have. I have checked the preview data for the stream. It seems to be showing up correctly. these source were processing the data correct since 3 months i would say. As its happening with all the datasources, im suspecting it could be because of some platform issues. im still try to debug the issue. i would request any one, if they seen this issue and mitigate it before, kindly let us know.

@sriramdas sivasai Below snippet makes me think there is some issue with the sequence number

Preconditions.checkNotNull

just after

seekablestream.SequenceMetadata

check

2023-10-30T13:19:21,192 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception while running task.
java.lang.NullPointerException: null
  at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.1.jar:?]
  at org.apache.druid.indexing.seekablestream.SequenceMetadata.<init>(SequenceMetadata.java:81) ~[druid-indexing-service-27.0.0.jar:27.0.0]

Was their a hard stop/no data at source level and the offset stored in druid is deleted now. If its ok, We could try a HARD RESET and then resume supervisor.

thanks @Bibek Sahoo ill check and update you. Hoping it will help