https://linen.dev logo
Join Slack
Powered by
Hi Team, our ingestion tasks are failing with the ...
# general
t
Hi Team, our ingestion tasks are failing with the below error. can someone please help to identify the root cause of this error.
Copy code
2024-06-06T08:50:06,191 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask - Encountered exception in parallel sub task.
org.apache.druid.java.util.common.ISE: Failed to add a row with timestamp[2023-08-11T12:32:07.000Z]
        at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask.generateAndPushSegments(SinglePhaseSubTask.java:469) ~[druid-indexing-service-29.0.0.jar:29.0.0]
        at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask.runTask(SinglePhaseSubTask.java:273) ~[druid-indexing-service-29.0.0.jar:29.0.0]
        at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:179) ~[druid-indexing-service-29.0.0.jar:29.0.0]
j
Do you have any tombstone segments on that datasource? I've seen a bug report with similar error that involves having tombstones present ...
t
what are tombstone segments ?
j
Tombstones are segments with no data in them. A normal segment must have at least one row of data. However in cases when you run a replace job, you are creating a new set of segments to "replace" an existing set of segments, e.g. in a compaction job where you are defragmenting your data segments. The original segments get thrown away in an indirect manner, by first being completely overshadowed by the new segments ... then coordinator recognizes that they are fully overshadowed and marks them as unused. In the case where you replaced a larger set of time intervals of data and the replacement data doesn't cover some intervals, then tombstones are used to cover those periods. Example -- let's say you are using DAY segment granularity and you have data for every day in January. Then you run a replace job for the entire month of January, but your replacement data does not have any data for Jan 12. The replace job would create a new tombstone segment for Jan 12 to hide the old data from Jan 12 that you don't way to see anymore.
t
Thanks for the explanation. But No, we dont have tombstone segments
Hi Team, can we get some solution for this issue please. we are ingesting multiple sub tasks in parallel and we get this error for all tasks with different segment date/time
Copy code
2024-06-21T09:18:19,517 WARN [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - Cannot allocate segment for timestamp[2024-05-17T05:20:28.000Z], sequenceName[index_parallel_146_event_summary_blbkiddk_2024-06-21T08:14:07.195Z_0].
2024-06-21T09:18:19,526 DEBUG [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Shutting down immediately...
2024-06-21T09:18:19,530 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask - Encountered exception in parallel sub task.
org.apache.druid.java.util.common.ISE: Failed to add a row with timestamp[2024-05-17T05:20:28.000Z]
j
Check your druid_pendingSegments table in the Meta DB ... might be orphanied records their blocking the segment allocation ...
Open in Slack
PreviousNext
Powered by