https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
d
Copy code
2022-08-24T03:01:31,084 WARN [[single_phase_sub_task_mytable_lmddempe_2022-08-23T23:23:41.205Z]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Failed to push merged index for segment[mytable_2022-08-21T00:00:00.000Z_2022-08-22T00:00:00.000Z_2022-08-23T19:57:06.595Z_29].
org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in line[mycolumn,,0,7956010,7956179]
    at org.apache.druid.java.util.common.io.smoosh.SmooshedFileMapper.load(SmooshedFileMapper.java:85) ~[druid-core-0.23.0.jar:0.23.0]
    at org.apache.druid.java.util.common.io.smoosh.Smoosh.map(Smoosh.java:32) ~[druid-core-0.23.0.jar:0.23.0]
    at org.apache.druid.segment.IndexIO$V9IndexLoader.load(IndexIO.java:546) ~[druid-processing-0.23.0.jar:0.23.0]
    at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:199) ~[druid-processing-0.23.0.jar:0.23.0]
    at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:189) ~[druid-processing-0.23.0.jar:0.23.0]
    at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:911) ~[druid-server-0.23.0.jar:0.23.0]
    at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:784) ~[druid-server-0.23.0.jar:0.23.0]
    at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) [guava-16.0.1.jar:?]
    at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) [guava-16.0.1.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
    at java.lang.Thread.run(Thread.java:833) [?:?]
2022-08-24T03:01:31,188 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask - Encountered exception in parallel sub task.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in line[mycolumn,,0,7956010,7956179]
    at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask.generateAndPushSegments(SinglePhaseSubTask.java:475) ~[druid-indexing-service-0.23.0.jar:0.23.0]
    at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask.runTask(SinglePhaseSubTask.java:261) [druid-indexing-service-0.23.0.jar:0.23.0]
    at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.run(AbstractBatchIndexTask.java:186) [druid-indexing-service-0.23.0.jar:0.23.0]
    at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) [druid-indexing-service-0.23.0.jar:0.23.0]
    at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) [druid-indexing-service-0.23.0.jar:0.23.0]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
    at java.lang.Thread.run(Thread.java:833) [?:?]
only happening to this 1 datasource.
g
it doesn't look like a data issue to me
the code there is trying to read the 
meta.smoosh
which is written as part of segment generation
oh
my guess is you have a column named 
mycolumn,
(with a 
,
literally as part of the column name)
apparently that is not allowed! since it's written into this csv style of format. we should validate this upfront when the data is first encountered you should be able to work around it by removing this column or by adding an exclusion for it (
dimensionExclusions
)
d
lol, ok thank you for the quick response!
g
we'll add some validation for this — probably skip those fields and mark the row as 'processed with errors' in the ingest metrics
d
wow! That 
mycolumn,
bug has been in the system for 3+ years! And it got exposed only after 0.23.0 upgrade. Good job, Druid!
5 Views
Open in Slack
PreviousNext
Powered by