https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
p
Can you please paste the full stack ?
k
Copy code
org.apache.druid.segment.loading.SegmentLoadingException: Exception loading segment[<datasource>_2023-05-20T00:00:00.000Z_2023-05-21T00:00:00.000Z_2023-05-23T12:09:11.679Z_1]
        at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:289) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:266) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:343) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.server.coordination.SegmentLoadDropHandler$1.lambda$addSegment$1(SegmentLoadDropHandler.java:572) ~[druid-server-25.0.0.jar:25.0.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_292]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_292]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]
Caused by: org.apache.druid.java.util.common.IAE: use read(ByteBuffer buffer, ObjectStrategy<T> strategy, SmooshedFileMapper fileMapper) to read version 2 indexed.
        at org.apache.druid.segment.data.GenericIndexed.read(GenericIndexed.java:185) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.query.aggregation.datasketches.tuple.ArrayOfDoublesSketchMergeComplexMetricSerde.deserializeColumn(ArrayOfDoublesSketchMergeComplexMetricSerde.java:71) ~[?:?]
        at org.apache.druid.segment.serde.ComplexMetricSerde.deserializeColumn(ComplexMetricSerde.java:60) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.serde.ComplexColumnPartSerde.lambda$getDeserializer$0(ComplexColumnPartSerde.java:89) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.column.ColumnDescriptor.read(ColumnDescriptor.java:111) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.IndexIO$V9IndexLoader.deserializeColumn(IndexIO.java:796) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.IndexIO$V9IndexLoader.registerColumnHolder(IndexIO.java:771) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.IndexIO$V9IndexLoader.registerColumnHolders(IndexIO.java:730) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.IndexIO$V9IndexLoader.load(IndexIO.java:654) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:202) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.loading.MMappedQueryableSegmentizerFactory.factorize(MMappedQueryableSegmentizerFactory.java:49) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.loading.SegmentLocalCacheLoader.getSegment(SegmentLocalCacheLoader.java:71) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.server.SegmentManager.getSegmentReference(SegmentManager.java:325) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.server.SegmentManager.loadSegment(SegmentManager.java:268) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:281) ~[druid-server-25.0.0.jar:25.0.0]
        ... 10 more
p
@Clint Wylie Do you have any idea about this ? We are getting the VERSION_TWO and throwing an exception from buffer while deserializing the theta sketch column
Copy code
final GenericIndexed<ArrayOfDoublesSketch> ge = GenericIndexed.read(buffer, ArrayOfDoublesSketchObjectStrategy.STRATEGY);
c
hmm, this is in ArrayOfDoubles serde i guess?
can look closer soonly
this might be a bug
but i haven’t looked yet
version 2 is for when the GenericIndexed is larger than java bytebuffer size (2gb)
it splits across multiple internal smoosh files
k
its happening for few segments not all
c
however, it takes different methods to read and stuff so if it isn’t actually wired up then you run into this explosion
p
yeah, we do not have smoosh file passed to the calling method read here.
c
i need to look closer at code to see if its setup to be able to handle v2 (or even should write it in the first place)
p
c
i guess 
LargeColumnSupportedComplexColumnSerializer
writes v2s, it does look wired up to potentialy use the v2 serializer
its a bug, it should be passing in the filemapper from the columnbuilder
should look like this
Copy code
@Override
  public void deserializeColumn(final ByteBuffer buffer, final ColumnBuilder builder)
  {
    final GenericIndexed<ArrayOfDoublesSketch> ge = GenericIndexed.read(buffer, ArrayOfDoublesSketchObjectStrategy.STRATEGY, builder.getFileMapper());
    builder.setComplexColumnSupplier(new ComplexColumnPartSupplier(getTypeName(), ge));
  }
👍 1
as a workaround you can probably reduce segment max row count
so it doesn’t spill into v2
well you’d have to reingest data though
so its not a very cool workaround
a bit surprised no one has run into this yet tbh
p
@Krishna can you please file a issue on with the details, we can take this up and fix it. Thank you
c
or if you’re motivated, feel free to submit a PR since the fix is that snippet pasted above
it looks like only ArrayOfDoubles sketch has the problem, the others are using the right method
k
Thanks @Pranav and @Clint Wylie for looking into this issue . never file a issue where should i do it
c
“Problem report”
good find 👍
p
yeah, this code change will be part of druid-datasketches extension and it should be easy to test it as well.
c
ish, the tricky part of testing is that this path is only hit when column is larger than 2gb, though iirc there are methods to make that size smaller
k
@Clint Wylie for now we will reingest data with less no.of rows per segment
c
rad, thanks
p
Open in Slack
PreviousNext
Powered by