https://pinot.apache.org/ logo
Join Slack
Powered by
Has anyone faced the issue with lz4 compression? `...
# troubleshooting
s
Has anyone faced the issue with lz4 compression?
Copy code
net.jpountz.lz4.LZ4Exception: Malformed input at 13
        at net.jpountz.lz4.LZ4JavaUnsafeSafeDecompressor.decompress(LZ4JavaUnsafeSafeDecompressor.java:180) ~[lz4-java-1.7.1.jar:?]
        at net.jpountz.lz4.LZ4SafeDecompressor.decompress(LZ4SafeDecompressor.java:145) ~[lz4-java-1.7.1.jar:?]
        at org.apache.pinot.segment.local.io.compression.LZ4Decompressor.decompress(LZ4Decompressor.java:42) ~[pinot-segment-local-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f0
0e324e2dc]
        at org.apache.pinot.segment.local.segment.index.readers.forward.BaseChunkSVForwardIndexReader.decompressChunk(BaseChunkSVForwardIndexReader.java:137) ~[pinot-segment-local-0.10.0-dev-471.j
ar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.segment.local.segment.index.readers.forward.BaseChunkSVForwardIndexReader.getChunkBuffer(BaseChunkSVForwardIndexReader.java:118) ~[pinot-segment-local-0.10.0-dev-471.ja
r:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.segment.local.segment.index.readers.forward.VarByteChunkSVForwardIndexReader.getStringCompressed(VarByteChunkSVForwardIndexReader.java:72) ~[pinot-segment-local-0.10.0-
dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.segment.local.segment.index.readers.forward.VarByteChunkSVForwardIndexReader.getString(VarByteChunkSVForwardIndexReader.java:61) ~[pinot-segment-local-0.10.0-dev-471.ja
r:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.segment.local.segment.index.readers.forward.VarByteChunkSVForwardIndexReader.getString(VarByteChunkSVForwardIndexReader.java:35) ~[pinot-segment-local-0.10.0-dev-471.ja
r:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.dociditerators.SVScanDocIdIterator$StringMatcher.doesValueMatch(SVScanDocIdIterator.java:176) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4
bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.dociditerators.SVScanDocIdIterator.applyAnd(SVScanDocIdIterator.java:88) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e3
24e2dc]
        at org.apache.pinot.core.operator.docidsets.AndDocIdSet.iterator(AndDocIdSet.java:128) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.DocIdSetOperator.getNextBlock(DocIdSetOperator.java:67) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.DocIdSetOperator.getNextBlock(DocIdSetOperator.java:38) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:49) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.ProjectionOperator.getNextBlock(ProjectionOperator.java:61) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.ProjectionOperator.getNextBlock(ProjectionOperator.java:33) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:49) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.transform.PassThroughTransformOperator.getNextBlock(PassThroughTransformOperator.java:48) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3f
ecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.transform.PassThroughTransformOperator.getNextBlock(PassThroughTransformOperator.java:31) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3f
ecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:49) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.query.AggregationGroupByOrderByOperator.getNextBlock(AggregationGroupByOrderByOperator.java:107) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.query.AggregationGroupByOrderByOperator.getNextBlock(AggregationGroupByOrderByOperator.java:46) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:49) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.combine.GroupByOrderByCombineOperator.processSegments(GroupByOrderByCombineOperator.java:137) ~[pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.operator.combine.BaseCombineOperator$1.runJob(BaseCombineOperator.java:100) [pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at org.apache.pinot.core.util.trace.TraceRunnable.run(TraceRunnable.java:40) [pinot-core-0.10.0-dev-471.jar:0.10.0-dev-471-91c2ebbf297c4bf3fecb5f98413e9f00e324e2dc]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) [guava-30.1.1-jre.jar:?]
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) [guava-30.1.1-jre.jar:?]
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) [guava-30.1.1-jre.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
h
@Jackie ^
r
No, could this be related to the overflow reported here though? https://github.com/apache/pinot/issues/8701
the LZ4 library isn't going to be at fault here, it's used by e.g. elasticsearch and has been deployed to hundreds of thousands of servers, so I can only imagine this happening if the offsets were corrupted, which would correlate with the overflow reported above
i.e. I think this is a symptom of overflow, LZ4 is not a causal factor here; Snappy would have failed the same way (modulo the sizes of compressed chunks being different in general)
s
@Richard Startin yes the above issue was related to the issue that we filed. Thanks for the quick PR!
Open in Slack
PreviousNext
Powered by