https://pinot.apache.org/ logo
Join Slack
Powered by
Hi All, I ran into a bit of a concerning issue du...
# troubleshooting
j
Hi All, I ran into a bit of a concerning issue during my POC. I was in the process of creating a brand new table (no segments yet). I created all the segments and validated that they were created properly (no errors and all segments were created). I then started ingesting them to the server and I had some of them ~1% get stuck in a 
bad
state. I tried reloading the segments, but they remained in the 
bad
state. I'm assuming that I'll need to recompute those segments. What made matters worse is that the bad segments made it such that the table was completely unusable. My questions are: 1. Is it expected that loading a bad segment into a table would make the table unusable? Or is this just because I was initializing it, so it never reached a 
good
state? My hope is that a bad segment would simply be cutoff, allowing for queries to still run just not over the bad segments. 2. Is there a way to be certain that an offline segment is "good" and that when it's loaded the table will remain in the 
good
state (before loading, during load, and after load)? If not, then it seems like the only way to work around this is to have a dummy table that's used to first validate the segment, before loading it into the table; otherwise, the issue in (1) would result in an otherwise healthy table becoming unusable.
Stacktrace:
Copy code
2022/02/17 01:53:29.747 ERROR [SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Caught exception in state transition from OFFLINE -> ONLINE for resource: poc_OFFLINE, partition: 13_23
java.io.IOException: Failed to retrieve file descriptor of /data/scratch/pinot/server/index/poc_OFFLINE/13_23/v3/star_tree_index: Unable to make field private int java.io.FileDescriptor.fd accessible: module java.base does not "opens <http://java.io|java.io>" to unnamed module @3e07d849
        at xerial.larray.mmap.MMapBuffer.<init>(MMapBuffer.java:73) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.segment.spi.memory.PinotNativeOrderLBuffer.mapFile(PinotNativeOrderLBuffer.java:49) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.segment.spi.memory.PinotDataBuffer.mapFile(PinotDataBuffer.java:194) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.segment.local.startree.v2.store.StarTreeIndexContainer.<init>(StarTreeIndexContainer.java:53) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:167) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:89) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.core.data.manager.offline.OfflineTableDataManager.addSegment(OfflineTableDataManager.java:52) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:372) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:318) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:162) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
        at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
2022/02/17 01:53:29.748 ERROR [HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Exception while executing a state transition task 13_23
java.lang.reflect.InvocationTargetException: null
        at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
        at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.io.IOException: Failed to retrieve file descriptor of /data/scratch/pinot/server/index/poc_OFFLINE/13_23/v3/star_tree_index: Unable to make field private int java.io.FileDescriptor.fd accessible: module java.base does not "opens <http://java.io|java.io>" to unnamed module @3e07d849
        at xerial.larray.mmap.MMapBuffer.<init>(MMapBuffer.java:73) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.segment.spi.memory.PinotNativeOrderLBuffer.mapFile(PinotNativeOrderLBuffer.java:49) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.segment.spi.memory.PinotDataBuffer.mapFile(PinotDataBuffer.java:194) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.segment.local.startree.v2.store.StarTreeIndexContainer.<init>(StarTreeIndexContainer.java:53) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:167) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:89) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.core.data.manager.offline.OfflineTableDataManager.addSegment(OfflineTableDataManager.java:52) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:372) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:318) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:162) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
        ... 11 more
k
1. bad segments will be skipped during query.. few bad segments should not make the table unusable. 2. once it goes to good state, it will never get into bad state. A segment typically goes into bad state during load time
👍 1
r
could you also share some error log when you mention the table is unusable? what query did you run and what error message did you get when you try to query this table? as kishore mentioned bad segments are skipped during query so it might've been something else that causes the query to fail
k
looking at the exception, looks like the segment was corrupted during download
try forcereload for that segment so that it downloads it again
r
hi @James Mnatzaganian which java version are you running?
Copy code
java.io.IOException: Failed to retrieve file descriptor of /data/scratch/pinot/server/index/poc_OFFLINE/13_23/v3/star_tree_index: Unable to make field private int java.io.FileDescriptor.fd accessible: module java.base does not "opens <http://java.io|java.io>" to unnamed module @3e07d849
        at xerial.larray.mmap.MMapBuffer.<init>(MMapBuffer.java:73) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
loks like you might be on JDK16+?
j
Thanks everyone! This is good to know. I'm suspecting then that because the table never entered a good state, the bad segments caused it to be in the bad state. The stack trace was from the server. All queries failed, even a simple 
select * from ... limit 1
. I worked around this by simply deleting the segments. I was running out of time, so I didn't try anything more than a simple refresh and delete. I'm using OpenJDK 17.0.2. --- Note that my inquiry was primarily focused on what the expected behavior is and less of how to fix this issue. It seems like if the table had been in a good state, then the bad segments wouldn't have been an issue, and in this specific instance it was presumably a bad download.
r
ok that explains it
we don't support JDK17. You have two options: 1. be the guinea pig and figure out which modules need to be added with --add-opens 2. downgrade to JDK11
j
ah, that's good to know. I see now that the docs say JDK11+, but not JDK16. The version of JDK that I used didn't matter (this was an isolated setup), I just picked latest thinking 11+ is fine. Thanks for the help! It's good to know this was potentially just an incompatibility issue. It's odd that almost all of the segments worked, but this could potentially explain a lot of the issues I've been running into with the batch ingestion process (it's been flaky and slow).
r
I am very keen to get pinot working on JDK17 but it needs prioritisation, if you go further with pinot and want to use JDK17 it's possible we can partner up on making it work
👍 1
j
Right now I wanted to see if Pinot made sense for my use case. I created a clean environment for this, so it's unfortunate (completely my fault 🤦‍♂️) that I used an unsupported version of the JDK. If this ever makes it to prod, then we'd be running it in k8s and would still be flexible on which version of the JDK we used (likely building off the provided helm charts). It's interesting that it seemed to work, but this could explain the roughness I was encountering. The good news is that I was at least able to get data loaded in and still validate that once the data is there (and has been indexed properly) it will work. The bad news is that I'm out of time for now, so I can't go back and see if my batch ingestion problems were a result of versioning or something else. Ultimately, I'd like to use Spark for this pipeline, but ran into lots of issues there too (see this).
r
it will work until it does an illegal reflective access
it would just warn JDK9-15
j
Ah, that's good to know. I think that explains some of the other issues.
Open in Slack
PreviousNext
Powered by