https://pinot.apache.org/ logo
Join Slack
Powered by
Hi All, had a few questions about using `Pinot man...
# troubleshooting
k
Hi All, had a few questions about using 
Pinot managed offline flows
. Any help would be greatly appreciated! 1. Does the OFFLINE table config need to have the 
RealtimeToOfflineSegmentsTask
match the one added to the REALTIME table config? 2. I'm seeing this 
TASK_ERROR to DROPPED
in the minion log. What does this signify?
Copy code
20 START:INVOKE /PinotCluster/INSTANCES/Minion_172.19.0.6_9514/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor@157c6932 type: CALLBACK
Resubscribe change listener to path: /PinotCluster/INSTANCES/Minion_172.19.0.6_9514/MESSAGES, for listener: org.apache.helix.messaging.handling.HelixTaskExecutor@157c6932, watchChild: false
Subscribing changes listener to path: /PinotCluster/INSTANCES/Minion_172.19.0.6_9514/MESSAGES, type: CALLBACK, listener: org.apache.helix.messaging.handling.HelixTaskExecutor@157c6932
Subscribing child change listener to path:/PinotCluster/INSTANCES/Minion_172.19.0.6_9514/MESSAGES
Subscribing to path:/PinotCluster/INSTANCES/Minion_172.19.0.6_9514/MESSAGES took:0
The latency of message 6a8ac921-3913-43e8-a777-b15c16185245 is 7 ms
Scheduling message 6a8ac921-3913-43e8-a777-b15c16185245: TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1636993325945:TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1636993325945_0, TASK_ERROR->DROPPED
Submit task: 6a8ac921-3913-43e8-a777-b15c16185245 to pool: java.util.concurrent.ThreadPoolExecutor@67024f54[Running, pool size = 40, active threads = 0, queued tasks = 0, completed tasks = 221]
Message: 6a8ac921-3913-43e8-a777-b15c16185245 handling task scheduled
20 END:INVOKE /PinotCluster/INSTANCES/Minion_172.19.0.6_9514/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor@157c6932 type: CALLBACK Took: 8ms
handling task: 6a8ac921-3913-43e8-a777-b15c16185245 begin, at: 1636993355435
handling message: 6a8ac921-3913-43e8-a777-b15c16185245 transit TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1636993325945.TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1636993325945_0|[] from:TASK_ERROR to:DROPPED, relayedFrom: null
Merging with delta list, recordId = TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1636993325945 other:TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1636993325945
Instance Minion_172.19.0.6_9514, partition TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1636993325945_0 received state transition from TASK_ERROR to DROPPED on session 1005c465f540008, message id: 6a8ac921-3913-43e8-a777-b15c16185245
Merging with delta list, recordId = TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1636993325945 other:TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1636993325945
Removed /PinotCluster/INSTANCES/Minion_172.19.0.6_9514/CURRENTSTATES/1005c465f540008/TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1636993325945
Message 6a8ac921-3913-43e8-a777-b15c16185245 completed.
Delete message 6a8ac921-3913-43e8-a777-b15c16185245 from zk!
message finished: 6a8ac921-3913-43e8-a777-b15c16185245, took 14
Message: 6a8ac921-3913-43e8-a777-b15c16185245 (parent: null) handling task for TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1636993325945:TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1636993325945_0 completed at: 1636993355449, results: true. FrameworkTime: 1 ms; HandlerTime: 13 ms.
Subscribing changes listener to path: /PinotCluster/INSTANCES/Minion_172.19.0.6_9514/MESSAGES, type: CALLBACK, listener: org.apache.helix.messaging.handling.HelixTaskExecutor@157c6932
Subscribing child change listener to path:/PinotCluster/INSTANCES/Minion_172.19.0.6_9514/MESSAGES
Subscribing to path:/PinotCluster/INSTANCES/Minion_172.19.0.6_9514/MESSAGES took:0
3. The tasks/scheduler/information API endpoint returns "Task scheduler is disabled". I've added entry to controller config 
"controller.task.frequencyInSeconds": 3600
is there some other setting I need to configure? 4. The tasks/task/taskname/state is giving a 
500 Index 1 out of bounds for length 1"
but tasks/tasktype/taskstates shows completed. I'm not seeing any segments added to my OFFLINE table though. Any idea on what's missing?
n
1. No need to set anything in offline table 2. Looks like the task had some exceptions, there should be some more logs about why the task failed and went into TASK_ERROR state (and then from TASK_ERROR to DROPPED). Any exception/error logs from before what you’ve pasted? 3. Not sure why is says disabled. As long as you’re seeing controller create tasks from the logs, and monion pick up the tasks, you’re good. If you’re not seeing that, try adding this 
controller.task.scheduler.enabled: true
to the controller config 4. lets see more logs from controller/minion?
k
1 ✅ 2 - I didn't go far back on the log. There is exception an exception
Copy code
Caught exception while fetching segment from: <s3://bucket/pinot-data/pinot-s3-example-docker/controller-data//profiles/profiles__0__0__20211112T1935Z> to: /tmp/PinotMinion/data/RealtimeToOfflineSegmentsTask/tmp-106dfc56-8986-48a1-98cb-c97c7b2bc767/tarredSegmentFile_0
java.lang.IllegalStateException: PinotFS for scheme: s3 has not been initialized
I am using S3 for deep storage. I do see segments being written there. I'm guessing I need to pass in 
env
values for access key and secret key? 3/4 - Maybe fixing the above will fix these? @Neha Pawar
n
ah.. you need to add deep store properties to your minion components. you must’ve added some deep store configs to controller/server?
k
Yes, I added the configs to controller/server and the segments are being written to S3. Are minion specific configs documented?
n
no.. let me add it
would love some detailed feedback about the docs from you after this 😛
k
Great! Thank you Neha!
For sure 🙂
I'm about to watch your presentation on this topic from back in July!
🙂 1
n
added for all the FS. it’s the same as server/controller, except the prefix is shorter
k
Great! Thank you!
In comparing the config for controller, my working version I had to add the following:
Copy code
controller.helix.cluster.name=PinotCluster
controller.zk.str=pinot-zookeeper:2181
controller.host=
controller.port=9000
Which is referenced in the tutorial https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot
For server.conf the following ( present in tutorial)
Copy code
pinot.server.netty.port=8098
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/tmp/pinot-tmp/server/index
pinot.server.instance.segmentTarDir=/tmp/pinot-tmp/server/segmentTars
So restarted Minion and that resolved the S3 error
However seeing errors on controller/server and OFFLINE table being in bad status.
Server log excerpt
Copy code
2021/11/15 20:59:35.301 ERROR [SegmentFetcherAndLoader] [HelixTaskExecutor-message_handle_thread] Attempts exceeded when downloading segment: consolidations_1550859587018_1550861765735_0 for table: consolidations_OFFLINE from: <s3://bucket/pinot-data/pinot-s3-example-docker/controller-data//consolidations/consolidations_1550859587018_1550861765735_0> to: /tmp/PinotServer/segmentTar/consolidations_OFFLINE/tmp-consolidations_1550859587018_1550861765735_0-fb7cfa39-13a5-4f3e-813b-f4e36a505290/consolidations_1550859587018_1550861765735_0.tar.gz
2021/11/15 20:59:35.302 ERROR [SegmentFetcherAndLoader] [HelixTaskExecutor-message_handle_thread] Cannot load segment : consolidations_1550859587018_1550861765735_0 for table consolidations_OFFLINE
org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3 attempts
	at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:61) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:72) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:146) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:141) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.server.starter.helix.SegmentFetcherAndLoader.downloadSegmentToLocal(SegmentFetcherAndLoader.java:198) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.server.starter.helix.SegmentFetcherAndLoader.addOrReplaceOfflineSegment(SegmentFetcherAndLoader.java:154) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:166) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2021/11/15 20:59:35.302 ERROR [SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel] [HelixTaskExecutor-message_handle_thread] Caught exception in state transition from OFFLINE -> ONLINE for resource: consolidations_OFFLINE, partition: consolidations_1550859587018_1550861765735_0
org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3 attempts
	at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:61) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:72) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:146) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:141) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.server.starter.helix.SegmentFetcherAndLoader.downloadSegmentToLocal(SegmentFetcherAndLoader.java:198) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.server.starter.helix.SegmentFetcherAndLoader.addOrReplaceOfflineSegment(SegmentFetcherAndLoader.java:154) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:166) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2021/11/15 20:59:35.303 ERROR [HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_thread] Exception while executing a state transition task consolidations_1550859587018_1550861765735_0
java.lang.reflect.InvocationTargetException: null
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3 attempts
	at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:61) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:72) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:146) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:141) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.server.starter.helix.SegmentFetcherAndLoader.downloadSegmentToLocal(SegmentFetcherAndLoader.java:198) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.server.starter.helix.SegmentFetcherAndLoader.addOrReplaceOfflineSegment(SegmentFetcherAndLoader.java:154) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:166) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
	... 12 more
2021/11/15 20:59:35.312 ERROR [StateModel] [HelixTaskExecutor-message_handle_thread] Default rollback method invoked on error. Error Code: ERROR
I see a new segment on S3 though.
controller error:
Copy code
2021/11/15 20:59:27.429 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(1ee314dc_DEFAULT)] Event 1ee314dc_DEFAULT : Unable to find a next state for resource: profiles_OFFLINE partition: profiles_1413387486771_1413405745431_0 from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:ONLINE
2021/11/15 20:59:27.448 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(9f672776_DEFAULT)] Event 9f672776_DEFAULT : Unable to find a next state for resource: profiles_OFFLINE partition: profiles_1413387486771_1413405745431_0 from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:ONLINE
2021/11/15 20:59:35.340 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(2bdf94fa_DEFAULT)] Event 2bdf94fa_DEFAULT : Unable to find a next state for resource: consolidations_OFFLINE partition: consolidations_1550859587018_1550861765735_0 from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:ONLINE
2021/11/15 20:59:35.340 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(2bdf94fa_DEFAULT)] Event 2bdf94fa_DEFAULT : Unable to find a next state for resource: profiles_OFFLINE partition: profiles_1413387486771_1413405745431_0 from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:ONLINE
2021/11/15 20:59:35.362 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(a26a1dc5_DEFAULT)] Event a26a1dc5_DEFAULT : Unable to find a next state for resource: consolidations_OFFLINE partition: consolidations_1550859587018_1550861765735_0 from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:ONLINE
2021/11/15 20:59:35.363 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(a26a1dc5_DEFAULT)] Event a26a1dc5_DEFAULT : Unable to find a next state for resource: profiles_OFFLINE partition: profiles_1413387486771_1413405745431_0 from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:ONLINE
2021/11/15 20:59:35.378 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(76e13678_DEFAULT)] Event 76e13678_DEFAULT : Unable to find a next state for resource: consolidations_OFFLINE partition: consolidations_1550859587018_1550861765735_0 from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:ONLINE
2021/11/15 20:59:35.378 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(76e13678_DEFAULT)] Event 76e13678_DEFAULT : Unable to find a next state for resource: profiles_OFFLINE partition: profiles_1413387486771_1413405745431_0 from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:ONLINE
2021/11/15 20:59:55.384 ERROR [CompletionServiceHelper] [grizzly-http-server-15] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 20:59:55.432 ERROR [CompletionServiceHelper] [grizzly-http-server-4] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:00:04.680 ERROR [CompletionServiceHelper] [grizzly-http-server-7] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:00:04.730 ERROR [CompletionServiceHelper] [grizzly-http-server-0] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:00:07.529 ERROR [CompletionServiceHelper] [grizzly-http-server-6] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:00:07.578 ERROR [CompletionServiceHelper] [grizzly-http-server-7] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:01:43.084 ERROR [CompletionServiceHelper] [grizzly-http-server-3] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:01:43.179 ERROR [CompletionServiceHelper] [grizzly-http-server-6] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:01:46.233 ERROR [CompletionServiceHelper] [grizzly-http-server-11] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:01:46.284 ERROR [CompletionServiceHelper] [grizzly-http-server-12] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:01:52.885 ERROR [CompletionServiceHelper] [grizzly-http-server-6] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:02:14.731 ERROR [CompletionServiceHelper] [grizzly-http-server-11] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:02:17.036 ERROR [CompletionServiceHelper] [grizzly-http-server-11] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:02:17.081 ERROR [CompletionServiceHelper] [grizzly-http-server-15] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:02:20.184 ERROR [CompletionServiceHelper] [grizzly-http-server-4] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:02:20.276 ERROR [CompletionServiceHelper] [grizzly-http-server-6] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:03:44.386 ERROR [CompletionServiceHelper] [grizzly-http-server-15] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:03:44.438 ERROR [CompletionServiceHelper] [grizzly-http-server-10] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:06:56.253 ERROR [CompletionServiceHelper] [grizzly-http-server-12] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:06:56.287 ERROR [CompletionServiceHelper] [grizzly-http-server-1] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:10:43.545 ERROR [CompletionServiceHelper] [grizzly-http-server-9] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:10:43.575 ERROR [CompletionServiceHelper] [grizzly-http-server-13] Server: Server_172.19.0.5_8098 returned error: 404
2021/11/15 21:12:16.096 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(d838849b_DEFAULT)] Event d838849b_DEFAULT : Unable to find a next state for resource: consolidations_OFFLINE partition: consolidations_1550859587018_1550861765735_0 from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:ONLINE
2021/11/15 21:12:16.097 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(d838849b_DEFAULT)] Event d838849b_DEFAULT : Unable to find a next state for resource: profiles_OFFLINE partition: profiles_1413387486771_1413405745431_0 from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:ONLINE
n
were any changes made to the controller/server side deep store configs?
k
Nope. I just added config to minion and restarted the docker container.
@Neha Pawar, any suggestions on how to resolve this?
n
adding deep store configs on minion shouldn’t have caused any issues on controller/server. Can you paste your minion/server/controller configs here?
adding @Xiang Fu @Mayank for more eyes
k
controller:
Copy code
pinot.role=controller
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=us-east-1
controller.data.dir=<s3://bucket/>
controller.local.temp.dir=/tmp/pinot-tmp-data/
controller.helix.cluster.name=PinotCluster
controller.zk.str=pinot-zookeeper:2181
controller.host=xxx.x.x.x (ec2 host IP)
controller.port=9000
controller.enable.split.commit=true
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
server:
Copy code
pinot.server.netty.port=8098
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/tmp/pinot-tmp/server/index
pinot.server.instance.segmentTarDir=/tmp/pinot-tmp/server/segmentTars


pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=us-east-1
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
minion:
Copy code
storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
storage.factory.s3.region=us-east-1
segment.fetcher.protocols=file,http,s3
segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
@Neha Pawar, @Mayank, @Xiang Fu
x
is s3 path
Copy code
<s3://bucket/pinot-data/pinot-s3-example-docker/controller-data//consolidations/consolidations_1550859587018_1550861765735_0>
accessible from the container
I assume you are using same credential for server and minion
is this path correct or it has file extension like .tar.gz
k
Same credentials.
I actually deleted the files to start over but the files did not have the .tar.gz extension.
So restarted everything and no luck. Still same errors.
The offline segment that's generated does not have the .tar.gz extension.
Used the debug endpoint and below is the status for the offline table @Xiang Fu
Copy code
[
  {
    "tableName": "consolidations_OFFLINE",
    "numSegments": 1,
    "numServers": 1,
    "numBrokers": 1,
    "segmentDebugInfos": [],
    "serverDebugInfos": [
      {
        "serverName": "Server_172.19.0.5_8098",
        "numMessages": 0,
        "errors": 1
      }
    ],
    "brokerDebugInfos": [],
    "tableSize": {
      "reportedSize": "-1 bytes",
      "estimatedSize": "-1 bytes"
    },
    "ingestionStatus": {
      "ingestionState": "UNKNOWN",
      "errorMessage": "Cannot retrieve ingestion status for Table : consolidations_OFFLINE since it does not use the built-in SegmentGenerationAndPushTask task"
    }
  }
]
@Neha Pawar, noticed your reply on this thread https://apache-pinot.slack.com/archives/CDRCA57FC/p1637249713314700?thread_ts=1637249339.314600&amp;cid=CDRCA57FC and this could be part of my issue. Does this mean in order to use Upsert, I have to go with the hybrid approach and set-up ingestion jobs to load the offline data?
n
upsert doesn’t work with hybrid tables. The answer was not specifically about realtimeToOffline
k
Thanks @Neha Pawar. So if I'm setting up a realtime table with upsert, can I use realtimetooffline pinot managed offline flow?
n
setting up Pinot managed offline flow implies that you have a hybrid table. so upsert wont work
👍 1
k
@Xiang Fu, just wanted to circle back on these errors. Any suggestions on troubleshooting?
x
Can you try to download a segment from the s3 path and see if you can untar it?
Also is there more logs from server side to show where server side error is
k
This is excerpt from pinot-server log @Xiang Fu
pinot-server-log.txt
The files generated don't have the 
.tar.gz
extension. I downloaded the file, added the extension and was able to untar it
x
From the log it says that the server doesn't have s3 access to download segments
Do you need to configure accessKey and secret for your aws access?
k
This is docker inspect pinot-server
Copy code
"Env": [
                "AWS_ACCESS_KEY_ID=xxxxxxx",
                "AWS_SECRET_ACCESS_KEY=xxxxxxxxxx",
                "JAVA_OPTS=-Dplugins.dir=/opt/pinot/plugins -Xms32G -Xmx32G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log",
                "PATH=/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "JAVA_HOME=/usr/local/openjdk-11",
                "LANG=C.UTF-8",
                "JAVA_VERSION=11.0.12",
                "PINOT_HOME=/opt/pinot"
            ],
I'm passing in the access key and secret here.
Is there any other config I need to change? I've provided the conf in this thread above.
x
hmm, this config looks very correct, it’s weird that you cannot download files. shall we do a zoom session to go over it?
onething i’m looking for is the exception inside the downloadToLocal
this should be using S3PinotFS class
but I don’t see the error logs for it
k
I can search the log. I'm also available for a zoom session.
n
shouldn’t this include the accesskey, secretekey @Xiang Fu?
Copy code
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=us-east-1
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
or will the defaultCredentialsProvider be able to just pick the env variables?
Open in Slack
PreviousNext
Powered by