Hi. Pinot team. Have you ever experienced a phenom...
# troubleshooting
c
Hi. Pinot team. Have you ever experienced a phenomenon in pinot 1.2 where one replica of the consuming segment remains as COINSUMING? The previous consuming segment has 2 ONLINE and 1 CONSUMING replica. The newly created consuming segment becomes under-replica as only 2 CONSUMING segments are created. ex) segment__9__2027__timestampZ { "server1" : "ONLINE", "server2" : "ONLINE", "server3" : "CONSUMING", } segment__9__2028__timestampZ { "server1" : "CONSUMING", "server2" : "CONSUMING" } I've been analyzing this phenomenon for a while, but I can't find the cause.
x
how long this situation last? After the commit, pinot allow server to commit segment locally, if you want to disable this behavior, you can change this
completionMode
to
DOWNLOAD
https://docs.pinot.apache.org/configuration-reference/table#segmentsconfig to always download from the deep store
m
Also check for the segment name in the server log to see what is happening with it
c
@Xiang Fu @Mayank This phenomenon continues until the problematic server is restarted. I have already tested this setting, and it also occurs when the 'completionMode' setting is 'DOWNLOAD'. I have now reverted this setting. In addition, I have tested forceCommit, reload, refresh, etc. I have checked the controller + zookeeper + server logs several times. In the pinot server, there is no progress on the problematic segment since the log below. 2025/05/17 141609.972 org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandlersendRequest212 Controller response {"streamPartitionMsgOffset":"481047630","buildTimeSec":-1,"isSplitCommitType":true,"status":"KEEP"} for http://pinot-cdpick-pg-mgmt03.example.com:9001/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=481047630&instance=Server_pinot-cdpick-pg-worker04.example.com_8001&offset=-1&name=account_zzzz_user_meta__0__1937__20250517T0207Z&rowCount=187219&memoryUsedBytes=1553646378 And I changed the log level of the packages below to DEBUG, but I still couldn't find the log. org.apache.pinot.core.data.manager.realtime org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler org.apache.pinot.common.utils.http.HttpClient org.apache.helix.messaging.handling.HelixStateTransitionHandler org.apache.pinot.controller.helix.core.realtime org.apache.pinot.controller.helix.core.assignment org.apache.pinot.controller.api.resources
This log is the last helix log. 2025/05/17 141607.773 org.apache.helix.messaging.handling.HelixStateTransitionHandlerinvoke339 Instance Server_pinot-cdpick-pg-worker04.example.com_8001, partition account_zzzz_user_meta_01937_20250517T0207Z received state transition from CONSUMING to ONLINE on session 10066f609420052, message id: ddbca879-1dea-4cef-bff5-df95868dce91
m
Can you check the message queue size in the problematic server (via ZK)? If logging configs are correct then one possibility is that there are more state transitions than the server can handle and this one is queued. Also, what's the CPU/mem usage on the server?
c
@Mayank How do I check the message queue size?
m
Check debug api first, or if swagger has an api. If not, then it should accessible via ZK browser (inside of instances)
c
@Mayank These are the metrics that occurred around 18:55 on May 31st. The following log is the last log related to the segment on the server. -- helix log 2025/05/31 185558.906 INFO [HelixTaskExecutor-message_handle_thread_22] org.apache.helix.messaging.handling.HelixStateTransitionHandlerinvoke339 Instance Server_pinot-cdpick-gs-worker06.example.com_8001, partition account_zzzz_user_meta__9__2027__20250531T0646Z received state transition from CONSUMING to ONLINE on session 20066cf85a200a8, message id: 22fbea8c-3aab-465b-9049-9e5b32a316a1 -- pinot log 2025/05/31 185559.201 INFO [account_zzzz_user_meta__9__2027__20250531T0646Z] org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandlersendRequest212 Controller response {"buildTimeSec":-1,"streamPartitionMsgOffset":"497322627","isSplitCommitType":true,"status":"KEEP"} for http://pinot-cdpick-gs-mgmt03.example.com:9001/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=497322627&instance=Server_pinot-cdpick-gs-worker06.example.com_8001&offset=-1&name=account_zzzz_user_meta__9__2027__20250531T0646Z&rowCount=185376&memoryUsedBytes=1550035288 The CPU/MEM usage of the pinot server does not deviate from the usual pattern. CPU usage is less than 5%. The server's queue is unknown, but this is a screenshot of the cluster's message processing status confirmed by Zookeeper. As expected, it does not deviate from the usual usage pattern. The charts around 18:00 seem to be influenced by GC. However, the time the segment issue occurred on this day is around 18:55, so the time does not match.
m
If the server received a state transition, it should either process it (and log it), or it is still in the queue. It won't just not do anything.
c
@Mayank In the Zookeeper browser, the following message remains in the server's message list: "MSG_TYPE": "STATE_TRANSITION", "FROM_STATE": "CONSUMING', "TO_STATE" : "ONLINE" On the controller server, this message is logged repeatedly as "Message already exists ... "
m
cc @Jackie
j
@coco I assume the states you shared are from the external view. Can you also share the states from the ideal state?
Based on this log:
org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler:sendRequest:212 Controller response {"buildTimeSec":-1,"streamPartitionMsgOffset":"497322627","isSplitCommitType":true,"status":"KEEP"}
, the server is supposed to build the segment locally after receiving this, so we need to figure out why that is not happening. Is it possible that server runs out of memory for some reason, and some threads are killed somehow? I do see a big GC spike from the matrics
c
@Jackie Are you talking about the GC Spike in the metric chart I uploaded? I guess I didn't explain enough. This GC metric chart is the metric of the Zookeeper servers. And the issue on that day occurred around 18:55, and the GC Spike of Zookeeper occurred around 18:00, so I don't see this as the cause. This is the Ideal state and External View of the segments at that time. Is this what you wanted?
@Jackie Here are the metrics from the server where the issue occurred. There is some usage due to other offline table collection batches running, but I haven't noticed anything unusual. Anything suspicious?
j
There are multiple possibilities. Given there is no OFFLINE (initial state) segment in EV, it might be related to ZK stats somehow not written to the ZK.
Please try to find more clue from the log around that time
c
@Jackie > There are multiple possibilities. Given there is no OFFLINE (initial state) segment in EV, it might be related to ZK stats somehow not written to the ZK. Could you explain in more detail? I have analyzed the logs at the INFO log level several times, but I have not found any clues. This is the log that we changed to the DEBUG level. I have not found any clues in this DEBUG log either. Which log level should I change to further analyze this phenomenon? > org.apache.pinot.core.data.manager.realtime > org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler > org.apache.pinot.common.utils.http.HttpClient > org.apache.helix.messaging.handling.HelixStateTransitionHandler > org.apache.pinot.controller.helix.core.realtime > org.apache.pinot.controller.helix.core.assignment > org.apache.pinot.controller.api.resources (νŽΈμ§‘λ¨)
j
Do you see any ERROR or WARN in both pinot log and helix log around that time?
You can also search for INFO log in pinot log to see if the segment is successfully built locally
One possibility is some temporary network issue between server and ZK. Not sure if you can find evidence of that
Per the helix log, server did receive the state transition. Do you see the state transition being handled within the pinot log?
c
@Jackie Yes, there is no WARN, ERROR log. If I found something, I would have submitted it to the thread. πŸ˜₯ I am analyzing the source code and I have some background information that I am curious about. 1) Change the segment in CONSUMING state to ONLINE state. (RealtimeTableDataManager:addOnlineSegment) 2) Create a new CONSUMING segment. (RealtimeTableDataManager:addConsumingSegment) Looking at the log recording time, it seems that the Non-Commiter server is performing the above two actions simultaneously. Are there any dependencies between these two actions? Does the failure of 1) cause the failure of 2)? Does the failure of 2) cause the failure of 1)? I think that when I look at the ExternalView of the segment, 2) failed because 1) failed. (I also heard the same thing in your comment: " there is no OFFLINE (initial state) segment in EV" ) Am I right?
And I found the controller log for the problematic server among the 2 Non-Committer servers. Is the COMMITTED:KEEP state of this login an issue? 2025/05/17 141609.971 INFO [grizzly-http-server-21] SegmentCompletionFSM_account_zzzz_user_meta__0__1937__20250517T0207Zkeep660 COMMITTED:KEEP for instance=Server_pinot-cdpick-pg-worker04.dakao.io_8001 offset=481047630 2025/05/17 141609.971 INFO [grizzly-http-server-21] org.apache.pinot.controller.api.resources.LLCSegmentCompletionHandlerssegmentConsumed152 Response to segmentConsumed for segment:account_zzzz_user_meta__0__1937__20250517T0207Z is :{"status":"KEEP","buildTimeSec":-1,"streamPartitionMsgOffset":"481047630","isSplitCommitType":true}
j
KEEP
means server should keep its local copy and just load it as the
ONLINE
segment, so everything is as expected
If you are running
1.2
release, you might want to checkout the code for that version. IIRC we did some refactoring on the segment management in
1.3
s
@Jackie Hi ! We've identified a problematic sequence when non-winner server try to change segment state (CONSUMMING -> ONLINE) 1. segmentConsumed request to Controller -> HOLD response 2. onBecomeOnlineFromConsuming 3. addOnlineSegment 4. segmentConsumed request to Controller -> KEEP response 5. no more logs 1~3 step is as expected. But, I guess 4 step is not as expected. The server receives a Helix notification to transition to ONLINE (steps 2&3), but then, while it's already in the process of becomming ONLINE, it sends another segmentConsumed request (step4) It seems that receiving this KEEP response at an out-of-order or unexpected time (when the segment is already transitioning to ONLINE) Could you confirm the reason for this out-of-order ? cc. @coco
j
Do you see segment being built locally? If not, maybe some other thread is holding the segment lock thus causing starvation
That actually reminds me, this might be the root cause: https://github.com/apache/pinot/pull/13632
s
@Jackie We didn't see any logs indicating the segment was built locally. In normal case, there is logs like this. org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager_realtimetable__3__2092__20250611T2209ZbuildSegmentInternal1021 Trying to build segment Could the bug you mentioned indeed be the cause? cc. @coco
@Jackie We checked pinot server thread dump. thread9 is waiting thread1, thread1 is waiting another thread. (message is like "parking to wait for <>") Then, we checked pinot server logs. thread1 is about adding CONSUMING segment (seg__8__2114). thread9 is about adding ONLINE segment (seg_8_2113) In fact, seg__8__2113 is problematic segment. The segment state didn't change yet. (not ONLINE)
j
Yeah, then #13632 should be the cause. It is a deadlock
c
@Jackie Hi, we have patched #13632 in pinot-release-1.2.0 source code. We will have to monitor it more, but there have been no issues for 2 weeks. Thanks for your help.
πŸ‘ 2