https://pinot.apache.org/ logo
Join Slack
Powered by
Hi team, I have a table in production, in which i ...
# troubleshooting
s
Hi team, I have a table in production, in which i have deleted a few consuming segments as the servers were not able to locate it. After triggering resumeConsumption, consuming segments are not getting created. please help
i am using pinot 1.0
Screenshot 2025-11-21 at 12.13.21 AM.png
@Mayank
hi team, this is a production issue and require immediate help
m
Why did you delete consuming segments? Also, why are you on > 2years old release?
s
the disk had reached 100 % and then had to rebalance servers. after rebalancing, 4 consuming segments were showing 235 unavailable and even after force commiting it did not change. So deleted the segments but now it is not starting consumption
will upgrading to 1.3 solve this?
m
Upgrade and run validation manager
s
1.3 or 1.4?
m
1,4
s
resumeConsumption will run the validation mnager right?
@Mayank how do I run the validation manager?
m
check swagger for rest api call
s
@Mayank why is it still showing 1.0?
m
check /version api
s
Screenshot 2025-11-21 at 10.56.44 AM.png
m
yes this is 1.4
s
I don't see rest endpoint for validation manager
m
it is probably called periodic task, validation manager is one type
s
I ran the periodic task. but still it isn't consuming
Screenshot 2025-11-21 at 11.43.23 AM.png
validationManager completed in 0 ms
@Mayank
@Mayank any resolution on this?
m
@harnoor - Could you please take a look ?
s
@harnoor Hi, could you please join here if possible https://meet.google.com/wvv-ssms-tec
ingestion status shows healthy
here the consuming status shows active
but not consuming segments are being created
h
hi, checking it now
Can you share the Ideal state and external view? It will be visibile under zookeeper browser tab, under table
s
external view
Copy code
"id": "bot_cx_master_data_REALTIME",
  "simpleFields": {
    "BATCH_MESSAGE_MODE": "false",
    "BUCKET_SIZE": "0",
    "HELIX_ENABLED": "true",
    "IDEAL_STATE_MODE": "CUSTOMIZED",
    "INSTANCE_GROUP_TAG": "bot_cx_master_data_REALTIME",
    "MAX_PARTITIONS_PER_INSTANCE": "1",
    "NUM_PARTITIONS": "23323",
    "REBALANCE_MODE": "CUSTOMIZED",
    "REPLICAS": "2",
    "STATE_MODEL_DEF_REF": "SegmentOnlineOfflineStateModel",
    "STATE_MODEL_FACTORY_NAME": "DEFAULT",
    "enableCompression": "true",
    "isTablePaused": "false",
    "pauseState": "{\"reasonCode\":\"ADMINISTRATIVE\",\"timeInMillis\":\"2025-11-21 08:58:23.16\",\"paused\":false}"
  },
Ideal state
Copy code
"id": "bot_cx_master_data_REALTIME",
  "simpleFields": {
    "BATCH_MESSAGE_MODE": "false",
    "HELIX_ENABLED": "true",
    "IDEAL_STATE_MODE": "CUSTOMIZED",
    "INSTANCE_GROUP_TAG": "bot_cx_master_data_REALTIME",
    "MAX_PARTITIONS_PER_INSTANCE": "1",
    "NUM_PARTITIONS": "23323",
    "REBALANCE_MODE": "CUSTOMIZED",
    "REPLICAS": "2",
    "STATE_MODEL_DEF_REF": "SegmentOnlineOfflineStateModel",
    "STATE_MODEL_FACTORY_NAME": "DEFAULT",
    "enableCompression": "true",
    "isTablePaused": "false",
    "pauseState": "{\"reasonCode\":\"ADMINISTRATIVE\",\"timeInMillis\":\"2025-11-21 08:58:23.16\",\"paused\":false}"
  },
you need all segment mapping as well?
@harnoor the segments I deleted were created on 8th november. there are 16 partitions out of which 4 segments were stuck and not consuming which is why i deleted them
h
So you deleted 4 consuming segments and ran the validation manager job, right?
Do you see any consuming segments for those partitions in Ideal state?
s
no
also since, these 4 were created on 8th november, the offsets might not be available on kafka as we have 7 day data retention in kafka. nd segments get created almost daily
h
If the partitions exists in upstream, then realtime segment validation job will create the consuming segments for all partitions. When you ran realtime segment validation job, did you see any error in logs?
s
no
image.png
h
that's weird. can u share the full ideal state. Interested in segment mapping
s
idealState.json
idealState.json
h
There is no consuming segment I can see.
s
that is what i am saying
nothing is getting consumed since yesterday
h
Validation manager calls kafka API and if partitions exists in kafka/has not reached end of life, it will automatically create consuming segments for it.
There must be some log related to it. Can you re-run validation job just for this table again?
s
need to check controller logs right?
h
yes
s
image.png
Copy code
Getting Helix leader: 10.220.184.79_9000, Helix version: 1.3.2, mtime: 1763715624143
Instance 10.220.183.87_9000 is not leader of cluster pinot-prod due to current session 1011ec5b00d2531 does not match leader session 3011ec647f724ea
[TaskRequestId: api-4c7c6f15] Sending periodic task message to all controllers for running task RealtimeSegmentValidationManager against  table 'bot_cx_master_data_REALTIME', with properties null."
Query returned 3 rows
Send 3 messages with criteria instanceName=%resourceName=leadControllerResourcepartitionName=partitionState=
119 START: CallbackHandler 1, INVOKE /pinot-prod/INSTANCES/Controller_10.220.183.87_9000/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@4c9bed65 type: CALLBACK
CallbackHandler 1 subscribing changes listener to path: /pinot-prod/INSTANCES/Controller_10.220.183.87_9000/MESSAGES, callback type: CALLBACK, event types: [NodeChildrenChanged], listener: org.apache.helix.messaging.handling.HelixTaskExecutor@4c9bed65, watchChild: false
CallbackHandler1, Subscribing to path: /pinot-prod/INSTANCES/Controller_10.220.183.87_9000/MESSAGES took: 0
[TaskRequestId: api-4c7c6f15] Periodic task execution message sent to 3 controllers.
Handled request from 10.151.210.4 GET <http://10.220.183.87:9000/periodictask/run?taskname=RealtimeSegmentValidationManager&tableName=bot_cx_master_data&type=REALTIME>, content-type null status code 200 OK
The latency of message 32c47740-aba0-4919-8a73-db704c276ad7 is 25 ms
Scheduling message 32c47740-aba0-4919-8a73-db704c276ad7: leadControllerResource:, null->null
Submit task: 32c47740-aba0-4919-8a73-db704c276ad7 to pool: java.util.concurrent.ThreadPoolExecutor@1c149539[Running, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
Message: 32c47740-aba0-4919-8a73-db704c276ad7 handling task scheduled
119 END:INVOKE CallbackHandler 1, /pinot-prod/INSTANCES/Controller_10.220.183.87_9000/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@4c9bed65 type: CALLBACK Took: 8ms
handling task: 32c47740-aba0-4919-8a73-db704c276ad7 begin, at: 1763718568019
[TaskRequestId: api-4c7c6f15] Handling RunPeriodicTaskMessage by executing task RealtimeSegmentValidationManager
[TaskRequestId: api-4c7c6f15] Schedule task 'RealtimeSegmentValidationManager' to run immediately. If the task is already running, this run will wait until the current run finishes.
[TaskRequestId: api-4c7c6f15] Start running task: RealtimeSegmentValidationManager
[TaskRequestId: api-4c7c6f15] Finish running task: RealtimeSegmentValidationManager in 0ms
Message 32c47740-aba0-4919-8a73-db704c276ad7 completed.
Delete message 32c47740-aba0-4919-8a73-db704c276ad7 from zk!
message finished: 32c47740-aba0-4919-8a73-db704c276ad7, took 3
Message: 32c47740-aba0-4919-8a73-db704c276ad7 (parent: null) handling task for leadControllerResource: completed at: 1763718568022, results: true. FrameworkTime: 2 ms; HandlerTime: 1 ms.
119 START: CallbackHandler 1, INVOKE /pinot-prod/INSTANCES/Controller_10.220.183.87_9000/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@4c9bed65 type: CALLBACK
CallbackHandler 1 subscribing changes listener to path: /pinot-prod/INSTANCES/Controller_10.220.183.87_9000/MESSAGES, callback type: CALLBACK, event types: [NodeChildrenChanged], listener: org.apache.helix.messaging.handling.HelixTaskExecutor@4c9bed65, watchChild: false
CallbackHandler1, Subscribing to path: /pinot-prod/INSTANCES/Controller_10.220.183.87_9000/MESSAGES took: 0
No Messages to process
119 END:INVOKE CallbackHandler 1, /pinot-prod/INSTANCES/Controller_10.220.183.87_9000/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@4c9bed65 type: CALLBACK Took: 0ms
Handled request from 10.220.188.248 GET <http://10.220.183.87:9000/tables>, content-type null status code 200 OK
^C
Screenshot 2025-11-21 at 3.20.34 PM.png
h
Can you share the entire log file? There must be some log like: 
Processing 1 tables in task: RealtimeSegmentValidationManager
s
controller-2025-11-21:09-00-27.log
controller-2025-11-21:09-00-27.log
table name is bot_cx_master_data, for which consumption is not happening
h
The logs are not parsed to have date/thread and class info attached to it. So its difficult to navigate. But what i can see is the the job did not run for the table and i see there are leader session logs as well from helix. Because there is no severity in the logs i believe that is a warning/error log.
Have you restarted the controller ?
s
yes
have restarted all controllers, servers and brokers
anyway I can resume consumption? or I need to replicate the whole table??
?
@harnoor
@harnoor Will you be able to help??
Open in Slack
PreviousNext
Powered by