https://pinot.apache.org/ logo
Join Slack
Powered by
Team, Just created a test table consuming a test k...
# troubleshooting
m
Team, Just created a test table consuming a test kafka topic.. it works in other cluster, but in on one cluster it points to 
Specified EXTERNALVIEW pinot_test_REALTIME is not found!
There are other tables in prod that works from same kafka cluster, no new tables are working now. All ended up in this error or consuming_segment_info shows all servers as not responding, but they are working for others and are up. Tried, rebalance, reload, pause and resume.. seems externalview i dont see this table at all. Should we have to go for restarts? am worried what if currently working table runs into similar issue and unrecoverable.. any thoughts here?
m
Does the table exist in IdealState? Can you delete the table, ensure it is removed from idealstate and external view and the recreate it?
m
Yes they exist.. but am creating new table every time, doesn't matter if old exist right.. so all new ends up in same shape
m
If old exists, it needs to be fully removed before new with same name can be created
m
Nope, Different name every time.. its all to test like pinottest1 , pinottest2 like that.. What am seeing is already working cluster even working for current tables all of a sudden cudnt create new table with sample minimalist fields, works in other cluster and not here..
m
Then check controller log for table creation, it will likely have some error
What release?
m
1.2
m
Check controller log
m
Ack, will share the logs
again deleted the table and recreated, and checked the logs, 163401.829 [pool-12-thread-5] ERROR org.apache.pinot.controller.util.CompletionServiceHelper - Server: Server_acc-hdppinot307.phonepe.mhb_8098 returned error: 404, reason: Not Found for uri: http://bloomfil1.dom:8097/table/pinot_test_table2_REALTIME/size The same works in another cluster. Nothing much in controller logs when i create the table apart from this. Also tried Airline stat offline table , even that doesn't get created in external view. Do u think I should stash out ZK and rebuild it? I see this at the starting of ctl nodes. 164702.592 [grizzly-http-server-34] ERROR org.apache.helix.manager.zk.ZkBaseDataAccessor - paths is null or empty _Consuming_segment_info_ { "serversFailingToRespond": 6, "serversUnparsableRespond": 0, "_segmentToConsumingInfoMap": { "pinot_test_table2__0__0__20260316T1103Z": [], "pinot_test_table2__1__0__20260316T1103Z": [], "pinot_test_table2__2__0__20260316T1103Z": [] } } But there are pinot servers 9 servers overall. and all are up and running fine. Kinldy suggest. thanks!!
m
Do the servers appear as live instances in ZK browser
m
under liveinstances yes. all are there.
m
I can't think of a reason why this would happen, this needs dedicated debugging
s
Is the server tag set correctly in the table config? is that consistent with the tag in helix for the servers/
m
they are all under DefaultTenant_REALTIME and same for server tag.
They all are circling around this log: 141429.207 [grizzly-http-server-93] ERROR org.apache.helix.manager.zk.ZkBaseDataAccessor - paths is null or empty
Got a breakthrough. • Cleared znode data for pinot. • Restarted entire cluster. • ZK repopulated data from cluster restart reported. • Created new tables. • Now able to see tables showing up in external views. 😌 cc: @Mayank @Subbu Subramaniam @Xiang Fu
s
This does not seem like a viable solution if indeed there is a problem. Maybe it could have been an issue with capacity on the servers (or controller), or perhaps zk quorum (unlikely). If former, it may show uo in a different form later.
m
yeah, i doubt that.. as all the health were good and no capacity issues as such, when no tables on the cluster , on a basic setup we endup create table not updating externalview. Helixmanager failed to do it based on above log.
s
Maybe your lead helix controller is overloaded. By default, Pinot installs with helix controller built into pinot controlller. One of the pinot controllers acts as the lead helix controller. You can check the helix logs in your lead controller to see if you find some clues.
m
There is no overload as such i observe.. as this is just empty cluster.. trying to create one table.. Zookeeper data wipeout worked..
2 Views
Open in Slack
PreviousNext
Powered by