https://pinot.apache.org/ logo
Channels
Powered by
j

Jonathan Meyer

03/31/2021, 3:30 PM
Hello 👋 What could be the reasons for an external view to be missing ? (Specifically for a REALTIME table) Thanks
m

Mayank

03/31/2021, 3:31 PM
Missing or empty? Also, was it there before and got deleted?
May be zk issues?
j

Jonathan Meyer

03/31/2021, 3:32 PM
(For more context, it's a new table) It really is missing, I'm currently setting up multiple tables and all of them are missing an external view
For instance 
curl -s -X GET pinot-controller.infra-dev.svc.cluster.local:9000/tables/user_events
-> Returns the table definition But 
curl -s -X GET pinot-controller.infra-dev.svc.cluster.local:9000/tables/user_events/externalview
-> Return 
{"code": 404,"error": "Table not found"}
m

Mayank

03/31/2021, 3:34 PM
Hmm, how are you creating the table? Do you have access to controller logs to see what happens when you create a table? Also is ideal state created?
Can you try idealstate if it exists?
j

Jonathan Meyer

03/31/2021, 3:34 PM
Yep, idealstate is there I've got access to logs - I'll take a closer look at the moment the table are created Tables are created via the API
m

Mayank

03/31/2021, 3:35 PM
Yeah controller log should say something
j

Jonathan Meyer

03/31/2021, 3:36 PM
Probably unrelated because I've had these warning on a working table (other cluster) But I'm seeing multiple 
The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
(and other related fields) [On 0.7]
m

Mayank

03/31/2021, 3:37 PM
Any other errors or logs stating whether table was created or not?
j

Jonathan Meyer

03/31/2021, 3:45 PM
I can't seem to find a log telling that the table is indeed created, but via the API I can get the table config (and schema) back However, while creating a new table (after deleting schema & definition), I'm getting quite some warnings (see snippet above)
I find 
[ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-dev-(0dac30d3_TASK)] Fail to read record for paths: {/pinot-dev/INSTANCES/Server_pinot-server-0.pinot-server-headless.infra-dev.svc.cluster.local_8098/MESSAGES/e5d83f70-3ee9-4837-a9bd-58f06729ccb8=-101}
especially suspicious
While deleting the table beforehand, I got some 
[ZkClient] [grizzly-http-server-1] Failed to delete path /pinot-dev/PROPERTYSTORE/SEGMENTS/document_open_REALTIME! org.I0Itec.zkclient.exception.ZkException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /pinot-dev/PROPERTYSTORE/SEGMENTS/document_open_REALTIME
m

Mayank

03/31/2021, 3:54 PM
I think that might just mean you need to first delete all segments of a table before deleting table
It might be easier to collaborate on a zoom call if you are up for it
j

Jonathan Meyer

03/31/2021, 3:54 PM
I think that might just mean you need to first delete all segments of a table before deleting table
That actually one thing I've been wondering for a bit 🙂
m

Mayank

03/31/2021, 3:55 PM
I’d say do a clean start for a table and let’s look at logs
If you can create a new test cluster altogether, even better - to eliminate any weird state your current cluster might be in
✅ 1
j

Jonathan Meyer

03/31/2021, 3:56 PM
Thanks I'll try that If that doesn't solve it, we could do that
It might be easier to collaborate on a zoom call if you are up for it
m

Mayank

03/31/2021, 3:57 PM
Sure, ping me when you think zoom might be better, we can continue here until then
🙂 1
j

Jonathan Meyer

03/31/2021, 4:20 PM
By deleting the segments (while keeping the table around), the external view still doesn't exist (not very surprising) But The UI (Cluster Manager / Tables / Tables / <table_name>) now opens up instead of being stuck in an infinite loop And the status goes to Green, but still no external view (checked in ZK - it's missing), nor segments (expecting 1 consuming) I've also tried creating the table from the UI - same result : • Status: Bad (from UI) • external view: ❌ (missing) • ideal state ✅
The table I'm creating is very simple From the UI, I've basically only changed / input the table name, kafka broker & topic name
m

Mayank

03/31/2021, 4:22 PM
Can you paste the ideal state?
j

Jonathan Meyer

03/31/2021, 4:23 PM
Copy code
{
  "OFFLINE": null,
  "REALTIME": {
    "user_events__0__0__20210331T1617Z": {
      "Server_pinot-server-0.pinot-server-headless.infra-dev.svc.cluster.local_8098": "CONSUMING"
    }
  }
}
Copy code
$ curl -s -X GET pinot-controller.infra-dev.svc.cluster.local:9000/segments/user_events

[{"REALTIME":["user_events__0__0__20210331T1617Z

$ curl -s -X GET pinot-controller.infra-dev.svc.cluster.local:9000/segments/user_events/user_events__0__0__20210331T1617Z

{"code":404,"error":"Segment user_events__0__0__20210331T1617Z or table user_events not found in /var/pinot/controller/data/user_events/user_events__0__0__20210331T1617Z"}
m

Mayank

03/31/2021, 4:24 PM
Any logs in the server 
Server_pinot-server-0.pinot-server-headless.infra-dev.svc.cluster.local_8098
? Although, I don't expect there might be (since no external view), but would help to check
j

Jonathan Meyer

03/31/2021, 4:25 PM
I don't seem to find new logs vs the logs I've sent above
m

Mayank

03/31/2021, 4:26 PM
message has been deleted
In the zk explorer, can you check the znodes under INSTANCES/<server> to see if there are any errors?
j

Jonathan Meyer

03/31/2021, 4:28 PM
node 
server-.../ERRORS
is 
{}
same for 
MESSAGES
& 
STATUSUPDATE
m

Mayank

03/31/2021, 4:28 PM
and currentstates?
j

Jonathan Meyer

03/31/2021, 4:29 PM
m

Mayank

03/31/2021, 4:30 PM
And you don't see any logs in the server that suggest it was told by controller to start consuming?
j

Jonathan Meyer

03/31/2021, 4:31 PM
Sorry for the wall of text, maybe you can grep some interesting logs in there
Untitled
m

Mayank

03/31/2021, 4:48 PM
The errors I see in the log seem to suggest they are from the time you tried to delete the table. And the fact that I see a log line below seems to suggest the table was not properly created to being with. Wondering if we can start with a fresh table creation to check the logs (if fresh pinot cluster, even better):
Copy code
2021-03-31 18:00:56	2021/03/31 16:00:56.576 WARN [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Resource: user_events_REALTIME is not set up in idealState or ExternalView, won't do anything
j

Jonathan Meyer

03/31/2021, 4:51 PM
Yes, the table are missing the external view right from the start (after creation via the API) I've sent your logs surrounding table creation from a fresh start (no tables, but no fresh cluster)
Powered by