https://pinot.apache.org/ logo
#troubleshooting
Title
# troubleshooting
d

Diogo Baeder

04/17/2022, 9:34 PM
Hi guys, I'm having an issue running Pinot 0.10 locally where, sometimes after I stop my local docker-compose services and start them again, my segments go missing and marked as in "bad" status. It doesn't happen all the time, like now I tested and after the 3rd restart of the services I managed to have them in a "bad" state. I compared directories before and after and couldn't find out what's happening that caused them to "spoil". Any clues why this might be happening? More details in the thread here.
Log file for the docker-compose execution:
My docker-compose file:
The segments are being saved to
/tmp/data/segments
, which is mounted in the containers so that I don't lose the data between restarts.
m

Mayank

04/17/2022, 9:38 PM
Are you tearing down the entire cluster, including ZK? If so you will lose data
d

Diogo Baeder

04/17/2022, 9:38 PM
Well, it's just a local experiment really, not a production cluster. I'm aware that in production I shouldn't have it running like that, of course.
For development, however, it's annoying to keep losing my test data, I'd like to make sure it will always be there after I restart the containers.
m

Mayank

04/17/2022, 9:44 PM
If you’d like durablity then do a rolling restart or ensure ZK and deep store are not completely shutdown
d

Diogo Baeder

04/17/2022, 9:48 PM
What's the order of when the components can go down? I cannot afford keeping ZK running forever (since it's a test cluster running in my personal computer), but "deep store" is easy to ensure since I just mount a directory as a volume there to keep it between restarts. Currently I have: • ZK starting first • Controller depending on ZK (so starting second) • Broker and Server depending on Controller (so starting third)
This is the order in which docker-compose is stopping them now:
Copy code
^CGracefully stopping... (press Ctrl+C again to force)
[+] Running 5/5
 ⠿ Container pinot-broker-xavier      Stopped                                                                                                                                                                                          10.4s
 ⠿ Container trino-server-xavier      Stopped                                                                                                                                                                                           0.4s
 ⠿ Container pinot-server-xavier      Stopped                                                                                                                                                                                           0.6s
 ⠿ Container pinot-controller-xavier  Stopped                                                                                                                                                                                           1.0s
 ⠿ Container zookeeper-xavier         Stopped                                                                                                                                                                                           0.6s
does it make sense?
m

Mayank

04/17/2022, 10:32 PM
Shutdown order server broker controller zk
You can snapshot ZK and bring it back up from snapshot
d

Diogo Baeder

04/17/2022, 10:33 PM
Currently I'm mounting a volume for ZK, instead of having to manage snapshots or anything like that. Locally it's easier to handle.
Thanks for the component order, I'll change that here!
m

Mayank

04/17/2022, 10:46 PM
If zk state isn’t snapshoted and everything is torn apart, it will lead to data loss. Not just for Pinot, any system out there.
d

Diogo Baeder

04/17/2022, 10:56 PM
I was taking a look at the ZK image docs, and it seems that the snapshots are saved to
/data
, and I have both
/data
and
/datalog
mounted as volumes, so I believe I should be safe.
👍 1
d

Daniel Lavoie

04/18/2022, 12:12 PM
Also double check that you configured Pinot component uses hostname as registration to Helix. This preserve identity between restart. By default, Pinot will use local IP which may change between docker compose restarts
d

Diogo Baeder

04/18/2022, 12:36 PM
Oh, I didn't know that! How can I check that?
d

Daniel Lavoie

04/18/2022, 12:38 PM
<http://pinot.set.instance.id.to|pinot.set.instance.id.to>.hostname=true
must be configured on all components
d

Diogo Baeder

04/18/2022, 12:40 PM
I'll check if there's an env var I can use to set that, since I'm using the docker image
d

Daniel Lavoie

04/18/2022, 12:41 PM
Otherwise you might have to mount property files through host volumes mount.
There is no env var for that to my knowledge
d

Diogo Baeder

04/18/2022, 12:56 PM
😞
@User I checked the image and that configuration is already set, so I didn't have to make any change for that. Is there any other way I can find out if there's any missing piece that might be corrupting the segments between restarts?
d

Daniel Lavoie

04/22/2022, 7:54 PM
Logs and segment state should tell you more about the nature of the bad segment
d

Diogo Baeder

04/22/2022, 7:54 PM
OK, I'll keep an eye on those parts when it happens again. Thanks man! 🙂