https://pinot.apache.org/ logo
Join Slack
Powered by
Hi guys, I'm having an issue running Pinot 0.10 lo...
# troubleshooting
d
Hi guys, I'm having an issue running Pinot 0.10 locally where, sometimes after I stop my local docker-compose services and start them again, my segments go missing and marked as in "bad" status. It doesn't happen all the time, like now I tested and after the 3rd restart of the services I managed to have them in a "bad" state. I compared directories before and after and couldn't find out what's happening that caused them to "spoil". Any clues why this might be happening? More details in the thread here.
Log file for the docker-compose execution:
pinot-missing-segments.log
My docker-compose file:
docker-compose-databases.yml
The segments are being saved to 
/tmp/data/segments
, which is mounted in the containers so that I don't lose the data between restarts.
m
Are you tearing down the entire cluster, including ZK? If so you will lose data
d
Well, it's just a local experiment really, not a production cluster. I'm aware that in production I shouldn't have it running like that, of course.
For development, however, it's annoying to keep losing my test data, I'd like to make sure it will always be there after I restart the containers.
m
If you’d like durablity then do a rolling restart or ensure ZK and deep store are not completely shutdown
d
What's the order of when the components can go down? I cannot afford keeping ZK running forever (since it's a test cluster running in my personal computer), but "deep store" is easy to ensure since I just mount a directory as a volume there to keep it between restarts. Currently I have: • ZK starting first • Controller depending on ZK (so starting second) • Broker and Server depending on Controller (so starting third)
This is the order in which docker-compose is stopping them now:
Copy code
^CGracefully stopping... (press Ctrl+C again to force)
[+] Running 5/5
 ⠿ Container pinot-broker-xavier      Stopped                                                                                                                                                                                          10.4s
 ⠿ Container trino-server-xavier      Stopped                                                                                                                                                                                           0.4s
 ⠿ Container pinot-server-xavier      Stopped                                                                                                                                                                                           0.6s
 ⠿ Container pinot-controller-xavier  Stopped                                                                                                                                                                                           1.0s
 ⠿ Container zookeeper-xavier         Stopped                                                                                                                                                                                           0.6s
does it make sense?
m
Shutdown order server broker controller zk
You can snapshot ZK and bring it back up from snapshot
d
Currently I'm mounting a volume for ZK, instead of having to manage snapshots or anything like that. Locally it's easier to handle.
Thanks for the component order, I'll change that here!
m
If zk state isn’t snapshoted and everything is torn apart, it will lead to data loss. Not just for Pinot, any system out there.
d
I was taking a look at the ZK image docs, and it seems that the snapshots are saved to 
/data
, and I have both 
/data
and 
/datalog
mounted as volumes, so I believe I should be safe.
👍 1
d
Also double check that you configured Pinot component uses hostname as registration to Helix. This preserve identity between restart. By default, Pinot will use local IP which may change between docker compose restarts
d
Oh, I didn't know that! How can I check that?
d
<http://pinot.set.instance.id.to|pinot.set.instance.id.to>.hostname=true
must be configured on all components
d
I'll check if there's an env var I can use to set that, since I'm using the docker image
d
Otherwise you might have to mount property files through host volumes mount.
There is no env var for that to my knowledge
d
😞
@User I checked the image and that configuration is already set, so I didn't have to make any change for that. Is there any other way I can find out if there's any missing piece that might be corrupting the segments between restarts?
d
Logs and segment state should tell you more about the nature of the bad segment
d
OK, I'll keep an eye on those parts when it happens again. Thanks man! 🙂
Open in Slack
PreviousNext
Powered by