Hello! Appreciate that there's such an active slac...
# troubleshooting
s
Hello! Appreciate that there's such an active slack/ troubleshooting channel. Way to go Pinot team! I am trying out backup and restore procedure on different clusters/environments that are sized differently i.e. different # of servers/brokers/ etc. The zk metadata seems to contain data regarding num servers/minions/etc. What's the recommended approach to restore an existing pinot DB to a different cluster that might be sized differently? The deep storage data can be copied freely so there's a 1:1 relationship with pinot controller-deep storage
My thought is logical backup/restore via querying pinot api for the tables/schemas, but wanted some input if there's a better approach.
Related question: If kafka isn't set up similarly to the original cluster, can we still query with just the existing segment data in deep store?
v
Yeah pinot doesn't provide any backup tool or cli. You will have segments in tar.gz format in deep store and metadata in zk. Basically you can offload the deepstore data and zk meta in the new setup and reload the tables. This should work I guess
s
Context: back up and restoring pinot to another cluster I've created a new pinot instance with fresh zk. Recreated the schema and table (but configured as offline instead of realtime) Trying to re-upload the segments, but the segments go in a bad state. Error log in đź§µ
Untitled
x
is segment on s3?
is this file exist at
s3://pinot/TableName/TableName__7__1__20241115T1814Z
or
s3://pinot/TableName/TableName__7__1__20241115T1814Z.tar.gz
s
Yup, segment is on s3 No .tar.gz suffix, but it is gzip compressed
Copy code
❯ file Apilot_risk_profile__4__0__20241003T0050Z
Apilot_risk_profile__4__0__20241003T0050Z: gzip compressed data, original size modulo 2^32 18944
x
ok
can you try to rename the s3 file to
s3://pinot/TableName/TableName__7__1__20241115T1814Z.tar.gz
then push the segment
s
No luck. Same error.
Thanks for helping by the way. 🙂
x
what’s the error and logs?
s
Same error and logs as above. Including without the .tar.gz suffix, even though segment on s3 is appended with .tar.gz as well as the local file uploaded with the /segments api.
x
hmm, did you configure s3 segment fetcher in your server
https://docs.pinot.apache.org/basics/data-import/pinot-file-system/amazon-s3#server-config something like :
Copy code
pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=ap-southeast-1
pinot.server.storage.factory.s3.httpclient.maxConnections=50
pinot.server.storage.factory.s3.httpclient.socketTimeout=30s
pinot.server.storage.factory.s3.httpclient.connectionTimeout=2s
pinot.server.storage.factory.s3.httpclient.connectionTimeToLive=0s
pinot.server.storage.factory.s3.httpclient.connectionAcquisitionTimeout=10s
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher