https://pinot.apache.org/ logo
Join Slack
Powered by
i confused on Deep store and replica factor. As I ...
# general
s
i confused on Deep store and replica factor. As I understood The deep store is the permanent store for segment files.if server crash then it pull records form deep store.Then what is the role of replica here.If we donot have Deep store configuration and only replica configuration,will server recover on crash,
x
replication is for HA, it ensures all requests coming won’t fail during server crush or restart.
m
And to add, deep-store is for durability of data.
s
is it possible configure deep-store in local storage.in our product we donot have hadoop or s3 bucket.is it safe to install pinot in production.
x
Then how big is your local storage? Is it a NFS or just pure local
s
its a pure local storage .our local storage has 5tb.We are taking snapshot of machine everyday
controller.helix.cluster.name=pinot-labtest controller.port=9000 controller.data.dir=/data/pinot/backup controller.local.temp.dir=/tmp/pinot-tmp-data/ controller.zk.str=pinot-zookeeper:2181 pinot.set.instance.id.to.hostname=true controller.task.scheduler.enabled=true
is above configuration work ?
x
Then you can try this, just the total storage of the cluster is 5tb compressed. So likely 15-20tb
m
In production you will need > 1 controllers. And the storage across the controllers has to be shared. If not deepstore, then it has to be something like NFS.
s
Ok understood and i have to give same nfs path in controller.data.dir of controller configuration.am I correct
👍 1
e
@Mayank Can the pvc function as a deep store or is that just the segment store?
m
One requirement is that all controllers need to share the same dataDir. I think that is not feasible with pvc, right?
e
No I don’t think not in the current helm/kubernetes setup at least. I’m just wondering could relying on pvcs for the servers be wrong, should I wire in a deepstore even if I’ve allocated 1TB of pvc per server.
m
Any reason for avoiding S3 or similar deepstore?
e
None just waiting for docs on ADLS integration
m
I see, in that case, my recommendation is to set it up.
e
What I’ve noticed is it only leverages a small amount of the pvc(in the server) and starts complaining IO/disk space in the controller logs
And I’m wondering is this the fuse
Cause*
m
server pvc is not visible to controllers. Controllers need their own (deep) storage.
e
Ah perfect so that’s the cause of the bottle neck potentially?
Is there a hard limit on server pvc utilisation, what I’m seeing is it sends requests to controller (which complains due to lack of deep storage), however as you point out when the controller lacks storage itself it’ll fail. I thought it would keep using the server pvc and just ignore deepstorage.
x
for realtime table, ingestion will pause when the controller(deep store) write failure.
which means you need enough controller disk space
e
Ahhh @Xiang Fu this is perfect it explains my current issue thanks so much!
@HARIHARAN read this thread
Did you guys get any docs together for ADLS/azure storage accounts as deep store yet?
@Daphne Tan
x
e
That’s import though right?
x
under it there are sections for controller/servers/minions
cause we need to configure adls as deep store to host segments and server needs permission to load data .
e
Ah perfect thanks Xiang come to Dublin sometime I’ll buy you a Guinness
x
🙂 🍻
❤️ 1
m
Xiang is more of a Scotch guy, fyi 😁
e
Noted 😂 😂 😂
Open in Slack
PreviousNext
Powered by