This message was deleted.
# general
s
This message was deleted.
e
in my case I would usually have 1 version, initialised as empty (empty push job) and I am populating it with a streaming writer (from Pulsar). so my understanding is that: • we only write to the xxxx_rt topic • the version topic is empty
f
No
e
so there is something that copies the data from the realtime topic to the version topic
f
The Venice leader replica consumes from the RT, resolves the result of the write (which could be a put, a delete or an update, and which can be subject to conflict resolution against what is already in Venice) and writes the output to the VT.
Venice follower replicas consume from VT and blindly apply that (no conflict res or anything like that)
So the Venice leader is kind of a constrained stream processor, in a sense
I need to go AFK for a meeting for a bit. I’ll be back later, if you still have more Qs.
e
no worries
thanks
So basically the data is stored on Kafka for durability and RocksDB is used because you cannot keep all the data in memory on the Venice servers, correct ?
there is no way to release disk space on Kafka
if my store is long lived, like months or years then the data will be kept forever in Kafka
on Pulsar long lived data will be moved to tiered storage (so for instance to an S3 bucket)
if everything goes well the Venice servers won’t read from Kafka anymore, even if you reboot them, correct ?
z
Yeah, if they have the data locally then they won't need to reconsume
It's only in cases of rebalance that they'll need to reingest or if the machine is wiped
e
how do we deal with partitions and replicas ?
which tools I can use to monitor and especially how many venice servers should store one store ?
I expect that I can shard/partition one store to multiple servers an also have multiple replicas per each shard
z
Exactly
The Replication factor is configurable, but by default it's three
e
can you please point me some venice-admin-tool commands ?
z
One moment
under update-store there is an argument called replication-factor (rf)
Setting this value configures how many ways each partition in a store is replicated
e
cool
how do I control the number of partitions ?
z
Same command, argument: count
er
partition-count
Take note, these configs will only come into effect as of the NEXT version push
so if you have a version of the data already present and set these things, they'll only apply as of the next version (as opposed to immediately)
f
Also, the partition count is currently not allowed to change on hybrid stores… so it needs to be provisioned correctly before configuring the store to be hybrid.
e
in my case I use incremental push, is it enough to make it “hybrid” ?
f
For batch-only stores, it is allowed to change (on the next push, as Zac said)
Yes, inc push is for hybrid stores
Hybrid is maybe not the best name… but it is how the code and store configs call it currently
e
👍