This message was deleted.
# general
s
This message was deleted.
f
We do, for hybrid stores (as opposed to batch-only stores)
e
what happens if you don’t configure compaction ?
f
Venice auto-creates and auto-configures all its topics.
And in some cases, if you mess with the configs, Venice will set them back the way it likes 😂
e
When using Starlight for Kafka, so basically storing the Data on Pulsar, we ignore those configurations, because they are configured somewhere else in Pulsar (so the Admin API calls to set compaction are ignored)
f
I like to say that the relation between Venice and Kafka is akin to that of a cell and its mitochondria (Kafka is fully engulfed within Venice)
Then we would need to implement a Pulsar impl for our KafkaAdmin interface that goes and sets those configs in the right place
For compaction, there is no correctness issue if it’s missing… just an efficiency penalty. And we have alternative ways of triggering compaction (via repush), though that is not automatic as of today.
Are there other topic configs Pulsar doesn’t honor? Partition count in particular would be a must have…
e
Kafka Partitions are mapped 1-1 to Pulsar partitions, so no problems there
🤝 1
One core difference is about “replication”. Kafka clients see the topics as being served by only 1 broker, so replication factor is ALWAYS 1. This is because in Pulsar the Broker layer is separated from the Storage layer. Little teaser: In Pulsar you can configure (generally per namespace, but you can do it any level) 3 parameters for Replication. Per each message you have: • how many storage nodes (bookies, see Apache BooKeeper) contain the data • how many copies for each message • how many answers to wait (number of acks) For instance if you configure replication as 5-3-2 for a given “segment” of a topic you use: • 5 bookies • each message is written to 3 nodes • the writer (the broker, not the client) waits for 2 acks Also in case of storage failures the broker moves to writing to a different set of nodes automatically and seamlessly.