This message was deleted.

We have both the producer and consumer abstractions now… we still need the admin abstraction.

I missed the Consumer abstraction 🤔

What kind of offsets does Pulsar use? Is it numerical or some kind of opaque object?

it is opaque, you can serialize it as a string/byte[] and you can “compare” offsets, in the case of Pulsar you have a “MessageId”

Ok, so we intend to support non-numerical offsets, but that part is not done yet. For now the code assumes the offset type is always a long.

in Pulsar, especially when you enable the Kafka emulation, you can still have a “long” offset, that works like Kafka. The name of this stuff is “index” in Pulsar terms. But the only downside is that you cannot “seek” to a specific Index

Here: https://github.com/linkedin/venice/commit/0e4fafe143542d00a5f6482e30110506570754de

I believe that also we have to take into consideration: • compaction • partitioning • subscription types (in Pulsar you can have multiple consumers on the same topic even without partitions)

Yeah, no doubt… there will be lots of details to iron out. These systems are all a little different.

btw the very main thing that I would like it to be able to use the ‘/’ character in the storeName or, better, to have a way to customise the mapping between the storeName and the kafka topics in Pulsar you have the format “tenant/namespace/topicName” and in kafka only “topicName” it would be great to be able to name a store “tenant/namespace/storeName” and map its topics to “tenant/namespace/storeName-rt” ….

Sure, feel free to start as many threads as you’d like 😄 … sounds like an interesting idea.