This message was deleted.
# generals
Slackbot
05/08/2023, 5:18 PMThis message was deleted.
f
Felix GV
05/08/2023, 5:37 PMThanks, I'll take another look.
Felix GV
05/08/2023, 5:38 PMI have a question on the general design/intent though: you will have a Pulsar topic which goes into this sink (some sort of serverless function hosted inside Pulsar, IIUC) which will then write it to Venice. Correct?
a
Andrey
05/08/2023, 5:39 PMCorrect
f
Felix GV
05/08/2023, 5:42 PMBut Venice is itself a collection of topics:
• A real-time topic, which this sink writes to, and which is consumed by Venice server leader replicas. Akin to a write-ahead log.
• One or many version-topic(s), written to by leader servers (after applying partial updates and conflict resolution rules, if any), and consumed by follower servers. Akin to a commit log.
So by using the Pulsar sink design, the write path now contains 3 sequential topics rather than 2, and the first 2 of these 3 topics are going to be essentially identical (though perhaps partitioned differently...).
Is there a benefit to having this first topic prepended to the pipeline? Versus just writing directly into the Venice RT topic?
I could imagine scenarios where it's useful but not sure if it's a good default recipe (at least from the efficiency standpoint, but also tail latency/availability, since there are more moving parts).
a
Andrey
05/08/2023, 5:49 PM+ @Enrico Olivelli
This is a first stage. It helps make Pulsar data usable in Venice without tying architecture to Kafka explicitly (e.g. we could use existing Pulsar Kafka sink). Still this would use Pulsar and (Venice + Kafka)
Later on we'd work on not having Kafka in the path to Venice (use Pulsar instead).
Andrey
05/08/2023, 5:51 PMOne of the benefits would be allowing co-existence of Pulsar and Kafka (e.g. used for different usecases) and existing Venice cluster
f
Felix GV
05/08/2023, 6:06 PMWhat would be the benefit of this co-existence? (besides as a transient state while migrating from one to the other)
Felix GV
05/08/2023, 6:06 PMJust to be clear, I’m not against merging the PR, but would like to understand what use cases it is good for, and less good for, so we can advise users accordingly.
e
Enrico Olivelli
05/08/2023, 7:50 PMLet's see the Pulsar Sink as a third way of writing to Venice.
In the Pulsar eco system it is common to have these connectors.
https://pulsar.apache.org/docs/3.0.x/io-overview/
Enrico Olivelli
05/08/2023, 7:52 PMThis Sink work is not about storing Venice data on Pulsar, but letting Pulsar users write to Venice.
There are already many connectors to write to any DB. It is the time for Venice
Enrico Olivelli
05/08/2023, 7:53 PMAt Datastax we have many users who build pipelines to connect various datasources (using Pulsar Sources) with other systems (using Sinks) like Cassandra, ElasticSearch, Oracle.....
Enrico Olivelli
05/08/2023, 7:54 PMWith this sink for instance it is super easy to build a pipeline who gets data from Cassandra using CDC, then you process it and they you push it to Venice
Enrico Olivelli
05/08/2023, 7:55 PMSo the benefit is more about having the ability to easily interconnect the systems.
Enrico Olivelli
05/08/2023, 7:55 PMI know that it may sound inefficient but it is actually very powerful
➕ 1
f
Felix GV
05/11/2023, 5:45 PMThanks for the efforts on this. I have merged it now. Cheers!
🎉 1
e
Enrico Olivelli
05/11/2023, 7:07 PMCongratulations!
2 Views