I am wondering if there is any significant interes...
# ask-aia
Alex Koshterek
05/20/2021, 6:54 PMI am wondering if there is any significant interest in rather dummy streaming destinations like Kafka, PubSub, etc? It could enable writing custom code for handling messages (Kafka Streams, Dataflow, other streaming tools). Just write your own transformation and forward it further to a storage/warehouse/another stream/possibly another Airbyte streaming source.
airbyte heart 2
s
s
05/20/2021, 7:40 PM
Definitely! Kafka is a highly requested destination and we plan on bringing it to Airbyte in the next month or two
🙌 1
s
05/20/2021, 7:41 PM
it would helpful for us to know
• What volume of data do you anticipate replicating? bytes or number of records estimate is helpful to know.
• Does the data you write need to be partitioned? What partitioning scheme do you need?
• What data format would you expect to be written to kafka?
• What is the UX you want for interacting with the connector? What inputs/configuration do you want to give it, and how do you want the output to look like?
a
Alex Koshterek
05/21/2021, 7:55 AMSeveral millions per day. No partitioning, raw Airbyte JSON messages would be good enough for me. Avro format also would work.
UI should be pretty simple. Topic to publish, secrets and that’s it
u
[DEPRECATED] Marcos Marx
05/27/2021, 5:34 PM
should we take this to Github issue? 😄
s
s
05/27/2021, 5:38 PM
5 Views