Hi All, I’m Welly from scifin.tech. I’m looking into singer and found out that airbyte is inspired on that. That’s a great one. However i see that the communication between source and sink still using json via I/O. Is there any plan on changing that into more performant model like arrow flight etc? Previously we are using Spark for ingestion and utilize the parallization there to put the data into parquet format. any thought on this?

Hi, we have had conversations about this with no concrete outcome. This would be a pretty big change since JSON is the common data format all Airbyte components use to talk to each other. We aren’t opposed to changing this though. What is your concern here? Is it mainly speed?