Andrey Morskoy10/11/2021, 9:50 AM
, as well as
. Seems that ~60% of time source spends converting data into AirbyteMessage (before transformers) and later making
. Are there any plans on making these conversions less painful? I would be happy to get any info to understand in which direction this architecture moves generally. 2. Are there any plans for scalability? At this moment conversions and transformations, performed in
container, both are obvious subject to run in parallel. For me it looks pretty perspective to have
responsible only for data fetch in some raw form (byte arrays?) and delegate or complex conversions, transformations/normalization to scalable middle layer (even naive Apache Spark Streaming would be good improvement I suppose). May I ask which direction does Airbyte follow to deal with scalability?
user10/11/2021, 11:37 AM
user10/11/2021, 11:48 AM