hello! question on how airbyte transformation work. I've a use case to read the data from a kafka topic (source data is in JSON format) and perform transformation of data to our data models (common data model - target JSON format). If anyone in this group has done similar work? I'd need help on understanding how to setup the transformation (convert source JSON to target JSON)

Hi @Haritha Gunawardana - Airbyte replicates data in JSON format today so by default, we will output records as a single-column JSON blob along with some additional Airbyte fields. If some formatting is required to get your data into a particularly specific JSON format, you apply your own custom transformations with Airbyte using dbt. Here is a bit more detail on how Airbyte manages transformation

hi @Wass (Airbyte) - Where does this transformation fits into the pipeline, Can we configure the custom transformation between source and destination? e.g. read the source data from an API listener -> transform the source data into a different data model -> publish to destination (maybe kafka)

The transformation occurs after initial replication. Today, all data selected in the replication tab is synced to the destination in raw json. From there, we use dbt models to normalize the data using the destination compute.

Hello, Seems like you can achieve it by multi step pipeline: 1) Copy data from source kafka topic to target db (postgres, sql server, other db). 2) Run transformation over replicated data. 3) Publish transformed data into other destinations (kafka or any other). There are two options to transform data. 1. You can do it by manually coding using built in capabilities (https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-sql) and then add this code to your pipeline. 2. Another option is to transform incoming json into target data model on Datuum, using its no-code UI. Here is a . Datuum generates the whole Airbyte connection with link to the dbt project on Git added.