Does Airbyte support simple data transformations during data migration? I'm referring to basic data manipulation logic, similar to the following SQL example:

CASE WHEN column_name IS NOT NULL THEN 'True' ELSE 'False' END AS new_column_name

The case I'm dealing with is that I have a PII data column in the SOURCE, which cannot be transferred to the DESTINATION. What I need is to set a flag in the DESTINATION if this PII column contains any value. I can't implement this logic on the DESTINATION (because data can't leave SOURCE), and I only have READ ONLY access to the SOURCE table, so I can't create a new column or view with this logic applied.

Airbyte is an ELT (extract, load, transform) tool, which means that raw data is first loaded into a destination and then after the data has been written, it goes through a transformation stage. The transformation stage is where you can apply your data manipulation logic. However, this happens after the data has been loaded into the destination, which might not be suitable for your use case as you mentioned that the PII data cannot be transferred to the destination at all. Airbyte does not currently support transformations at the extraction stage. The data that is extracted from a single record on the source system is sent to the destination as a single JSON blob, where it remains unmodified. This is a core tenet of the ELT philosophy, which is that data should be untouched as it moves through the extracting and loading stages so that raw data is always accessible. For post-sync transformations, Airbyte Cloud integrates natively with dbt. However, this feature is not available for Airbyte Open-Source. Also, this would not help in your case as the transformation happens after the data is loaded into the destination. In some cases, you can use views in the source database to remove the PII fields before extraction. However, this might not be applicable in your case as you mentioned that you only have read-only access to the source table and cannot create a new column or view. In conclusion, based on the current features of Airbyte and the constraints of your use case, it seems that Airbyte might not be able to support the specific data transformation you need during the data migration process. Sources: - [Airbyte's approach to ELT](https://airbyte.com/tutorials/full-data-synchronization#step-1) - [Custom Transformations in Airbyte](https://docs.airbyte.com/using-airbyte/core-concepts/#custom-transformations) - [Forum Post: MySQL source: loading from SQL query, not tables/views?](https://discuss.airbyte.io/t/620) - [Forum Post: Using dbt tranformation to aggregate data during a sync?](https://discuss.airbyte.io/t/3455)