how to make a custom transformation in Airbyte

Creating custom transformations in Airbyte depends on whether you are using Airbyte Cloud or the open-source version. Here are the steps for each: ### Airbyte Cloud Airbyte Cloud integrates natively with dbt to allow you to use dbt for post-sync transformations. Here’s how you can set up a custom transformation: 1. Generate a Service Token: - Generate a [service token](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens#generating-service-account-tokens) for your dbt Cloud transformation. - Ensure you are using a paid version of dbt Cloud and that the service token has Member, Job Admin, or Account Admin permissions. 2. Set Up the dbt Cloud Integration in Airbyte Cloud: - In the Airbyte UI, click Settings. - Click Integrations. - Paste the service token and click Save changes. - Click Connections and select the connection you want to add a dbt transformation to. - Go to the Transformation tab and click + Add transformation. - Select the transformation from the dropdown and click Save changes. The transformation will run during subsequent syncs until you remove it. - You can have multiple transformations per connection. - To remove a transformation, click X on the transformation and click Save changes. For more detailed steps, refer to the [Airbyte Cloud dbt integration documentation](https://docs.airbyte.com/cloud/managing-airbyte-cloud/dbt-cloud-integration). ### Airbyte Open-Source Custom transformations are not available for Airbyte Open-Source. According to the [core concepts documentation](https://docs.airbyte.com/using-airbyte/core-concepts#custom-transformations), custom transformation is a feature exclusive to Airbyte Cloud. ### Deprecated Features It's important to note that normalization and custom transformation features are deprecated and will be removed. For more information, you can visit the [discussion on GitHub](https://github.com/airbytehq/airbyte/discussions/34860). ### Example of Using a Custom Run with Variables If you want to use a custom run and pass variables, you can use the following syntax:

bash run --vars '{"table_name":"sample","schema_name":"other_value"}'

This string must have no spaces. For more details, you can refer to the [custom run with variables documentation](https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-airbyte#using-a-custom-run-with-variables). If you have any further questions or need more detailed guidance, feel free to ask!

@kapa.ai can i refactor code for adding custom transformation to another destination?

It appears that the custom dbt transformation feature has been deprecated in Airbyte, as indicated in multiple sources: 1. Issue #29775: The feature was deprecated, and the issue was closed as "won't do" ([source](https://github.com/airbytehq/airbyte/issues/29775)). 2. Issue #5091: The feature was deprecated, and the issue was closed as "won't do" ([source](https://github.com/airbytehq/airbyte/issues/5091)). Given this deprecation, it is not recommended to refactor or add custom transformations to another destination using the deprecated feature. Instead, you might want to explore alternative methods or tools for handling transformations. For more information on the deprecation and potential alternatives, you can refer to the discussion linked in the issues: - [Discussion on deprecating custom dbt transformation](https://github.com/airbytehq/airbyte/discussions/34860) If you are looking for current methods to handle transformations, you might want to explore the dbt Cloud integration in Airbyte Cloud, which is still supported. Here are the steps to set it up: 1. Click Settings and then Integrations. Enter your service token and click Save changes. 2. Click Connections and select the connection you want to add a dbt transformation to. Go to the Transformation tab and click + Add transformation. 3. Select the transformation from the dropdown and click Save changes. The transformation will run after the subsequent syncs until you remove it. You can repeat these steps to add additional transformations for a connection. 4. To remove a transformation, click X on the transformation and click Save changes. For detailed instructions, you can refer to the [dbt Cloud integration documentation](https://docs.airbyte.com/cloud/managing-airbyte-cloud/dbt-cloud-integration#step-2-set-up-the-dbt-cloud-integration-in-airbyte-cloud). If you have any further questions or need more specific guidance, feel free to ask!