Hi! How can I add custom python script to transfor...
# ask-community-for-troubleshootingv
Vladimir Kochetkov
10/12/2021, 8:51 AMHi! How can I add custom python script to transform data before loading to destination?
✅ 1
g
George Claireaux (Airbyte)
10/13/2021, 6:20 PM
Hi @Vladimir Kochetkov, Airbyte operates as an ELT flow rather than ETL, so data is loaded to the destination and then transforms are applied.
You can set up custom transforms using dbt like so 🙂
v
Vladimir Kochetkov
10/14/2021, 6:14 AMHi @George Claireaux (Airbyte), thanks for your answer!
It is ok for me to do transformation on last step after load, but I need to run python script to transform data - not dbt sql.
Vladimir Kochetkov
10/14/2021, 6:15 AMMain reason - complicated logic like lemmatization. As I know - dbt is not for this.
g
George Claireaux (Airbyte)
10/14/2021, 8:39 AM
In that case, you'll probably want to orchestrate the flow with something like Airflow. You can trigger the airbyte sync and then call a python operator to perform your transformations. Airbyte is more specifically targeted at EL (and T with dbt) rather than being a generic orchestration tool.
v
Vladimir Kochetkov
10/15/2021, 6:43 AM@George Claireaux (Airbyte) thank you!
3 Views