Hi community, I am having a question: we have connection to 3rd party Snowflake instance which have many tables with a lot of columns. We are interested in only 10 tables and subset of these columns in there. Currently, we are having a separate python connector and running sql query to make joins and extract 1 final table. With Airbyte we would copy all 10 tables to destination and run sql there, which is not optimal, because then we would store unnecessary information + sensitive data. We are still willing to use airbyte for standardisation. Does it make sense to build a customer connector which would do the transformations before uploading everything into destination? I know that it is not ELT anymore, but still. Do you have any other recommendations on how to treat this differently? Thanks in advance!

What about using custom dbt to make the transformations and at the end remove the sensitive data?

do you mean already in destination or in source? Since it is 3rd party source we can not change it or install integrations

in the destination, you transfer all data use dbt to transform and remove any sensitive data

Denis, let me know if Marcos' suggestion helps!

Thank you very much for your replies 🙂 Yes, this strategy I know, it is just I really do not want to store/transfer 10 tables with many columns that I do not need. It is also expensive to do regular updates for this kind of data, I would prefer to have just part which I need and do transfromation before.

Unfortunately Airbyte only does transformations after moving the data!

Would you recommend any product/tools that could do ETL and are similar ideologically to airbyte?

I cannot recommend another tool, unfortunately! Sorry for the inconvenience!

Thank you!