Title
#advice-data-transformation
Moiz Sherwani

Moiz Sherwani

06/30/2022, 11:39 PM
Hello everyone, newbie to airbyte. We have a simple requirement of taking a postgres database table, performing some transformations and then saving the result as another table. I have set the source and destination but cannot figure out how to do the transformations. Can someone please provide me a tutorial video or simple instructions on how to do this (for context, I am also new to github so high level instructions, like write code in github unfortunately won't cut it for me) so as simple as you can keep it. e.g. source table has 100 rows with ID 1 to 100 and destination has to have only the first 10 rows (ID = 1 to 10)
a

Adam Bloom

06/30/2022, 11:55 PM
The destination table is in the same database as the source table?
Moiz Sherwani

Moiz Sherwani

07/03/2022, 11:07 AM
well it is in this case (it might not be in the future) but you can, to keep in simple, assume it to be the same
a

Adam Bloom

07/05/2022, 2:59 PM
In the spirit of simplicity, I'd ask you to think about why you want to use airbyte for this. Airbyte's strength is in connecting different sources and destinations, which is not what you're doing here. That strength comes along with deployment complexities (nothing obviously unsolvable, but just more work!). You're asking about transformations: airbyte uses
dbt
to power its transformations. You can, however, just use
dbt
on its own. Note that the tutorials refer to bigquery/redshift/snowflake/etc, but dbt supports postgres as well: https://docs.getdbt.com/guides/getting-started/learning-more/getting-started-dbt-core What you've described is defining a single dbt model for the destination table.
Moiz Sherwani

Moiz Sherwani

07/06/2022, 2:40 AM
Thanks for your reply Adam, so there isn't any GUI / interface that can be used to be these transformation (for e.g. the ones which are available in the likes of SSIS/SSAS, Talend, Pentaho)?
a

Adam Bloom

07/06/2022, 3:37 PM
I'm not familiar with those other tools so I can't comment on that