Gopinath Jaganmohan

01/03/2022, 1:13 AM
Hello Team, I was evaluating Airbyte Integration for our product. My setup and first run was easy and success full. One of the requirement we have is to push down SQL queries(run custom queries) on SQL based DBs. Is there way we can do that now?

Davin Chia (Airbyte)

01/03/2022, 7:14 AM
What do you mean by push down SQL queries?

Gopinath Jaganmohan

01/03/2022, 7:55 AM
I guess he wants the flexibility to define a custom query and filter the results from the source and not the entire table

Vladimir Klimontovich

01/03/2022, 11:01 AM
Probably you can create a view? Would airbyte allow to sync views along with the tables?

Johan Wärlander

01/03/2022, 2:32 PM
Hi @Gopinath Jaganmohan could you please be more specific about what you'd like to achieve. As suggested by @Vladimir Klimontovich if you'd like to model / filter your input data I'd suggest using a view. If you're more interested into downstream transformation we offer custom transformation with DBT, we've a guide here.
Thanks all for the suggestions. Following are my requirements 1. We connect to customer database ERP systems. 2. Database might 100s of tables with history of couple of years 3. We have built queries that we run on top of these 100 tables and extract data and load into our system 4. And setup Sync for these in set frequencies. 5. We can't ask our customer to create views, we can only run sql Instead of just syncing the table, run custom query and then extract and load
Alright @Gopinath Jaganmohan, my suggestion would be to ingest the raw tables and perform the custom transformation in DBT, migrating your existing SQL query to DBT and performing them on your destination warehouse. The guide I linked above will help you set this up.
Thanks Augustin. In my search, I did came through this. Since my requirement was different, thought of checking with community