Hello Team,
I was evaluating Airbyte Integration f...
# contributing-to-airbyteg
Gopinath Jaganmohan
01/03/2022, 1:13 AMHello Team,
I was evaluating Airbyte Integration for our product. My setup and first run was easy and success full. One of the requirement we have is to push down SQL queries(run custom queries) on SQL based DBs. Is there way we can do that now?
d
Davin Chia (Airbyte)
01/03/2022, 7:14 AM
What do you mean by push down SQL queries?
g
Gopinath Jaganmohan
01/03/2022, 7:55 AMI guess he wants the flexibility to define a custom query and filter the results from the source and not the entire table
v
Vladimir Klimontovich
01/03/2022, 11:01 AMProbably you can create a view? Would airbyte allow to sync views along with the tables?
j
Johan Wärlander
01/03/2022, 2:32 PMHi @Gopinath Jaganmohan could you please be more specific about what you'd like to achieve. As suggested by @Vladimir Klimontovich if you'd like to model / filter your input data I'd suggest using a view. If you're more interested into downstream transformation we offer custom transformation with DBT, we've a guide here.
Johan Wärlander
01/03/2022, 4:22 PMThanks all for the suggestions. Following are my requirements
1. We connect to customer database ERP systems.
2. Database might 100s of tables with history of couple of years
3. We have built queries that we run on top of these 100 tables and extract data and load into our system
4. And setup Sync for these in set frequencies.
5. We can't ask our customer to create views, we can only run sql
Instead of just syncing the table, run custom query and then extract and load
Johan Wärlander
01/03/2022, 4:30 PMAlright @Gopinath Jaganmohan, my suggestion would be to ingest the raw tables and perform the custom transformation in DBT, migrating your existing SQL query to DBT and performing them on your destination warehouse. The guide I linked above will help you set this up.
Johan Wärlander
01/03/2022, 5:50 PMThanks Augustin. In my search, I did came through this. Since my requirement was different, thought of checking with community