Hi all. I am new to airbyte and require your help. I am working on an datawarehouse project initiative in my company. In short, we are creating a datawarehouse (batch based 1 day latency) for our product customers (10000) where we will get data from 5 different products (total 10000 customer on premise sources). We have evaluated snowflake or bigquery and it is right fit based on our scale and usecase. We are evaluating airbyte tool to get data from 10000 on premise customers ( 5 unique products) . Source dB are SQL server , sybase. We want to go with push mechanism so that 10000 customers can push data from on premise SQL server dB or Sybase dB to snowflake dB. This will help us distribute compute. Does airbyte has any agent for on premise dB to cloud DW connectivity? Any other method to achieve push mechanism. As per our understanding, elt doing pull from 10000 sources wouldn't be feasible

Hi Amit! I'm not sure I quite understand what you are asking, could you clarify? We have connectors for various databases and datawarehouses, have you seen the list in the documentation? https://docs.airbyte.com/integrations/ By push mechanism, do you mean scheduling? Here are docs on that: https://docs.airbyte.com/understanding-airbyte/connections/ Here's a doc on scaling that would probably be useful in your case: https://docs.airbyte.com/operator-guides/scaling-airbyte

Hi Nataly. I have to get the data from 10000 sources. So as per my understanding, if the source system can connect to an agent which can initiate a call and push the data from source db to cloud db. This way , in my 10000 source systems , the call is made at source to puch data. This will distribute the compute and I don't have to initiate the call to 10000 systems from my ELT. Some of the ETL or ELT provide such agents to connect on premise source database to cloud DW

I don't believe Airbyte supports this feature at this point in time!

Do you think I can achieve this with scaling option that you have mentioned. https://docs.airbyte.com/operator-guides/scaling-airbyte

I think it depends - there are several different things you can scale when deploying Airbyte.