A question about best to approach structuring a solution - any thoughts welcome. Basic problem is a situation where daily data is on one endpoint and historical data on another, and how to go about merging the data. 1. One API gives access to all data for every stock in a stock exchange (eg NYSE). A date can be given as a query parameter. 2. Another API gives all historic data for one stock. You'd need to iterate over all stocks in an exchange to get all historic data. To backfill the database I could call API 1 from an initial start date in the 70s - but this will hit daily API limits very quickly. They allow 1000 (exchange, date) pairs per day (which is fine for daily updates). Backfill via the API 2 would mean a call for every possible stock which would be much easier on the rate limiting. But, that API isn't suitable for daily updates, that's the purpose of API1. Any suggestions on how go about backfilling the DB such that everything is compatible with incremental sync for the daily updates?

Hi. Multi threading with asyncio (or any other library). 1st api is for daily data which max daily limit is 1k stocks. 2nd api - for each exchange return all it`s historical for each stock in create thread that can be added to the async api call task and while getting another stock. run threaded ->>> get stock ->>> while you wait for 2nd stock run in thread api call for history data (if ALL the history is in 1 api call then while you wait for the response go to the 2nd stock history api call) (if the history is 1 api call per date then asyncio it) ->>> while you wait for the response start the 2nd thread to run also the history since by the time you started from the stock api to the history you already got the 2nd stock (make sure every thing is async so you wont be blocked) 1 function for the threaded api calls for the 1st api call 1 function for the threaded history api call 1 function to save data as you get it and 1 function to control the flow of things (I like this one makes it easy to 1 change control all logic) last function to stream the data to db (depends on db api limits ofcourse) so all in async and threaded

That sounds like a solution outside of Airbyte? I was asking about how to structure things using Airbyte. So one logical destination but two source APIs, one for history (one stock, all dates) the other for daily (all stocks, one date). And how to make things consistent with Airbyte records that deal with state, incremental updates, etc

since airbyte already know how to do parallel , threaded and async logic you just need an orchestration tool such as apache airflow for example. you can create parallel and async tasks in it. basically the same logic using airflow + airbyte

Hello Jonny Wray, it's been a while without an update from us. Are you still having problems or did you find a solution?