Alessandro Duico

02/02/2022, 10:48 AM
What is the proper way to set up a Stream when there are multiple clients whose data need to be fetched? I was thinking of having stream_state store a list[dict] with a different "last_fetched_record_date" for every customer. Is there a better way to achieve this?
Hey wouldn't it be better if you can create multiple connections in airbyte rather handling it over in the connector
Can multiple connections output to the same BigQuery destination table?

Daniel Eduardo Portugal Revilla

02/02/2022, 3:23 PM
Sorry same table would be a chaos I think. Unless their catalogs are different if catalogs are same then it is more chaos
Ok. With this in mind, what would be the best approach? Storing a list[dict] in stream_state with a different "last_fetched_record_date" for every customer?
I think you have a prefix of based on customer_id when you create the connection and later bringing all together into one table can be done in custom DBT. I think this would be good approach
I see, we would like to fetch all the data in a single run, though
Because many new customers might be added, and that would mean a lot of sources have to be set up
Then yeah I think you have use custom connector and try doing the same you have already idea of