What is the proper way to set up a Stream when the...
# connector-developmenta
Alessandro Duico
02/02/2022, 10:48 AMWhat is the proper way to set up a Stream when there are multiple clients whose data need to be fetched? I was thinking of having stream_state store a list[dict] with a different "last_fetched_record_date" for every customer.
Is there a better way to achieve this?
Alessandro Duico
02/02/2022, 11:04 AMHey wouldn't it be better if you can create multiple connections in airbyte rather handling it over in the connector
Alessandro Duico
02/02/2022, 12:31 PMCan multiple connections output to the same BigQuery destination table?
d
Daniel Eduardo Portugal Revilla
02/02/2022, 3:23 PMSorry same table would be a chaos I think. Unless their catalogs are different if catalogs are same then it is more chaos
Daniel Eduardo Portugal Revilla
02/02/2022, 3:52 PMOk. With this in mind, what would be the best approach?
Storing a list[dict] in stream_state with a different "last_fetched_record_date" for every customer?
Daniel Eduardo Portugal Revilla
02/02/2022, 3:57 PMI think you have a prefix of based on customer_id when you create the connection and later bringing all together into one table can be done in custom DBT. I think this would be good approach
Daniel Eduardo Portugal Revilla
02/02/2022, 4:23 PMI see, we would like to fetch all the data in a single run, though
Daniel Eduardo Portugal Revilla
02/02/2022, 4:24 PMBecause many new customers might be added, and that would mean a lot of sources have to be set up
Daniel Eduardo Portugal Revilla
02/02/2022, 4:53 PMThen yeah I think you have use custom connector and try doing the same you have already idea of
2 Views