Does anyone know what happens when you have multiple sources pointing to the same destination? Specifically in our case we want to set things as full_refresh/overwrite, but I don't know if things will overwrite each other.
m
Marcos Marx (Airbyte)
12/06/2022, 5:34 PM
They are the same Source connector using the same Namespace?
e
Ethan Brouwer
12/06/2022, 7:02 PM
Basically, to speed up extraction and deal with some issues we were seeing with 400 Million rows, we set up 12 sources on the same bucket pointing to different prefixes (different months). Then we have them all filter into the same redshift destination and don't alter the namespace at all because it is really nice to have them all in the same table.
a
Avi Sagal
12/06/2022, 7:09 PM
so we tried to do something like that with 2 google analytics sources to the same table in the postgres destination and got into a little bit of an issue when we reset one one of the connections.
this erases all of the data in destination table (including the data that came from the second connection), I’m
not sure if its a specific case with the postgres destination but its something worth to check:)
e
Ethan Brouwer
12/06/2022, 10:57 PM
Yeah, that makes a lot of sense. I just wish the S3 -> Redshift didn't take so long or if there was a better way to parallelize it.