I understand the Shopify Source is unsupported. We...
# replication-troubleshooting
j
I understand the Shopify Source is unsupported. We’ve been unable to perform a single sync with production volumes. After multiple 23-26h runtimes they all fail with similar reasons. Wanted to provide the logs in case they help. Also we currently only have 1.7M populated users out of expected 10M, so fetching Metadata 1 at time is painful. I believe the customers object fetch has metafields already.
s
Hey Jon, sorry to hear you’re having issues with the Shopify connector. How much data are you extracting from shopify when you say production volumes? The sync worker stopped working is a generic error so it’s a bit hard to debug from a first glance. Could you describe what your airbyte deployment looks like? As far as the performance issue with fetching, it’s possible that some streams could be more written more efficiently. We’re open to contributions there if folks have ideas.
j
It’s an initial sync, it looks like it finishes each stream and commits it to our destination but then fails. Subsequent syncs start all over again. There’s ~1.7M customers with 2 metafields each. Very few of everything else. Maybe 100 products, sample orders. We’re not in production but from the Github issues I’ve seen the largest test I saw references was like 200k customers.
u
Hey Jon, what destination are you loading data into? It could potentially be a destination issue as well.
j
Redshift directly, without the S3 copy configuration
u
Hello Jon Simpson, it's been a while without an update from us. Are you still having problems or did you find a solution?