Anyone have success scaling Klaviyo connectors? We ingest data from multiple customers, and some companies have Events in the ~10TB range. Some of these syncs have been running for months and still aren’t caught up to the present day. Fivetran was able to sync everything in about 2 weeks time, so I’m wondering what kind of optimizations I can make (I really don’t want to manage querying the API directly). At first, I was running airbyte on a server, but moved to k8s recently. Any tips on scaling would be appreciated. Already read all of the docs, tried changing the memory configs, editting the variables for the number of sync workers, etc. Not sure what I’m doing wrong, because I haven’t seen any speedups here — but maybe it’s an issue specifically with the airbyte Klaviyo connector?

New message text here

Have you been able to fix a solution for this @zackgow? interesting thing I found was to also increase the following:

worker:
  replicaCount: 2
  # helps in keeping workers informed about pipelines.
  # they are the messengers
  maxNotifyWorkers: 20

I fixed it, but am not sure exactly what the initial issue was or what I did that fixed it 🤣