i wrote a connector for klaviyo using nodejs (https://hub.docker.com/r/neonblueai/airbyte-klaviyo-source) and note that it syncs ~500M events per day (compared to the official klaviyo connector at https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-klaviyo which i benchmarked at.... 5M per day). i do this by pipelining the requests and buffering disk writes to saturate the theoretical api limit. also added oauth support for authentication. is this something that would be useful to the community? (not sure how to handle improvements to connectors that airbyte already supports. cpu stays cool and memory usage <500MB while doing it (whereas the official one seems quite hungry for resources).
😍 1
u
[DEPRECATED] Marcos Marx
10/22/2024, 9:12 PM
Hey Jasper this is amazing airbyte rocket . Probably adding nodejs to the stack isn’t something the team will accept now, but I’d recommend to share your code/flow and open a discussion in Github to see what it is possible to move to the official connector.