Having an issue with the

destination-aws-datalake

connector. We have a table that’s more than 15+ million rows and whenever we try to run a sync using that connector, the docker container that gets spun out to run the sync ends up chewing all the host memory and eventually crashing the EC2 instance. Tried setting some global docker limits (as i can’t do it on the container level since it gets spun up when the sync job starts) to no success as

docker stats

kept showing the available memory on the host. Tried resizing the instance and even with 64GB of RAM on the host, still ran into the same issue. https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-aws-datalake Anyone run into something similar using that connector and how did you end up resolving it? Here’s a bit more info on the issue itself - https://github.com/mitodl/ol-data-platform/issues/371

@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2812 to ensure timely resolution!

Hey @sar thanks for the question. Could you share any logs and some more info about the tables that fail to sync? (Number of records, average row size, etc)