hey everyone; first off sorry if this isn't the right place, but I'm running into some roadblocks after installing airbyte and would love any suggestions.. I've got a connection setup with a

postgres

source and

snowflake

destination. Trying to do an initial sync (incremental | append + deduped). on a fairly narrow table with ~1mil rows. I've tried restarting a few times, and airbyte keeps getting 'stuck' part-way through the sync. I'm using a fairly beefy 16core 64gb ram ec2 instance, and it looks healthy. The last couple lines in the sync logs are like:

2025-10-15 17:53:04 source INFO main i.a.i.s.p.c.CtidStateManager(generateStateMessageAtCheckpoint):79 Emitting ctid state for stream public_accounts, state is io.airbyte.integrations.source.postgres.internal.models.CtidStatus@4a9486c0[version=2,stateType=ctid,ctid=(5996,66),incrementalState={"state_type":"cursor_based","version":2,"stream_name":"accounts","stream_namespace":"public","cursor_field":["id"],"cursor":"1057496","cursor_record_count":1},relationFilenode=54853845,additionalProperties={}]
2025-10-15 17:53:04 source INFO main i.a.c.i.s.r.s.SourceStateIterator(computeNext):50 sending state message, with count per stream: {public_accounts=10000} 
2025-10-15 17:53:05 replication-orchestrator INFO Records read: 150000 (265 MB)
2025-10-15 17:53:07 replication-orchestrator INFO Records read: 155000 (274 MB)

and then nothing. There is no query running on postgres, nor on snowflake, so I imagine something is getting stalled out within airbyte. I wonder if it's possible there's something in the data that's causing it to fail? I can't find any additional logs, so will look into somehow setting up airbyte to produce more logs, but am somewhat out of ideas at the moment. This is my first experience/sync with airbyte, so feeling a bit disheartened at this point 💔

I'm sorry you're running into this issue—syncs getting "stuck" part-way through is a problem that has come up for other Airbyte users, especially with Postgres → Snowflake connections. Based on the knowledge sources, here are some possible causes and troubleshooting steps: 1. Resource Limits & Buffering Even with a powerful EC2 instance, syncs can get stuck if there are bottlenecks in how Airbyte handles memory, buffering, or if the destination (Snowflake) is slow to flush data. For example, one user found that increasing the buffer parameter in the Snowflake destination to 50 helped resolve a similar issue where the sync would never finish and files would never be copied to the target tables. Also, limiting the number of tables per connection (ideally ≤30) can help if you have many tables, though your case is a single table, so this is less likely the issue. Upgrading both source and destination connectors to the latest versions is also recommended, as performance and stability improvements are frequent in new releases. See: [Postgres CDC connector stuck](https://discuss.airbyte.io/t/4373) and [Scaling Airbyte](https://docs.airbyte.com/operator-guides/scaling-airbyte/). 2. Destination Backpressure If the destination (Snowflake) is slow to process incoming data, it can cause the sync to stall. This is often visible as a sudden drop in throughput or the process hanging after a certain number of records. Monitoring Snowflake's logs for timeouts or errors can provide more insight. One user reported that increasing the resources (RAM/CPU) did not help, but adjusting buffer parameters and checking for destination-side timeouts did. See: [File Source → Snowflake Destination stuck](https://discuss.airbyte.io/t/725). 3. Airbyte Internal State/Checkpoints Airbyte emits state messages during syncs, and sometimes issues with state emission or checkpointing can cause the sync to hang, especially with large tables or CDC. If the sync always stalls at the same record count, it could be related to a specific data value or a bug in state management. Reviewing the full logs (not just the last lines) may reveal more clues. If possible, try syncing a subset of the data (e.g., a smaller table or a view with fewer rows) to see if the problem persists. See: [Initial load of huge table fails - Postgres Connector](https://discuss.airbyte.io/t/3694). 4. Disk Space and Workspace Location Ensure that the Airbyte workspace (WORKSPACE_ROOT) is on a partition with sufficient space. Running out of disk space can cause syncs to silently fail or hang. One user resolved a similar issue by moving the workspace to a larger partition. See: [Postgres to Snowflake only sync part of the data](https://discuss.airbyte.io/t/2188). 5. Logs and Debugging If logs are not showing enough detail, you can increase Airbyte's log verbosity. Also, check both the Airbyte server logs and the logs for the specific sync job. If you are running Airbyte in Docker, you can use

docker logs <container_id>

to get more details. Next Steps: - Upgrade both your Postgres source and Snowflake destination connectors to the latest versions. - Try increasing the buffer parameter in the Snowflake destination to 50. - Check disk space on your Airbyte workspace. - Try syncing a smaller subset of your data to isolate if a specific record is causing the issue. - Review full logs for any hidden errors or clues. - If possible, try syncing to a different destination (e.g., S3 or Redshift) to see if the issue is specific to Snowflake. If after these steps the issue persists, please consider sharing the full sync logs (with sensitive info redacted) on the Airbyte forum or GitHub for more targeted help. Let me know if you need guidance on any of these steps!

hi! Did you check the pods logs ?

hi @Camille Lemaitre! I couldn't even figure out how to do this; I was using abctl, which made it a bit challenging to get in and find logs, etc. I'm restarting using the helm approach, hoping that will be easier to debug, and also maybe won't have the same issues if I can tweak memory settings etc more easily. Does that sound like a reasonable approach?