https://linen.dev logo
Join Slack
Powered by
Hi, I've been trying to debug this issue where I a...
# ask-community-for-troubleshooting
a
Hi, I've been trying to debug this issue where I am trying to run a sync from MSSQL to Snowflake and regardless of table size, it will hang for 30 minutes and then say the sync succeeded with 0 rows added. I've tried using version 2.0.0 and 1.8.5, increased memory of my machine to 16gb as well, but none of this has been working.
k
Based on the information in the knowledge sources, there are several possible causes for your issue where a sync from MSSQL to Snowflake hangs for 30 minutes and then reports success with 0 rows added: 1. Resource Constraints: While you have increased your machine's memory to 16GB, some users have reported that even with sufficient resources, syncs can hang or fail, especially with large numbers of tables or records. However, your issue occurs regardless of table size, so this may not be the root cause in your case, but it's still worth monitoring CPU and memory usage during the sync to rule out resource exhaustion as a factor. Also, ensure that your Airbyte deployment is not hitting any resource limits set in Docker or Kubernetes, if applicable. 2. Connector or Airbyte Version Issues: There are reports of syncs hanging or failing due to bugs in certain Airbyte or connector versions. Upgrading Airbyte and connectors to the latest versions has resolved similar issues for some users. For example, one user resolved a hanging issue by upgrading Airbyte to version 0.39.x. If you haven't already, try upgrading both Airbyte and the MSSQL/Snowflake connectors to the latest available versions and see if the issue persists. If you are already on a recent version, this may not apply, but it's a common troubleshooting step that has worked for others [MSSQL CDC connector hangs for long time](https://discuss.airbyte.io/t/1323). 3. Timeouts and Source Configuration: There are cases where syncs fail or hang due to timeouts in the source configuration, especially with large tables or views. For example, a user syncing a large MSSQL view encountered a JDBC timeout after 5 minutes, even though the server timeout was set to 2 hours. Check if your source connector settings have a timeout parameter and try increasing it. Also, ensure that the user account used by Airbyte has the necessary permissions to read the source tables or views [JDBC Timeout in MSSQL Connection](https://discuss.airbyte.io/t/924). 4. No Data Detected or CDC Configuration: If you are using Change Data Capture (CDC), ensure that CDC is properly enabled and configured on your MSSQL source. There are reports of syncs completing with 0 rows when CDC is not set up correctly or when there are no changes to replicate. Double-check your CDC setup and verify that there are changes to sync. Also, review the Airbyte logs for any errors or warnings related to CDC or data extraction [source-mssql slower speed when no changed data during CDC sync](https://github.com/airbytehq/airbyte/issues/51475). 5. NullPointerException or Intermittent Hangs: Some users have experienced intermittent hangs or NullPointerExceptions during MSSQL to Snowflake syncs. In these cases, rerunning the job sometimes succeeds. If you see similar errors in your logs, consider sharing the complete logs for further diagnosis. As a workaround, some users have implemented external monitoring (e.g., with Airflow) to cancel and rerun jobs that hang for too long [Source MSSQL connector hang forever intermittently (NPE)](https://discuss.airbyte.io/t/1492). Next Steps: - Upgrade Airbyte and connectors to the latest versions. - Check and increase any relevant timeout settings in the source connector. - Ensure CDC is correctly configured if you are using it. - Review Airbyte logs for errors or warnings. - If the issue persists, try running the sync with a single small table to isolate the problem. - If possible, share your Airbyte sync logs for more targeted troubleshooting. If none of these steps resolve your issue, please provide the relevant sync logs, as they can help identify the root cause more precisely. The knowledge sources do not contain a definitive solution for your exact scenario, but these are the most common troubleshooting steps and issues reported by other users.
a
@kapa.ai also limiting the resources in the values.yaml file doesn't seem to work.
k
A message from kapa.ai
There are multiple reports in the knowledge sources that resource limits set in the values.yaml file (or via environment variables like JOB_MAIN_CONTAINER_MEMORY_LIMIT) are not always respected by Airbyte jobs. This issue has been observed across several Airbyte versions and deployment types (Kubernetes, Docker, etc.). Key findings from the sources: - Users have reported that even after setting resource limits in values.yaml, Airbyte jobs may ignore these settings and use default or unexpected values. For example, jobs may request more CPU or memory than specified, or simply ignore the custom settings altogether. This has been seen in both memory and CPU limits and requests, and is a known issue in some Airbyte versions [Sync jobs do not appear to respect CPU / memory requests and limits](https://github.com/airbytehq/airbyte/issues/42921), [Cannot use Airbyte on Kubernetes due to OOM when setting up connector](https://github.com/airbytehq/airbyte/issues/48816), [Airbyte is ignoring 
JOB_MAIN_CONTAINER_MEMORY_REQUEST
in .env](https://github.com/airbytehq/airbyte/issues/30814). - In some cases, users found that after updating values.yaml, they needed to manually delete worker pods for the new resource settings to take effect. This suggests that a rolling restart or manual intervention may be required after changing resource configurations [Sync jobs do not appear to respect CPU / memory requests and limits](https://github.com/airbytehq/airbyte/issues/42921). - There is also a known issue where resource requirements set at the connector definition level (in the actor_definition table) are ignored unless a specific environment variable is set. For Kubernetes Helm deployments, setting the environment variable CONNECTOR_SPECIFIC_RESOURCE_DEFAULTS_ENABLED: "true" in your Helm values.yaml may be necessary for Airbyte to respect connector-specific resource requirements [Syncs don't respect the resource_requirements set in the actor_definition table in the database in latest Airbyte](https://github.com/airbytehq/airbyte/issues/38178). - The documentation confirms that resource requirements can be set at multiple levels (instance-wide, connector definition, connector-specific, connection-specific), but in practice, there have been bugs where only the instance-wide defaults are used, and more specific settings are ignored [Configuring Connector Resources](https://docs.airbyte.com/platform/next/operator-guides/configuring-connector-resources). Summary: It is a known issue that resource limits set in values.yaml are sometimes not respected by Airbyte jobs, depending on the version and deployment method. Workarounds include restarting worker pods after changing resource settings and, for Helm/Kubernetes deployments, setting CONNECTOR_SPECIFIC_RESOURCE_DEFAULTS_ENABLED: "true" to enable connector-specific resource limits. If these steps do not resolve the issue, it may be a bug in your Airbyte version, and upgrading to the latest version or following up on the relevant GitHub issues is recommended. If you need more specific guidance, please provide your Airbyte version, deployment method, and a sample of your values.yaml resource configuration.
a
@kapa.ai this is my values.yaml file. I am on airbyte 2.0.0 using abctl to deploy it
Copy code
jobs:
  resources:
    limits:
      cpu: 250m
      memory: 500m
    requests:
      cpu: 75m
      memory: 150m
k
A message from kapa.ai
Based on the knowledge sources, there are known issues with resource limits and requests set in the 
values.yaml
file not being respected by Airbyte jobs, especially when deploying with Helm or abctl. Several users have reported that even after specifying resource limits and requests (like in your example), Airbyte jobs may ignore these settings and use defaults or unexpected values instead. Key points from the sources: - Resource settings may be ignored: Users have observed that settings in 
values.yaml
for 
jobs.resources
are sometimes not applied to job pods. This has been reported in multiple Airbyte versions, including recent ones. For example, jobs may still request 1 CPU and 1Gi memory even if lower values are set in the config file. This is a known issue and has been discussed in several GitHub issues [Sync jobs do not appear to respect CPU / memory requests and limits](https://github.com/airbytehq/airbyte/issues/42921), [Cannot use Airbyte on Kubernetes due to OOM when setting up connector](https://github.com/airbytehq/airbyte/issues/48816), [Airbyte is ignoring 
JOB_MAIN_CONTAINER_MEMORY_REQUEST
in .env](https://github.com/airbytehq/airbyte/issues/30814). - Manual intervention may be required: In some cases, after updating 
values.yaml
, users needed to manually restart or delete worker pods (such as 
airbyte-workload-launcher
) for the new resource settings to take effect. This is because the environment variables are loaded at pod startup, and changes to the config map or values file are not automatically propagated to running pods [Sync jobs do not appear to respect CPU / memory requests and limits](https://github.com/airbytehq/airbyte/issues/42921), [Node selector and tolerations for job](https://github.com/airbytehq/airbyte/issues/10378). - abctl and Helm chart issues: There are open issues specifically mentioning that the Helm chart (and by extension, abctl which uses it) is broken in certain versions, including 0.30.1 and possibly 2.0.0, with respect to environment variable propagation and resource settings [Helm chart broken with abctl 0.30.1](https://github.com/airbytehq/airbyte/issues/65129). - Multiple environment variables may be needed: For full control, you may need to set not just the generic 
JOB_MAIN_CONTAINER_*
variables, but also job-type-specific variables (e.g., 
CHECK_JOB_MAIN_CONTAINER_MEMORY_LIMIT
, 
DISCOVER_JOB_MAIN_CONTAINER_MEMORY_LIMIT
, etc.) in your 
values.yaml
or environment configuration [Cannot use Airbyte on Kubernetes due to OOM when setting up connector](https://github.com/airbytehq/airbyte/issues/48816). What you can try: 1. After changing your 
values.yaml
, manually restart the 
airbyte-workload-launcher
deployment to ensure new environment variables are picked up. 2. Consider explicitly setting all relevant job-type-specific resource variables if you need fine-grained control. 3. If the issue persists, it may be a bug in the version you are using. Check for updates or open issues related to resource configuration in Airbyte 2.0.0 and abctl. Summary: Your configuration is correct, but due to known issues, Airbyte may not respect these settings unless you restart the relevant pods and possibly set additional environment variables. If these steps do not resolve the issue, it is likely a bug in the current version or deployment method you are using [Helm chart broken with abctl 0.30.1](https://github.com/airbytehq/airbyte/issues/65129), [Node selector and tolerations for job](https://github.com/airbytehq/airbyte/issues/10378), [Cannot use Airbyte on Kubernetes due to OOM when setting up connector](https://github.com/airbytehq/airbyte/issues/48816). If you need more targeted help, please provide logs from your job pods and confirm if you have tried restarting the workload launcher after updating your configuration.
Open in Slack
PreviousNext
Powered by