<@U04S8PLJWMT> im on self hosted airbyte version 0...
# ask-ai
d
@kapa.ai im on self hosted airbyte version 0.63.13 recently I have been getting the error
eadFromSource: exception caught
java.lang.IllegalStateException: Source process is still alive, cannot retrieve exit value.
The source is a MSSQL database and the destination is S3. It all started when i had wrong permissions on the database and when that was fixed i started getting these errors. Its reading almost every try between 300MB-370MB. What could be the issue?
k
Based on the information in the knowledge sources, the error message you are seeing:
Copy code
readFromSource: exception caught java.lang.IllegalStateException: Source process is still alive, cannot retrieve exit value.
is closely related to a class of issues where Airbyte is unable to properly stop or retrieve the status of the source connector process. This is often accompanied by errors like "No exit code found" or "Source cannot be stopped!" and is seen in various connectors, including MSSQL and others. From the forum and GitHub issues, several possible causes and contributing factors are mentioned: 1. Resource Constraints or Process Hangs: In some cases, the source connector process may hang or fail to terminate properly, especially after handling large data volumes or after encountering previous errors. This can prevent Airbyte from retrieving the exit code, resulting in the error you see. For example, in one case with large batch updates in MSSQL, the process failed to close properly after reading was finished, leading to a similar error ([Forum Post: Source MSSQL - CDC incremental load failed](https://discuss.airbyte.io/t/4013)). 2. Previous Permission Issues: Since you mentioned that the issue started after fixing database permissions, it's possible that the connector process encountered an unrecoverable state during the period of incorrect permissions. Even after fixing the permissions, the connector or Airbyte worker may be in a "bad state" or have cached/partial state from previous failed runs. 3. Large Data Volumes: The logs in similar issues show that when syncing large amounts of data (hundreds of MBs or more), the process is more likely to hang or fail to terminate cleanly, especially if there are resource constraints (memory, CPU) or network issues ([destination-s3-data-lake sync stalling](https://github.com/airbytehq/airbyte/issues/56926)). What you can try: - Restart Airbyte and the Connector Pods/Containers: Fully restart your Airbyte instance and ensure all connector containers are stopped and restarted. This can clear any lingering processes or bad state. - Check Resource Allocation: Ensure your Airbyte deployment (especially if on Kubernetes or Docker) has sufficient memory and CPU allocated, as resource exhaustion can cause connector processes to hang. - Review Database Permissions: Double-check that the MSSQL user has all required permissions for both reading data and any CDC (Change Data Capture) operations, if used. - Check for Stuck or Zombie Processes: If you have shell access to the host, check for any lingering connector processes that may not have terminated and kill them if necessary. - Review Airbyte and Connector Logs: Look for any additional errors or stack traces in the logs that might indicate why the process is not terminating. If the problem persists after these steps, it may be a bug or edge case in the connector or Airbyte platform itself, especially given the similarity to other reported issues. In that case, consider opening a GitHub issue with detailed logs and your configuration, referencing similar issues for context ([example issue](https://discuss.airbyte.io/t/4013)). I'm sorry, but the knowledge sources do not provide a definitive root cause or fix for this exact error, but the above steps are based on closely related issues and recommended troubleshooting actions.
d
@kapa.ai I see that the CPU is hitting a 100% would a restart fix the fact that its reading too much data or do i have to upstep the ec2 instance?
k
A message from kapa.ai
If your CPU is hitting 100%, a simple restart may temporarily relieve the issue, but it will not address the underlying cause—Airbyte is likely under-provisioned for your workload. High CPU usage can cause sync jobs to hang, fail, or behave unpredictably, especially when processing large amounts of data. The recommended approach is to increase the resources available to your Airbyte deployment. Specifically, you should consider scaling up your EC2 instance to one with more CPU and memory. Airbyte documentation suggests starting with a mid-sized instance (e.g., 4 or 8 cores) and tuning upwards as needed. If your workload has grown or your syncs are more demanding, scaling up is the best long-term solution. Restarting may only provide a short-term fix and the problem will likely recur under load [Scaling Airbyte After Installation](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale). Additionally, you can adjust resource limits for connector pods in your configuration (e.g.,
values.yaml
) to better match your workload, but ultimately, if the instance is maxed out, increasing instance size is necessary [Scaling Airbyte After Installation](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte). In summary: - Restarting may help briefly, but will not solve high CPU usage. - To reliably handle large data volumes, you should upsize your EC2 instance and/or adjust Airbyte resource settings. - Monitor CPU and memory usage, and scale resources as your data integration needs grow.
d
@kapa.ai But before this database change it wasnt like this it was always running for a few minutes while not it looks like its trying to do a full load or somethign
k
A message from kapa.ai
Based on the knowledge sources, if your syncs were previously running quickly and now appear to be doing a full load (full refresh) after a database change, there are a few possible explanations: 1. Incremental Sync Fallback: If Airbyte cannot perform an incremental sync (for example, if the cursor field is missing, permissions are insufficient, or the state is lost), it may fall back to a full refresh, which reads all data from the source. This can happen if the connector loses track of the last synced state, or if there was a reset or error that caused Airbyte to start over from scratch. This is consistent with your observation that it now reads a large amount of data every time, rather than just the changes. 2. Permission or State Issues: The knowledge sources mention that insufficient privileges or issues with the state can cause Airbyte to re-read all data from the source, even in incremental mode. For example, if the state file is lost or corrupted, or if the connector cannot access the necessary metadata to track changes, it will default to a full load ([How to fetch only updated data from source database?](https://discuss.airbyte.io/t/2166)). 3. Connector or Configuration Change: If the database change affected the structure of the tables, the cursor field, or the permissions, Airbyte may not be able to perform incremental syncs as before. This could also happen if the connector version was changed or reset. What you can do: - Check your connection settings in Airbyte to ensure that incremental sync is still selected and properly configured (with a valid cursor field). - Verify that the database user has all necessary permissions for incremental sync (especially if using CDC or similar features). - Review the Airbyte logs for messages about state resets, missing cursor fields, or fallback to full refresh. - If you recently reset the connection or changed the schema, Airbyte may need to perform a full refresh once, but subsequent syncs should be incremental if everything is configured correctly. If the issue persists after verifying these settings, it may be necessary to further investigate the connector logs or open a support ticket with detailed information. The knowledge sources do not provide a more specific root cause for your scenario, but the above points are consistent with known behaviors when Airbyte unexpectedly performs full loads instead of incremental syncs ([How to fetch only updated data from source database?](https://discuss.airbyte.io/t/2166)).