https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> sources with large schema fail to r...
# ask-ai
i
@kapa.ai sources with large schema fail to refresh ? Happening on kubernetes setup airbyte 1.1.0
k
Based on the knowledge sources provided, it appears that the issue of sources with large schemas failing to refresh is a known problem in Airbyte, including on Kubernetes setups. This issue is still present in recent versions of Airbyte, including version 1.1.0. The primary cause of this problem is that large schemas exceed the maximum allowed content length for API requests when fetching or writing the discovered catalog. This results in a "413 Request Entity Too Large" error. Here are some key points and potential solutions: 1. The error occurs during schema discovery, particularly for databases with a large number of tables or columns. 2. The default maximum allowed content length is 10485760 bytes (about 10 MB), which can be insufficient for large schemas. 3. Some users have reported success by increasing the 
MICRONAUT_SERVER_MAX_REQUEST_SIZE
environment variable. For example:
Copy code
yaml    environment:      - MICRONAUT_SERVER_MAX_REQUEST_SIZE=104857600
This increases the limit to about 100 MB. 4. For Kubernetes deployments, you may need to modify your values.yaml file:
Copy code
yaml    global:      env_vars:        MICRONAUT_SERVER_MAX_REQUEST_SIZE: "104857600"
5. In some cases, increasing the nginx 
client_max_body_size
setting may also be necessary. 6. For very large schemas, you might need to consider splitting your source into multiple connections with fewer tables each. It's worth noting that the Airbyte team is aware of this issue and is working on improvements for handling large catalogs, but as of the latest information available in the knowledge sources, a comprehensive fix has not yet been implemented. For more detailed information and discussion on this issue, you can refer to these GitHub issues: [Large schema fetching failure](https://github.com/airbytehq/airbyte/issues/4564) [Source Oracle - Can't save connection, error "Request Entity Too Large"](https://github.com/airbytehq/airbyte/issues/22631) If these solutions don't resolve your issue, you may need to reach out to Airbyte support for further assistance, as the problem could be specific to your particular setup or schema size.
i
@kapa.ai what does this error mean ? Client error : 410 Gone {"message":"Heartbeat a workload in a terminal state"}
k
A message from kapa.ai
This error message is related to Airbyte's heartbeat mechanism, which is used to monitor the health and responsiveness of various components during data synchronization. Let me explain what this specific error means based on the information from our knowledge sources. The error "Client error : 410 Gone {"message":"Heartbeat a workload in a terminal state"}" indicates that the system attempted to send a heartbeat for a workload (a task or job) that has already reached a terminal state (completed, failed, or cancelled). According to the [Airbyte Heartbeats documentation](https://docs.airbyte.com/understanding-airbyte/heartbeats), heartbeats are used to detect unresponsive sources or destinations. When a component doesn't send a heartbeat within a certain timeframe, Airbyte may trigger a restart of the job. In this case, the 410 Gone status suggests that the workload the heartbeat was trying to update no longer exists or has already finished. This could happen due to several reasons: 1. The workload completed or failed very quickly, and the heartbeat mechanism wasn't able to keep up. 2. There might be a race condition where the workload finished just as a heartbeat was being sent. 3. The workload might have been manually cancelled or terminated by another process. This error is typically transient and doesn't necessarily indicate a problem with your data synchronization. However, if you're seeing this error frequently, it might point to underlying issues with your Airbyte setup or configuration. If you're experiencing persistent problems related to this error, you may want to: 1. Check your Airbyte logs for any related errors or warnings. 2. Ensure that your Airbyte components (especially the server and workers) are properly configured and have sufficient resources. 3. Consider upgrading to the latest version of Airbyte, as heartbeat handling may have been improved in newer releases. For more detailed troubleshooting, you might need to provide additional context about when and how often you're seeing this error, and what specific operations were being performed when it occurred.
Open in Slack
PreviousNext
Powered by