https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# ask-ai
s
This message was deleted.
k
The error 
An unknown error occurred. (HTTP 504)
typically indicates a timeout issue. This can happen when fetching the schema of your data source takes longer than the server is prepared to wait. This could be due to slow internet connections or data sources with a large amount of streams. In one of the discussions, a user mentioned that they encountered this error when trying to set up a new connection between Shopify API (source) and Google Sheets (destination). They suspected that they were hitting a timeout limit and considered increasing the timeout config on the nginx template for the Airbyte proxy. However, they were unsure how to do this exactly. Unfortunately, the knowledge sources provided do not include specific instructions on how to increase the timeout config on the nginx template for the Airbyte proxy. You may want to look into the official Airbyte documentation or reach out to the Airbyte community for more specific guidance. If you're self-hosting Airbyte, it's also worth checking that your version of Airbyte and your connectors are up to date, as this can sometimes resolve issues. Sources: - [Discussion: An unknown error occurred (HTTP 504) while setting up a new connection between Shopify API (source) and Google Sheets (destination)](https://github.com/airbytehq/airbyte/discussions/40738) - [Github Issue (closed): [source-facebook-marketing] Failed to discover schema](https://github.com/airbytehq/airbyte/issues/39140)
Sorry @Suthira (Praer) Owlarn, I seem to be having a bit of trouble. Don't worry though, I've pinged my support folks and they will try to fix me up! 🧑‍💻 Perhaps try again in 5mins?
Sorry @Suthira (Praer) Owlarn, I seem to be having a bit of trouble. Don't worry though, I've pinged my support folks and they will try to fix me up! 🧑‍💻 Perhaps try again in 5mins?
s
@kapa.ai I increased the timeout limit, but it still failed with 
Failed to fetch
. Logs include eg. 
Workflow task processing failure
PotentialDeadlockException: Potential deadlock detected
Caused by: io.temporal.failure.TimeoutFailure: message='activity ScheduleToClose timeout', timeoutType=TIMEOUT_TYPE_SCHEDULE_TO_CLOSE
k
Based on the knowledge sources provided, it seems like you're encountering a timeout issue with Airbyte, specifically with the 
ScheduleToClose
timeout type. This issue has been reported in the past with the Source S3 connector, where the 
check_connection
function failed due to a large number of files in the S3 bucket ([source](https://github.com/airbytehq/airbyte/issues/25955)). The error message 
io.temporal.failure.TimeoutFailure: message='activity ScheduleToClose timeout', timeoutType=TIMEOUT_TYPE_SCHEDULE_TO_CLOSE
indicates that the activity did not complete within the timeout period specified. This could be due to various reasons such as network latency, the size of the data, or the performance of the system where Airbyte is running. Unfortunately, the provided knowledge sources do not contain specific solutions to this issue. However, you might want to consider the following general suggestions: 1. Increase the system resources: If your system is under high load, it might not be able to process the tasks within the specified timeout. Increasing the system resources (CPU, memory) might help. 2. Optimize your data source: If the data source contains a large number of files or large files, consider optimizing it by reducing the number of files or their size. 3. Check network latency: If your system and the data source are located in different geographical locations, network latency might cause delays. Consider using a system closer to the data source. 4. Update to the latest version of Airbyte: If you're not already using the latest version of Airbyte, consider updating it. Some issues might have been fixed in the newer versions. If none of the above suggestions work, I recommend reaching out to the Airbyte community on [Github](https://github.com/airbytehq/airbyte) or [Discourse](https://discuss.airbyte.io/) for further assistance.
s
@kapa.ai I noticed after the worker pods logged 
INFO i.a.w.p.ExitCodeWatcher(persistExitCode):117 - Received exit code 0 for pod ce-stripe-discover-2e56962b-2f33-45c9-9a49-52ea50ca1d80-0-uewan
there was no following 
INFO i.a.c.t.HeartbeatUtils(withBackgroundHeartbeat):64 - Stopping temporal heartbeating...
and 
INFO i.a.c.t.HeartbeatUtils(withBackgroundHeartbeat):73 - Temporal heartbeating stopped.
.. is this important?
k
A message from kapa.ai
The log messages 
INFO i.a.c.t.HeartbeatUtils(withBackgroundHeartbeat):64 - Stopping temporal heartbeating...
and 
INFO i.a.c.t.HeartbeatUtils(withBackgroundHeartbeat):73 - Temporal heartbeating stopped.
are related to the Temporal heartbeating process in Airbyte. Temporal is a microservice orchestration platform used by Airbyte for task scheduling and execution. The heartbeating process is a mechanism to signal that a task is still running. When a task finishes, the heartbeating process is stopped, which is indicated by these log messages. If these messages are not present after a worker pod finishes its task (indicated by the 
Received exit code 0
log message), it could mean that the heartbeating process was not stopped properly. This could potentially lead to issues, as Temporal might still consider the task as running. However, the absence of these log messages does not necessarily indicate a problem. It could also be that the logging level was set in a way that these informational messages are not logged. If you are not experiencing any issues with your data synchronization tasks, this is likely not a cause for concern. However, if you are noticing issues with tasks not completing as expected, this could be a part of the problem and it might be worth investigating further. For more detailed information or assistance, consider reaching out to the Airbyte community on [Github](https://github.com/airbytehq/airbyte) or [Discourse](https://discuss.airbyte.io/).
s
@kapa.ai I also see eg. 
Workflow task processing failure. [] If seen continuously the workflow might be stuck.
The syncs succeed at the moment, but they do fail in various attempts from this error. So I'd still like to understand them. Also this is a blocker for adding new connections with stripe sources.
k
A message from kapa.ai
The error message 
Workflow task processing failure. If seen continuously the workflow might be stuck.
is related to the Temporal workflow engine that Airbyte uses for task scheduling and execution. This error message indicates that there was a problem processing a task within the workflow. This could be due to various reasons such as a bug in the workflow code, a problem with the Temporal service, or a resource issue on the system running Airbyte. If this error is seen continuously, it could mean that the workflow is stuck in a state where it cannot progress, which could potentially lead to failed syncs. In the context of adding new connections with Stripe sources, if you're seeing this error, it could mean that the workflow for setting up the new connection is encountering an issue. This could be due to a problem with the Stripe source connector, a configuration issue, or a problem with the Temporal service. Unfortunately, the provided knowledge sources do not contain specific solutions to this issue. However, you might want to consider the following general suggestions: 1. Check the complete logs: The complete logs might contain more detailed error messages that could help identify the issue. You can access the logs from the Airbyte UI or directly from the system running Airbyte. 2. Check the configuration of the Stripe source: Make sure that the configuration of the Stripe source is correct. This includes the API key, the start date, and any other configuration parameters. 3. Update to the latest version of Airbyte and the Stripe source connector: If you're not already using the latest version of Airbyte and the Stripe source connector, consider updating them. Some issues might have been fixed in the newer versions. If none of the above suggestions work, I recommend reaching out to the Airbyte community on [Github](https://github.com/airbytehq/airbyte) or [Discourse](https://discuss.airbyte.io/) for further assistance.
s
@kapa.ai I get the timeout error when creating the connector via the UI, but not when I use terraform (which connects to the server directly). Is this a clue as to what's wrong?
k
A message from kapa.ai
The fact that you're not experiencing the timeout error when creating the connector via Terraform, but only through the UI, could indeed provide a clue to the issue. Terraform interacts directly with the Airbyte server's API, while the UI might involve additional layers of communication which could potentially introduce latency or other issues. Here are a few possibilities: 1. Network Latency or Issues: If the system running your browser has a slow or unstable network connection to the Airbyte server, it might cause requests from the UI to timeout. You can try accessing the UI from a different network or system to see if the issue persists. 2. Browser or System Issues: The issue might be related to the browser or the system you're using to access the UI. Try using a different browser or system to see if the issue persists. 3. Server Load or Performance Issues: If the Airbyte server is under heavy load or experiencing performance issues, it might respond more slowly to requests from the UI compared to direct API calls from Terraform. Check the server's resource usage (CPU, memory, disk I/O, etc.) to see if it's under heavy load. 4. UI Bug or Issue: There might be a bug or issue in the Airbyte UI that's causing the timeout. If this is the case, it would be best to report this to the Airbyte team via [Github](https://github.com/airbytehq/airbyte) or [Discourse](https://discuss.airbyte.io/). Remember, these are just possibilities and the actual cause might be different. It would be best to investigate further to identify the root cause of the issue.
Open in Slack
PreviousNext
Powered by