Hey all, I deployed helm with airbyte and I notic...
# replication-ask-ai
m
Hey all, I deployed helm with airbyte and I noticed a new temporal and temporal visibility database. It's now making a lot of idle connections. what is the temporal database and how do i fix it @kapa.ai
k
Hello, The Temporal database is part of Airbyte's technical stack. It is used for orchestrating hundreds to thousands of jobs simultaneously by scaling state management and proper queue management. Temporal offers primitives that allow serializing the jobs' current runtime memory into a DB. Since a job's entire state is stored, it's trivial to recover from failures, and it's easy to determine if a job was assigned correctly [source](https://docs.airbyte.com/understanding-airbyte/tech-stack). The Temporal database maintains multiple idle connections. By default, this value is
20
and you may want to lower or increase this number. One issue that has been noticed is that Temporal creates multiple pools and the number specified in the
SQL_MAX_IDLE_CONNS
environment variable of the
docker.compose.yaml
file might end up allowing 4-5 times more connections than expected. If you want to increase the amount of allowed idle connections, you will also need to increase
SQL_MAX_CONNS
as well because
SQL_MAX_IDLE_CONNS
is capped by
SQL_MAX_CONNS
[source](https://docs.airbyte.com/operator-guides/scaling-airbyte). Please note that you should be careful when making these changes as they can affect the performance and stability of your Airbyte deployment. If you're not sure about the implications, it's recommended to seek advice from the Airbyte community or your database administrator.
m
how do i adjust this in my current helm deployment. it's over spiking @kapa.ai
k
A message from kapa.ai
To adjust the number of idle connections in your current Helm deployment, you need to modify the environment variables
SQL_MAX_IDLE_CONNS
and
SQL_MAX_CONNS
in your Helm configuration. You can do this by creating a
values.yaml
file in your local folder and populating it with the default configuration override values. An example of a
values.yaml
file can be found in the [charts/airbyte](https://github.com/airbytehq/airbyte-platform/blob/main/charts/airbyte/values.yaml) folder of the Airbyte repository. In the
values.yaml
file, you can add the
SQL_MAX_IDLE_CONNS
and
SQL_MAX_CONNS
environment variables under the
extraEnv
property. Here's an example:
Copy code
yaml extraEnv:   - name: SQL_MAX_IDLE_CONNS     value: "your_desired_value"   - name: SQL_MAX_CONNS     value: "your_desired_value"
Replace
"your_desired_value"
with the number of connections you want to allow. After specifying your own configuration, run the following command to apply the changes:
Copy code
bash helm install --values path/to/values.yaml %release_name% airbyte/airbyte
Replace
path/to/values.yaml
with the path to your
values.yaml
file and
%release_name%
with the name of your Helm release [source](https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm). Please note that you should be careful when making these changes as they can affect the performance and stability of your Airbyte deployment. If you're not sure about the implications, it's recommended to seek advice from the Airbyte community or your database administrator.
m
what is the standard max connection and max idle connectiions @kapa.ai
k
A message from kapa.ai
The standard maximum number of connections (
SQL_MAX_CONNS
) and maximum idle connections (
SQL_MAX_IDLE_CONNS
) for the Temporal database in Airbyte are not explicitly mentioned in the provided extracts. However, it is mentioned that Temporal maintains multiple idle connections and by default, this value is
20
. Please note that these values can be adjusted based on your specific use case and infrastructure capacity. If you're experiencing issues with too many connections, you may want to consider reducing these values. Conversely, if you have the capacity and are looking to increase throughput, you may want to increase these values. Remember to adjust both
SQL_MAX_CONNS
and
SQL_MAX_IDLE_CONNS
as
SQL_MAX_IDLE_CONNS
is capped by
SQL_MAX_CONNS
[source](https://docs.airbyte.com/operator-guides/scaling-airbyte). For the exact default values and advice on what values would be appropriate for your use case, it would be best to consult the Airbyte documentation or reach out to the Airbyte community.
m
is it helm install or upgrade @kapa.ai
k
A message from kapa.ai
If you are installing Airbyte for the first time using Helm, you would use the
helm install
command. However, if you have already installed Airbyte and you want to update your configuration or upgrade to a new version, you would use the
helm upgrade
command. Here's how you can use these commands: To install Airbyte with your custom configuration:
Copy code
bash helm install --values path/to/values.yaml %release_name% airbyte/airbyte
To upgrade your existing Airbyte deployment with your custom configuration:
Copy code
bash helm upgrade -f path/to/values.yaml %release_name% airbyte/airbyte
In both commands, replace
path/to/values.yaml
with the path to your
values.yaml
file and
%release_name%
with the name of your Helm release [source](https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm).
s
I faced the same issue. A lot of jobs activity hanging @kapa.ai
k
A message from kapa.ai
If you're experiencing issues with a lot of job activities hanging in Airbyte, it could be due to a variety of reasons such as configuration issues, resource limitations, or network connectivity problems. Here are a few steps you can take to troubleshoot: 1. Check the logs: The logs provide detailed information about what's happening behind the scenes, and they can help pinpoint the root cause of the problem [source](https://docs.airbyte.com/troubleshooting). 2. Check your configuration: In some cases, users have resolved similar issues by adjusting configuration settings. For example, one user resolved hanging job activities by increasing the Temporal activity max timeout (
ACTIVITY_MAX_TIMEOUT_SECOND
) in the Kubernetes configuration [source](https://airbytehq.slack.com/archives/C021JANJ6TY/p1671223036014729). 3. Check your resources: Ensure that your Kubernetes cluster has sufficient resources (CPU, memory) to handle the workload. If resources are constrained, job activities may hang. 4. Reach out to the community: If you're still unable to resolve the issue, consider reaching out to the Airbyte community on their [Slack](https://slack.airbyte.io) or [Forum](https://discuss.airbyte.io). Other users may have encountered similar issues and can provide guidance [source](https://docs.airbyte.com/troubleshooting). Remember, any changes to your configuration or infrastructure should be done carefully, considering the potential impact on performance and stability. If you're unsure, it's always a good idea to seek advice from the community or your system administrator.
v
I noticed a similar symptom as you when checking on Temporal. There are some workflows with pending activities, which is causing manual sync requests from Airbyte to not be executed in Temporal. Do you have any ideas about pending activities on the workflow manager in Temporal? @kapa.ai
k
A message from kapa.ai
@Vũ Le That's a great question, but unfortunately I am only capable of responding to a maximum of five questions per thread. Don't hesitate to create a new thread for any further questions you have though. I'm here to help!