https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> trying to downgrade mysql source to...
# replication-ask-ai
f
@kapa.ai trying to downgrade mysql source to version 3.46.2 source-mysql-spec failed with
Copy code
Defaulted container "connector-sidecar" out of: connector-sidecar, main, init (init)
Unsetting empty environment variable 'DATA_PLANE_SERVICE_ACCOUNT_CREDENTIALS_PATH'
Unsetting empty environment variable 'AIRBYTE_URL'
Unsetting empty environment variable 'KEYCLOAK_INTERNAL_REALM_ISSUER'
Unsetting empty environment variable 'DATA_PLANE_SERVICE_ACCOUNT_EMAIL'
Unsetting empty environment variable 'KEYCLOAK_CLIENT_ID'
Unsetting empty environment variable 'AIRBYTE_ROLE'
Unsetting empty environment variable 'CONTROL_PLANE_AUTH_ENDPOINT'
2025-12-15 10:46:46,950 [main]	INFO	i.a.c.ApplicationKt(main):14 - Sidecar start

    ___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
 : airbyte-connector-sidecar :

2025-12-15 10:46:48,334 [main]	INFO	i.m.c.e.DefaultEnvironment(<init>):170 - Established active environments: [worker-v2, k8s, control-plane, edition-community, local-secrets]
2025-12-15 10:46:49,003 [main]	INFO	i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'INFO' for logger: 'io.netty'
2025-12-15 10:46:49,004 [main]	INFO	i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'ERROR' for logger: 'com.zaxxer.hikari'
2025-12-15 10:46:49,004 [main]	INFO	i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'INFO' for logger: 'io.grpc'
2025-12-15 10:46:49,005 [main]	INFO	i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'INFO' for logger: 'io.temporal'
2025-12-15 10:46:49,006 [main]	INFO	i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'ERROR' for logger: 'com.zaxxer.hikari.pool'
2025-12-15 10:46:49,006 [main]	INFO	i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'INFO' for logger: 'io.fabric8.kubernetes.client'
2025-12-15 10:46:49,037 [main]	INFO	i.m.r.Micronaut(start):183 - No embedded container found. Running as CLI application
2025-12-15 10:46:52,573 [pool-3-thread-1]	INFO	i.a.c.i.LineGobbler$Companion(gobble$lambda$2):108 -
2025-12-15 10:46:52,609 [pool-3-thread-1]	INFO	i.a.c.i.LineGobbler$Companion(gobble$lambda$2):108 - ----- START SPEC -----
2025-12-15 10:46:52,610 [pool-3-thread-1]	INFO	i.a.c.i.LineGobbler$Companion(gobble$lambda$2):108 -
2025-12-15 10:55:52,599 [main]	WARN	i.a.c.ConnectorWatcher(waitForConnectorOutput):108 - Failed to find output files from connector within timeout of 9 minute(s). Is the connector still running?
2025-12-15 10:55:52,622 [main]	INFO	i.a.c.ConnectorWatcher(failWorkload):324 - Failing workload e668e35b-1934-4c24-913c-133c3f3312b6_spec.
2025-12-15 10:55:52,687 [main]	INFO	i.a.c.ConnectorWatcher(exitFileNotFound):242 - Deliberately exiting process with code 2.
k
The logs show the connector sidecar starting correctly and then timing out after 9 minutes waiting for the 
SPEC
output file: > 
Failed to find output files from connector within timeout of 9 minute(s). Is the connector still running?
> 
Failing workload ..._spec.
This is the same behavior described in: - The MySQL check job issue, where the job hangs then fails with the same 9‑minute timeout for connector output [[MySQL 3.11.0 issue](https://github.com/airbytehq/airbyte/issues/52075)]. - The generic sidecar timeout issue, where 
SIDECAR_FILE_TIMEOUT_MINUTES
is fixed at 9 minutes and causes long operations (e.g. BigQuery discover) to fail [[Sidecar timeout config](https://github.com/airbytehq/airbyte/issues/48855)]. In both cases, the root symptom is: the connector container never produces the expected output file (here, 
SPEC
) within the sidecar’s hardcoded 9‑minute timeout, so the sidecar fails the workload. From the available sources: - The timeout is controlled by 
SIDECAR_FILE_TIMEOUT_MINUTES
(and 
SIDECAR_FILE_TIMEOUT_WITHIN_SYNC_MINUTES
), which are not currently exposed via Helm or standard configuration; users who need longer timeouts are building a custom 
connector-sidecar
image with higher values. [[Sidecar timeout config](https://github.com/airbytehq/airbyte/issues/48855)] - There is no specific information in the sources about MySQL 3.46.2 in particular, only about 3.11.0 and older MySQL versions, and no explicit fix for this behavior. [[MySQL 3.11.0 issue](https://github.com/airbytehq/airbyte/issues/52075)] Given that, the knowledge sources support only these actionable options: 1. Try another MySQL connector version known to work in your environment. In the referenced issue, downgrading from 3.11.0 to 3.5.1 resolved the check job hang. [[MySQL 3.11.0 issue](https://github.com/airbytehq/airbyte/issues/52075)] 2. Increase the sidecar timeout by using a custom sidecar image (if you control the deployment images), e.g. building from the official sidecar image with higher 
SIDECAR_FILE_TIMEOUT_MINUTES
/ 
SIDECAR_FILE_TIMEOUT_WITHIN_SYNC_MINUTES
, as described in the workaround: [[Sidecar timeout config](https://github.com/airbytehq/airbyte/issues/48855)]
Copy code
dockerfile    FROM airbyte/connector-sidecar:<your-airbyte-version>     ENV SIDECAR_FILE_TIMEOUT_MINUTES=30    ENV SIDECAR_FILE_TIMEOUT_WITHIN_SYNC_MINUTES=60
3. Inspect the connector container logs directly. The provided sources only show sidecar logs; they don’t include guidance on how to debug the connector container itself in this specific case. I’m sorry, but there’s no information in the sources about specific MySQL 3.46.2 failures beyond the sidecar timeout symptom. Because the knowledge sources don’t contain anything more specific about version 3.46.2 or a targeted fix for this exact downgrade scenario, I can’t confidently give a more precise root cause than: the MySQL connector did not emit its 
SPEC
output within the sidecar’s 9‑minute timeout, causing the job to fail.
f
@kapa.ai how to nspect the connector container logs directly.?
k
A message from kapa.ai
On Kubernetes, you can inspect connector (worker) pod logs and files using 
kubectl
as described in the docs. 1. Find the pod name (usually in the 
jobs
namespace for workers):
Copy code
bash kubectl get pods -n jobs
Look for a pod name matching your job, e.g. 
source-mysql-worker-<job-id>-0-xxxxx
. 2. Stream the logs from the connector container (the main container is usually named 
main
):
Copy code
bash kubectl logs -n jobs <pod-name> -c main
3. Open a shell inside the connector container and browse files (e.g. 
/config
):
Copy code
bash kubectl exec -it <pod-name> -n jobs -c main bash # then inside the pod: ls /config
4. Copy files (like configs or catalogs) from the pod to your local machine if needed:
Copy code
bash kubectl cp jobs/<pod-name>:/config/destination_catalog.json ./catalog.json
These patterns are documented for browsing worker pods and their 
/config
contents on Kubernetes.[\[Browsing logs\]](https://docs.airbyte.com/platform/operator-guides/browsing-output-logs#browsing-on-kubernetes)
f
@kapa.ai
Copy code
airbyte@airbyte-dump:~$ sudo docker exec -it airbyte-abctl-control-plane kubectl -n airbyte-abctl get pods
NAME                                                             READY   STATUS      RESTARTS      AGE
airbyte-abctl-bootloader                                         0/1     Completed   0             73d
airbyte-abctl-connector-builder-server-6d97549f58-h2zld          1/1     Running     1 (41d ago)   73d
airbyte-abctl-cron-746c86669-2qxk4                               1/1     Running     1 (41d ago)   73d
airbyte-abctl-server-dc4db98fb-6t97j                             1/1     Running     2 (41d ago)   73d
airbyte-abctl-temporal-68f545669c-6kpph                          1/1     Running     1 (41d ago)   73d
airbyte-abctl-worker-79c8b4bbb6-r5lbd                            1/1     Running     1 (41d ago)   73d
airbyte-abctl-workload-api-server-7bc4b4f685-77f24               1/1     Running     1 (41d ago)   73d
airbyte-abctl-workload-launcher-c95f87dd5-rmqpn                  1/1     Running     2 (41d ago)   73d
airbyte-db-0                                                     1/1     Running     1 (41d ago)   73d
airbyte-minio-0                                                  1/1     Running     1 (41d ago)   73d
source-mysql-spec-b025516c-3650-4b16-8497-f31ad5ec860b-0-jrunp   0/2     Error       0             19m
source-mysql-spec-bf106cb6-1f1e-4e6d-9ec6-7faa0af789eb-0-dozxl   0/2     Error       0             32m
source-mysql-spec-c216d0ea-8c22-4468-8b60-5150bf7f7ff1-0-wsmdo   0/2     Error       0             26m
source-mysql-spec-d8148311-5794-4a89-9ca1-1779d7f924a4-0-hnthu   0/2     Error       0             29m
source-mysql-spec-e668e35b-1934-4c24-913c-133c3f3312b6-0-ujkzl   0/2     Error       0             33m
airbyte@airbyte-dump:~$ sudo docker exec -it airbyte-abctl-control-plane kubectl -n airbyte-abctl get pods -n jobs
No resources found in jobs namespace.
airbyte@airbyte-dump:~$ sudo docker exec -it airbyte-abctl-control-plane kubectl -n jobs get pods
No resources found in jobs namespace.
k
A message from kapa.ai
In an 
abctl
-based local deployment, your Airbyte control-plane and all namespaces live inside the 
airbyte-abctl-control-plane
container. You’re already using the right pattern (wrapping 
kubectl
with 
docker exec
), but there are two key points: 1. The connector job pods are in the 
airbyte-abctl
namespace, not 
jobs
. This matches what the docs show for viewing pods with 
abctl
(no separate 
jobs
namespace is used there). [[Developing locally 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] 2. To see logs of a failing spec pod, you must: - Use the correct namespace (
airbyte-abctl
) - Target the right container in the pod (
main
for the connector, 
connector-sidecar
for the sidecar). Given your output, here’s exactly what to run from the host:
Copy code
bash # 1. List pods in the airbyte-abctl namespace (you already did this): sudo docker exec -it airbyte-abctl-control-plane \   kubectl -n airbyte-abctl get pods  # 2. Get logs from the connector container ("main") for one of the failing spec pods: sudo docker exec -it airbyte-abctl-control-plane \   kubectl -n airbyte-abctl logs source-mysql-spec-e668e35b-1934-4c24-913c-133c3f3312b6-0-ujkzl -c main  # 3. (Optional) Get logs from the sidecar in the same pod: sudo docker exec -it airbyte-abctl-control-plane \   kubectl -n airbyte-abctl logs source-mysql-spec-e668e35b-1934-4c24-913c-133c3f3312b6-0-ujkzl -c connector-sidecar
To open a shell in the connector container (to inspect 
/config
, etc.), adapt the standard Kubernetes pattern [[Browsing logs](https://docs.airbyte.com/platform/operator-guides/browsing-output-logs#browsing-on-kubernetes)]:
Copy code
bash sudo docker exec -it airbyte-abctl-control-plane \   kubectl -n airbyte-abctl exec -it source-mysql-spec-e668e35b-1934-4c24-913c-133c3f3312b6-0-ujkzl -c main -- bash
From there you can 
ls /config
, inspect files, etc. If 
kubectl
still reports “No resources found in jobs namespace”, that’s expected: 
abctl
deployments use the 
airbyte-abctl
namespace, not 
jobs
. [[Developing locally 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)]
f
@kapa.ai
Copy code
airbyte@airbyte-dump:~$ sudo docker exec -it airbyte-abctl-control-plane \
  kubectl -n airbyte-abctl exec -it source-mysql-spec-e668e35b-1934-4c24-913c-133c3f3312b6-0-ujkzl -c main -- bash
error: Internal error occurred: unable to upgrade connection: container not found ("main")
k
A message from kapa.ai
In your pod, the connector containers are not named 
main
, which is why 
kubectl
reports 
container not found ("main")
. From the logs you originally pasted, we know there is a 
connector-sidecar
container, and the issue description you shared shows that pods can have both 
main
and 
connector-sidecar
, but in your case only the sidecar appears to be present/visible. [[Custom source issue](https://github.com/airbytehq/airbyte/issues/68608)] The Airbyte docs only give the generic pattern for browsing a pod, not the exact container names; they use 
-c main
as an example, but note that you must supply the actual container name in your pod. [[Browsing logs](https://docs.airbyte.com/platform/operator-guides/browsing-output-logs#browsing-on-kubernetes)] Given the information available, here’s what you can do inside the 
airbyte-abctl-control-plane
container: 1. List containers for that pod to see the exact names:
Copy code
bash sudo docker exec -it airbyte-abctl-control-plane \   kubectl -n airbyte-abctl get pod source-mysql-spec-e668e35b-1934-4c24-913c-133c3f3312b6-0-ujkzl -o jsonpath='{.spec.containers[*].name}'
2. Once you see the container names, use one of them with 
kubectl logs
/ 
kubectl exec
. For example, if the only container is `connector-sidecar`:
Copy code
bash sudo docker exec -it airbyte-abctl-control-plane \   kubectl -n airbyte-abctl logs source-mysql-spec-e668e35b-1934-4c24-913c-133c3f3312b6-0-ujkzl -c connector-sidecar
and, if the connector container has some other name (e.g. 
source-mysql
):
Copy code
bash sudo docker exec -it airbyte-abctl-control-plane \   kubectl -n airbyte-abctl exec -it source-mysql-spec-e668e35b-1934-4c24-913c-133c3f3312b6-0-ujkzl -c <that-container-name> -- bash
The docs don’t list container names for spec/check pods explicitly, nor do they cover the failure mode where the 
main
container is missing and only the sidecar starts. I’m therefore limited to this general `kubectl`‑based inspection approach and can’t provide a more specific container name from the knowledge sources.
f
@kapa.ai
Copy code
airbyte@airbyte-dump:~$ sudo docker exec -it airbyte-abctl-control-plane \
  kubectl -n airbyte-abctl get pod source-mysql-spec-e668e35b-1934-4c24-913c-133c3f3312b6-0-ujkzl -o jsonpath='{.spec.containers[*].name}'
airbyte@airbyte-dump:~$ sudo docker exec -it airbyte-abctl-control-plane \-abctl-control-plane \
  kubectl -n airbyte-abctl logs source-mysql-spec-e668e35b-1934-4c24-913c-133c3f3312b6-0-ujkzl -c connector-sidecar
Unsetting empty environment variable 'DATA_PLANE_SERVICE_ACCOUNT_CREDENTIALS_PATH'
Unsetting empty environment variable 'AIRBYTE_URL'
Unsetting empty environment variable 'KEYCLOAK_INTERNAL_REALM_ISSUER'
Unsetting empty environment variable 'DATA_PLANE_SERVICE_ACCOUNT_EMAIL'
Unsetting empty environment variable 'KEYCLOAK_CLIENT_ID'
Unsetting empty environment variable 'AIRBYTE_ROLE'
Unsetting empty environment variable 'CONTROL_PLANE_AUTH_ENDPOINT'
2025-12-15 10:46:46,950 [main]	INFO	i.a.c.ApplicationKt(main):14 - Sidecar start

    ___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
 : airbyte-connector-sidecar :

2025-12-15 10:46:48,334 [main]	INFO	i.m.c.e.DefaultEnvironment(<init>):170 - Established active environments: [worker-v2, k8s, control-plane, edition-community, local-secrets]
2025-12-15 10:46:49,003 [main]	INFO	i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'INFO' for logger: 'io.netty'
2025-12-15 10:46:49,004 [main]	INFO	i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'ERROR' for logger: 'com.zaxxer.hikari'
2025-12-15 10:46:49,004 [main]	INFO	i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'INFO' for logger: 'io.grpc'
2025-12-15 10:46:49,005 [main]	INFO	i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'INFO' for logger: 'io.temporal'
2025-12-15 10:46:49,006 [main]	INFO	i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'ERROR' for logger: 'com.zaxxer.hikari.pool'
2025-12-15 10:46:49,006 [main]	INFO	i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'INFO' for logger: 'io.fabric8.kubernetes.client'
2025-12-15 10:46:49,037 [main]	INFO	i.m.r.Micronaut(start):183 - No embedded container found. Running as CLI application
2025-12-15 10:46:52,573 [pool-3-thread-1]	INFO	i.a.c.i.LineGobbler$Companion(gobble$lambda$2):108 -
2025-12-15 10:46:52,609 [pool-3-thread-1]	INFO	i.a.c.i.LineGobbler$Companion(gobble$lambda$2):108 - ----- START SPEC -----
2025-12-15 10:46:52,610 [pool-3-thread-1]	INFO	i.a.c.i.LineGobbler$Companion(gobble$lambda$2):108 -
2025-12-15 10:55:52,599 [main]	WARN	i.a.c.ConnectorWatcher(waitForConnectorOutput):108 - Failed to find output files from connector within timeout of 9 minute(s). Is the connector still running?
2025-12-15 10:55:52,622 [main]	INFO	i.a.c.ConnectorWatcher(failWorkload):324 - Failing workload e668e35b-1934-4c24-913c-133c3f3312b6_spec.
2025-12-15 10:55:52,687 [main]	INFO	i.a.c.ConnectorWatcher(exitFileNotFound):242 - Deliberately exiting process with code 2.
k
A message from kapa.ai
The sidecar logs confirm the same pattern as in the documented issues: the sidecar starts, launches the 
SPEC
workload, and after 9 minutes fails because it never sees the connector’s output file:
Copy code
text ----- START SPEC ----- ... Failed to find output files from connector within timeout of 9 minute(s). Is the connector still running? ... Failing workload e668e35b-1934-4c24-913c-133c3f3312b6_spec. ... Deliberately exiting process with code 2.
This is exactly the behavior described in the GitHub issues where: - The sidecar waits for connector output and then times out after 9 minutes with the same warning and exit code 2.[[MySQL 3.11.0 issue](https://github.com/airbytehq/airbyte/issues/52075); [Zendesk timeout](https://github.com/airbytehq/airbyte/issues/54684)] - The timeout is controlled by 
SIDECAR_FILE_TIMEOUT_MINUTES
(and 
SIDECAR_FILE_TIMEOUT_WITHIN_SYNC_MINUTES
), which are not configurable via Helm; the documented workaround is to build a custom 
connector-sidecar
image with higher values.[[Sidecar timeout config](https://github.com/airbytehq/airbyte/issues/48855)] From the knowledge sources, you have two levers: 1. Connector behavior / version In at least one case, downgrading the MySQL connector from a problematic version (3.11.0) to an older one made the check job complete successfully.[[MySQL 3.11.0 issue](https://github.com/airbytehq/airbyte/issues/52075)] The sources don’t mention 3.46.2 specifically, so I cannot say whether that version is expected to work or not. 2. Sidecar timeout You can extend the 9‑minute timeout by building a custom sidecar image, as shown in the documented workaround:[[Sidecar timeout config](https://github.com/airbytehq/airbyte/issues/48855)]
Copy code
dockerfile    FROM airbyte/connector-sidecar:1.2.0  # replace with your sidecar tag     ENV SIDECAR_FILE_TIMEOUT_MINUTES=30    ENV SIDECAR_FILE_TIMEOUT_WITHIN_SYNC_MINUTES=60
Then configure your deployment to use this custom sidecar image. The docs do not specify how to wire this custom image into `abctl`‑managed deployments, so I can’t give exact commands for that part. Because the sources: - Do not show logs from the connector container itself for your pod, and - Do not describe how to adjust the sidecar image or env vars specifically in an 
abctl
(kind) deployment, I can’t reliably provide a more detailed fix than: - The spec job fails because the MySQL connector never writes its 
SPEC
output within the sidecar’s 9‑minute timeout. - Your next steps, based on the available docs, are to either: - Try a different MySQL connector version; or - Increase the sidecar timeout via a custom sidecar image as per the documented workaround.
2 Views
Open in Slack
PreviousNext
Powered by