https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> `abctl local install` is not workin...
# replication-ask-ai
a
@kapa.ai 
abctl local install
is not working anymore on the same values.yaml, secrets.yaml and flags as before. I believe it might have some thing to do with the recent version upgrade. The same warning about not being able to get some kind of replica secret gets thrown ad infinitum. There is no reference to CONFIG_DATABASE_REPLICA_USER anywhere in the Airbyte docs Errors:
Copy code
$ abctl local install --insecure-cookies --no-browser --values values.yaml  --secret secrets.yaml 
  INFO    Using Kubernetes provider:
            Provider: kind
            Kubeconfig: /root/.airbyte/abctl/abctl.kubeconfig
            Context: kind-airbyte-abctl
 SUCCESS  Found Docker installation: version 29.4.0                                                                           
  INFO    No existing cluster found, cluster 'airbyte-abctl' will be created                                                  
 SUCCESS  Port 8000 appears to be available                                                                                   
 SUCCESS  Cluster 'airbyte-abctl' created                                                                                     
  INFO    Patching image airbyte/db:1.7.0-17                                                                                  
  INFO    Pulling image airbyte/async-profiler:2.1.0                                                                          
  INFO    Pulling image airbyte/bootloader:2.1.0                                                                              
  INFO    Pulling image airbyte/connector-sidecar:2.1.0                                                                       
  INFO    Pulling image airbyte/container-orchestrator:2.1.0                                                                  
  INFO    Pulling image airbyte/cron:2.1.0                                                                                    
  INFO    Pulling image airbyte/db:1.7.0-17                                                                                   
  INFO    Pulling image airbyte/manifest-server:7.10.0                                                                        
  INFO    Pulling image airbyte/server:2.1.0                                                                                  
  INFO    Pulling image airbyte/utils:2.1.0                                                                                   
  INFO    Pulling image airbyte/worker:2.1.0                                                                                  
  INFO    Pulling image airbyte/workload-api-server:2.1.0                                                                     
  INFO    Pulling image airbyte/workload-init-container:2.1.0                                                                 
  INFO    Pulling image airbyte/workload-launcher:2.1.0                                                                       
  INFO    Pulling image temporalio/auto-setup:1.27.2                                                                          
  INFO    Namespace 'airbyte-abctl' created                                                                                   
  INFO    Persistent volume 'airbyte-local-pv' created                                                                        
  INFO    Persistent volume claim 'airbyte-storage-pvc' created                                                               
  INFO    Persistent volume 'airbyte-volume-db' created                                                                       
  INFO    Persistent volume claim 'airbyte-volume-db-airbyte-db-0' created                                                    
 SUCCESS  Secret from '/root/abctl/secrets.yaml' created or updated                                                           
  INFO    Starting Helm Chart installation of 'airbyte/airbyte' (version: 2.1.0)                                              
 WARNING  Encountered an issue deploying Airbyte:                                                                             
            Pod: airbyte-abctl-bootloader.18a6a35136916be3
            Reason: Failed
            Message: Error: couldn't find key CONFIG_DATABASE_REPLICA_USER in Secret airbyte-abctl/airbyte-config-secrets
            Count: 6
 WARNING  Encountered an issue deploying Airbyte:                                                                             
            Pod: airbyte-abctl-bootloader.18a6a35136916be3
            Reason: Failed
            Message: Error: couldn't find key CONFIG_DATABASE_REPLICA_USER in Secret airbyte-abctl/airbyte-config-secrets
            Count: 7
 WARNING  Encountered an issue deploying Airbyte:                                                                             
            Pod: airbyte-abctl-bootloader.18a6a35136916be3
            Reason: Failed
            Message: Error: couldn't find key CONFIG_DATABASE_REPLICA_USER in Secret airbyte-abctl/airbyte-config-secrets
            Count: 8
...
secrets.yaml
Copy code
postgresql:
  enabled: false  # Using custom isolated postgres container instead
global:
  database:
    type: "external"
    secretName: "airbyte-config-secrets"
    host: "host.docker.internal"
    port: 5432
    name: "airbyte"
    userSecretKey: "database-user"
    passwordSecretKey: "database-password"
values.yaml
Copy code
apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  database-user: airbyte
  database-password: db_pass_changeme
k
Looking at your error, the bootloader pod is failing because it can't find the key 
CONFIG_DATABASE_REPLICA_USER
in your 
airbyte-config-secrets
secret. This is very similar to a known issue with Helm chart V2 where a required key is missing from the secret. Based on a related GitHub issue, Helm chart V2 includes certain keys that must exist in the 
airbyte-config-secrets
secret even if they aren't explicitly documented. The workaround is to add the missing key(s) to your secret. [Helm chart V2 secret issue] Fix: Add the missing key to your 
secrets.yaml
 Update your 
secrets.yaml
(which appears to actually be your Kubernetes Secret manifest ā€” note your files seem swapped in your question) to include the missing key:
Copy code
apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  database-user: airbyte
  database-password: db_pass_changeme
  CONFIG_DATABASE_REPLICA_USER: ""
Or patch the existing secret directly with kubectl:
Copy code
kubectl patch secret airbyte-config-secrets -n airbyte-abctl \
  --type merge \
  -p '{"stringData": {"CONFIG_DATABASE_REPLICA_USER": ""}}'
A few additional notes: 1. Your files appear to be swapped ā€” your 
secrets.yaml
contains the Kubernetes Secret manifest, and your 
values.yaml
contains the Helm values. Make sure you're passing them to the correct flags (
--secret
for the K8s secret manifest, 
--values
for the Helm values). 2. This appears to be a regression introduced in Airbyte v2.1.0 where the Helm chart V2 expects additional keys in the secret that weren't required before. [Helm chart V2 secret issue] 3. You may need to add a dummy key as well, since a similar issue was reported for 
AIRBYTE_DUMMY_SECRET
in Helm chart V2: [dummy secret issue]
Copy code
stringData:
  database-user: airbyte
  database-password: db_pass_changeme
  CONFIG_DATABASE_REPLICA_USER: ""
  AIRBYTE_DUMMY_SECRET: "dummy"
I should note that the knowledge sources don't explicitly document 
CONFIG_DATABASE_REPLICA_USER
as a required key, so this appears to be an undocumented requirement introduced in v2.1.0. You may want to open a GitHub issue with the Airbyte team to get this properly documented or fixed. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
How can I pin my version of the Helm Chart/ Airbyte in general, and what are the last known good versions of both in this respect?
@kapa.ai my above comment
k
Your query "my above comment" is a bit ambiguous ā€” could you clarify what you'd like to know or follow up on from our previous conversation? For example, are you asking about: ā€¢ Whether the workaround I suggested (adding 
CONFIG_DATABASE_REPLICA_USER
to your secret) worked or didn't work? ā€¢ A specific part of my previous answer you'd like me to clarify? ā€¢ Something else related to your 
abctl local install
issue? Please provide more details so I can give you the most accurate and helpful response! šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai How can I pin my version of the Helm Chart/ Airbyte in general, and what are the last known good versions of both in this respect?
k
Pinning Your Airbyte/Helm Chart Version Using abctl You can pin to a specific Helm chart version using the 
--chart-version
flag:
Copy code
abctl local install --chart-version <VERSION> --insecure-cookies --no-browser --values values.yaml --secret secrets.yaml
[abctl reference] Using Helm directly You can pin the version with the 
--version
flag:
Copy code
helm upgrade airbyte airbyte-v2/airbyte \
  --namespace airbyte-v2 \
  --values ./values.yaml \
  --version <CHART_VERSION>
[upgrading Airbyte] To see all available chart versions and their corresponding Airbyte app versions:
Copy code
helm search repo airbyte-v2 --versions
Last Known Good Versions Based on the GitHub issues in the knowledge sources, a user reported that downgrading to Airbyte 1.7.2 resolved their issues with `abctl 0.30.1`: > "I downgraded everything to 1.7.2 and now it is working: `abctl local install --chart-version=1.7.2`" [Helm chart broken issue] For those on Helm chart V1, 
airbyte/airbyte
chart version 1.8.5 (app version 1.8.5) was reported as working well, and chart version 1.9.1 (app version 2.0.1) was the fix provided by Airbyte for V1 users wanting to run Airbyte 2.0. [Helm chart V1 issue] For Helm chart V2, chart version 2.0.18 (app version 2.0.0) was confirmed working by users. [Multiple issues 2.0.0 chart] Given your specific issue with 
CONFIG_DATABASE_REPLICA_USER
appearing in Airbyte 2.1.0, pinning back to chart version 2.0.18 (Airbyte 2.0.0) may be a viable workaround until the issue is resolved:
Copy code
abctl local install --chart-version 2.0.18 --insecure-cookies --no-browser --values values.yaml --secret secrets.yaml
šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai How can I connect the pods in the abctl kind container to a sibling postgres container? I've tried exposing the postgres container on 0.0.0.054325432 on the host and providing host.docker.internal as the postgres host but that does not appear to be working.
k
The knowledge sources I have available don't contain specific information about networking between `abctl`/kind pods and a sibling Docker container using 
host.docker.internal
or other Docker networking approaches. What I can confirm from the documentation is that the recommended pattern for external database configuration with 
abctl
is to provide the host in your 
values.yaml
[External DB]:
Copy code
postgresql:
  enabled: false

global:
  database:
    type: external
    secretName: "airbyte-config-secrets"
    host: "your-db-hostname"
    port: "5432"
    name: "your_db_name"
    userSecretKey: "database-user"
    passwordSecretKey: "database-password"
However, the specific networking challenge you're describing ā€” connecting a kind cluster (created by 
abctl
) to a sibling Docker container ā€” is a Docker/Kubernetes networking problem rather than an Airbyte configuration problem. The knowledge sources don't cover this networking topology. General directions to investigate (not from Airbyte docs): ā€¢ 
host.docker.internal
works for Docker Desktop but may not resolve correctly inside kind nodes on Linux ā€¢ On Linux, you may need to use the Docker bridge gateway IP (e.g., 
172.17.0.1
) instead ā€¢ Alternatively, placing the Postgres container on the same Docker network as the kind container may help For Airbyte-specific guidance on this networking setup, I'd recommend asking in the Airbyte Community Slack where others may have solved this exact topology. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai what could be the possible cause of an abctl server going down after a few days of running a single connection? When the machine restarts, the connector is stalled and cannot be cancelled but will never finish. Eventually it errors, and I can restart the connection but eventually abctl will go down again and the webapp will be unreachable. The single connector from an API endpoint source to GCP pubsub destination. Approx 2-10 records synced every two minutes.
k
Based on the knowledge sources available, there are a few documented causes that could explain the symptoms you're describing: Possible Causes 1. Resource Exhaustion / Memory Leak A documented issue with the 
airbyte-worker
shows that over time, the worker can accumulate threads and memory until it becomes unresponsive. In one reported case, the worker had 29,821 running threads and was using 8+ GB of RAM, causing new jobs to fail to spawn threads and the entire deployment to freeze ā€” requiring a restart to recover. [Resource leak in worker] This matches your symptoms: the system runs fine initially, degrades over days, and requires intervention to recover. 2. Disk Space Exhaustion Another documented cause is the jobs/logs database growing too large over time, causing the 
airbyte-db
container to go down. One user reported: > "Our hypothesis is that the jobs database grew too much and there was no more space in the disk. As a consequence, the airbyte-db container went down." The fix was to configure automatic log deletion after X days. [Syncs failing suddenly] 3. Stalled Syncs After Restart The symptom of a sync being stuck in "Running" and unable to be cancelled after a restart is also well-documented. This is typically caused by the orchestration state (Temporal) losing track of the running job when the worker restarts mid-sync. [Syncs getting stuck] Recommended Checks 1. Monitor memory and disk usage on your host over time ā€” look for gradual growth leading up to the crash 2. Check disk space ā€” run 
df -h
and look at Docker volume usage 3. Review worker logs before the crash ā€” look for thread count or OOM indicators 4. Configure log retention in your 
values.yaml
to prevent unbounded log/job history growth The knowledge sources don't contain specific 
abctl
-focused diagnostics for this exact scenario, so for deeper investigation I'd recommend the Airbyte Community Slack or filing a GitHub issue with logs from just before the crash. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai Are there any values.yml, or otherwise, options that I can set to limit the overhead cost of source and destination containers spinning up. Maybe some optional checks or otherwise that I can turn off.
k
Based on the knowledge sources, here are the documented 
values.yaml
options to reduce the resource overhead of connector (source/destination) pods: Reducing Connector Pod Resource Usage 1. Set Lower Resource Limits/Requests for Connector Pods This is the primary lever ā€” reduce the CPU and memory requested/limited for each connector pod so they consume less overhead when spinning up:
Copy code
global:
  jobs:
    resources:
      limits:
        cpu: 250m        # lower than default
        memory: 500m     # lower than default
      requests:
        cpu: 75m
        memory: 150m
[Scaling Airbyte; Configuring connector resources] This is especially useful if your nodes have high requested CPU/memory but low actual usage ā€” a common cause of scheduling pressure. 2. Limit Concurrent Syncs Fewer concurrent syncs means fewer pods spinning up simultaneously, reducing peak overhead:
Copy code
worker:
  maxSyncWorkers: 5
  maxCheckWorkers: 5
[Scaling Airbyte] 3. Per-Connector or Per-Connection Overrides If only specific connectors are resource-heavy, you can target them via SQL rather than lowering limits globally:
Copy code
update actor_definition 
set resource_requirements = '{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "0.5", "cpu_request": "0.25", "memory_limit": "512Mi", "memory_request": "256Mi"}}]}' 
where id = '<connector-definition-id>';
[Configuring connector resources] The knowledge sources don't document options to disable specific checks (e.g., liveness/readiness probes on connector pods) or skip optional startup steps for source/destination containers beyond what's listed above. For your use case (2ā€“10 records every 2 minutes), lowering resource requests significantly should help reduce scheduling overhead without impacting sync performance. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
4 Views
Open in Slack
PreviousNext
Powered by