https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I tried to run abctl setup of airby...
# replication-ask-ai
p
@kapa.ai I tried to run abctl setup of airbyte running on EC2 locally. But there is infinite loading
@kapa.ai It was working well . but suddenly i am getting this error when i try to access. i have abctl setup running on ec2
k
The screenshot shows a 503 Service Temporarily Unavailable from nginx at 
localhost:8080
. ![](Screenshot 2026-07-16 at 12.14.26.png) This is the exact known bug with 
abctl v0.30.1
+ Airbyte 1.8.x, where the ingress incorrectly points to a non-existent 
airbyte-abctl-airbyte-webapp-svc
instead of 
airbyte-abctl-airbyte-server-svc
, causing nginx to return HTTP 503. [Ingress misconfigured] Your 
abctl local status
earlier confirmed you are on exactly this combination: • 
App Version: 1.8.2
• 
abctl v0.30.1
(with v0.30.3 available) Fix: Upgrade abctl to v0.30.3 The fix was shipped in 
abctl v0.30.3
which specifically includes: > "fix ingress for airbyte 1.8+ must target server instead of webapp" [abctl reliability issue] On your EC2 instance, run:
Copy code
# Upgrade abctl
sudo abctl upgrade

# Then reinstall to apply the corrected ingress
abctl local install
Or if you installed via brew:
Copy code
brew upgrade abctl
abctl local install
Immediate workaround (if you can't upgrade right now) Manually patch the ingress to point to the correct service: [Ingress misconfigured]
Copy code
kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig \
  -n airbyte-abctl \
  edit ingress ingress-abctl
Find the backend service name 
airbyte-abctl-airbyte-webapp-svc
and change it to:
Copy code
airbyte-abctl-airbyte-server-svc
Save and exit — nginx should start routing correctly within seconds. Recommended path: upgrade to 
abctl v0.30.3
(as the status output already suggested) and re-run 
abctl local install
so the ingress is permanently fixed. [Ingress misconfigured] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
p
@kapa.ai will my config be affected if i upgrade it. the config is present in postgres rds
k
Since your configuration is stored in an external PostgreSQL RDS database, upgrading 
abctl
and re-running 
abctl local install
should not affect your data. Here's why: • 
abctl local install
upgrades the Helm chart and fixes the ingress configuration — it does not touch your external RDS database. [Upgrading abctl] • The Airbyte bootloader runs database migrations automatically on startup, but these are additive (schema migrations only) and do not delete your existing configuration data. [Migration failing issue] • One user in a similar setup (external RDS + abctl upgrade) confirmed their configuration survived the upgrade. [abctl migration issue] However, take these precautions before upgrading: 1. Back up your RDS database before running 
abctl local install
. Even though problems are rare, a backup is critical if something goes wrong:pg_dump -h your-rds-host -U your-db-user -d your-db-name > airbyte_backup.sql[Helm chart V2 upgrade guide] 2. Keep your 
values.yaml
 with your external DB configuration intact and pass it during the upgrade:abctl local install --values values.yaml[Upgrading abctl] 3. Do not skip major versions. Your current setup is on 
v1.8.2
. If you plan to go beyond 
v2.0
, the docs explicitly state you must first be on at least v1.6.0 before migrating to Helm chart V2. [Helm chart V2 upgrade] In short: your RDS config is safe, but always back up before any upgrade. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
p
@kapa.ai How do i fix the 503 issue on localhost
k
Your status output confirms: ![](Screenshot 2026-07-16 at 12.18.18.png) • 
App Version: 1.8.2
• 
abctl v0.30.1
→ 
v0.30.4
now available This is the known ingress bug — 
abctl v0.30.1
+ Airbyte 1.8.x generates an ingress pointing to a non-existent 
webapp-svc
, causing the 503. The fix is in 
v0.30.3+
. [Ingress misconfigured] Step 1: Upgrade abctl
Copy code
curl -LsfS https://get.airbyte.com | bash -
Verify:
Copy code
abctl version
Should show 
v0.30.4
. [Upgrading abctl] Step 2: Re-run install (pass your values.yaml if using external RDS)
Copy code
abctl local install --values values.yaml
This will fix the ingress to correctly target 
airbyte-abctl-airbyte-server-svc
instead of the non-existent 
webapp-svc
. [Ingress misconfigured] Optional: Immediate workaround (without upgrading) If you need access right now before upgrading, manually patch the ingress:
Copy code
kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig \
  -n airbyte-abctl \
  edit ingress ingress-abctl
Change:
Copy code
airbyte-abctl-airbyte-webapp-svc
to:
Copy code
airbyte-abctl-airbyte-server-svc
Save and exit — the 503 should resolve within seconds. [Ingress misconfigured] Since your config is on external RDS, your data will not be affected by the 
abctl
upgrade or re-install. Just make sure to pass your 
values.yaml
with the RDS config in Step 2. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
p
@kapa.ai Why was it working for many months earlier?
k
Good question. Based on the knowledge sources, here is the most likely explanation: The webapp service existed until Airbyte 1.8 • In Airbyte 1.6 and earlier, the 
webapp
service was a fully separate, independently deployable service. The ingress correctly pointed to 
airbyte-webapp-svc
and everything worked fine. [Breaking changes] • In Airbyte 1.7, Airbyte began merging 
webapp
into 
server
, but 
webapp
still partially existed. [Ingress 1.7 update] • In Airbyte 1.8, the 
webapp
service was completely removed and fully merged into 
server
. The 
airbyte-webapp
image is no longer published at all. [Breaking changes] What happened to your setup At some point, 
abctl
automatically upgraded your Airbyte from a version where 
webapp
still existed (≤1.7) to 1.8.2, where it no longer exists. But 
abctl v0.30.1
had a bug where it still generated the ingress pointing to the old 
webapp-svc
, which no longer existed — causing the 503. [Ingress misconfigured] So in short: • ✅ Before: ingress → 
webapp-svc
(existed) → worked fine • ❌ After 1.8 upgrade: ingress → 
webapp-svc
(removed) → 503 This is why it worked for months and then suddenly broke — it coincided with the Airbyte 1.8 upgrade removing the 
webapp
service. [Breaking changes] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
p
@kapa.ai i tried abctl local install --values values.yaml . but getting below error
k
Sorry @Pradyumna Kulkarni, I'm not sure what you mean. Please try again. 👋
Your screenshots show two separate warnings during the upgrade to Airbyte 2.1.1: ![](Screenshot 2026-07-16 at 12.32.29.png) ![](Screenshot 2026-07-16 at 12.33.38.png) Warning 1: workload-api-server Unhealthy (Readiness probe failed)
Copy code
Pod: airbyte-abctl-workload-api-server-...
Reason: Unhealthy
Message: Readiness probe failed: Get "http://10.244.0.36:8007/health/liveness": 
context deadline exceeded (Client.Timeout exceeded while awaiting headers)
This is a startup timing issue — the 
workload-api-server
pod was not ready fast enough during the Helm install. This is a known transient issue during upgrades. [abctl migration issue] Warning 2: airbyte-abctl-server BackOff (CrashLoopBackOff)
Copy code
Pod: airbyte-abctl-server-...
Reason: BackOff
Message: Back-off restarting failed container airbyte-server-container
Count: 26958
This is more serious — the server pod is crash-looping. This is commonly caused by the server failing to connect to your external RDS database after the upgrade. [bootloader failed issue] Steps to diagnose and fix: 1. Check current pod status:
Copy code
kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig \
  -n airbyte-abctl get pods
2. Check server pod logs for the root cause:
Copy code
kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig \
  -n airbyte-abctl logs -l app.kubernetes.io/name=server --tail=100
3. Check bootloader logs (runs DB migrations, likely failing to connect to RDS):
Copy code
kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig \
  -n airbyte-abctl logs -l app.kubernetes.io/name=bootloader --tail=100
4. Verify your 
values.yaml
RDS config is correct for Helm chart V2 syntax. The knowledge sources show that upgrading to chart 
2.x
requires updated values syntax. For example, 
global.database.name
(V2) instead of 
global.database.database
(V1): [abctl Helm V2 values issue]
Copy code
global:
  database:
    type: "external"
    host: "<your-rds-host>"
    port: "5432"
    name: "<your-db-name>"          # V2 syntax (not "database")
    secretName: "airbyte-config-secrets"
    userSecretKey: "database-user"
    passwordSecretKey: "database-password"
The knowledge sources do not contain enough detail to pinpoint the exact crash cause without the actual server/bootloader logs. Please share the output of those 
kubectl logs
commands and I can help interpret them further. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
p
@kapa.ai
k
Sorry @Pradyumna Kulkarni, I'm not sure what you mean. Please try again. 👋
p
@kapa.ai I stopped and started my ec2. how do i start abctl again
k
After an EC2 stop/start, the Docker daemon may not have restarted automatically, and the 
abctl
kind cluster needs to be brought back up. Here's what to do: 1. Ensure Docker is running
Copy code
sudo systemctl start docker
Verify:
Copy code
docker ps
[EC2 with abctl] 2. Re-run abctl local install Simply re-run the install command with your values file — this will detect the existing cluster and bring everything back up without affecting your RDS data:
Copy code
abctl local install --values values.yaml
[abctl install] 3. Verify it's back up
Copy code
abctl local status
You should see both 
airbyte-abctl
and 
ingress-nginx
Helm charts with 
Status: deployed
. [abctl local status] Note on EC2 stop/start Every time you stop and start an EC2 instance, the public IP address changes. If you originally installed with 
--host <EC2_PUBLIC_DNS>
, you may need to reinstall with the new hostname:
Copy code
abctl local install --values values.yaml --host <NEW_EC2_PUBLIC_DNS>
[EC2 with abctl] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
p
@kapa.ai i tried everything but i still get error . how to check pod logs or all pod statueses or docker status?
k
Your screenshot shows a clear, consistent error across all pods:
Copy code
Error: couldn't find key CONNECTOR_BUILDER_SERVER_API_HOST 
in ConfigMap airbyte-abctl/airbyte-abctl-airbyte-env
This means the ConfigMap is missing a required key after the upgrade to 2.1.1. This is likely a 
values.yaml
compatibility issue with the new chart version. Check all pod statuses
Copy code
kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig \
  -n airbyte-abctl get pods
Check logs for a specific failing pod For example, for the server pod:
Copy code
kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig \
  -n airbyte-abctl logs airbyte-abctl-server-77cd7d4d54-57rg6 --tail=100
Replace the pod name with the exact name from 
get pods
output. [Troubleshooting abctl] Check the ConfigMap that's missing the key
Copy code
kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig \
  -n airbyte-abctl get configmap airbyte-abctl-airbyte-env -o yaml
This will show you all keys present — you can confirm if 
CONNECTOR_BUILDER_SERVER_API_HOST
is missing. Check Docker status
Copy code
# Is Docker running?
sudo systemctl status docker

# What containers are running?
docker ps

# Docker resource usage
docker system df
[EC2 with abctl] Most likely fix The 
CONNECTOR_BUILDER_SERVER_API_HOST
key missing from the ConfigMap suggests your 
values.yaml
may need updating for chart 
2.1.1
. The connector builder server was renamed to 
manifest-server
in recent versions. [Custom components issue] Try adding this to your `values.yaml`:
Copy code
manifestServer:
  enabled: true
Then re-run:
Copy code
abctl local install --values values.yaml
Please share the output of: 1. 
kubectl ... get pods
2. 
kubectl ... get configmap airbyte-abctl-airbyte-env -o yaml
So I can help narrow down the exact cause based on what the docs say. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
p
@kapa.ai All logs from commands
k
Looking at all three screenshots, here is a full diagnosis: Problem 1: Bootloader cannot connect to RDS (root cause) The bootloader logs show:
Copy code
Caused by: java.net.SocketTimeoutException: Connect timed out
Caused by: HikariPool-2 - Connection is not available, request timed out after 30000ms
The bootloader is timing out trying to reach your RDS database. This is a network connectivity issue between the EC2 instance and your RDS. [bootloader failed] This is likely because: • After EC2 stop/start, the EC2's IP changed, and your RDS security group only allowed the old EC2 IP. Fix: Update your RDS security group to allow inbound traffic on port 
5432
from the new EC2 private IP:
Copy code
# Get current EC2 private IP
curl http://169.254.169.254/latest/meta-data/local-ipv4
Then go to AWS Console → RDS → Security Group → Inbound Rules and update the rule to allow port 
5432
from this new IP. [bootloader failed] Problem 2: All pods in CreateContainerConfigError
Copy code
couldn't find key CONNECTOR_BUILDER_SERVER_API_HOST 
in ConfigMap airbyte-abctl/airbyte-abctl-airbyte-env
This is a secondary issue caused by the upgrade to chart 
2.1.1
— the ConfigMap is missing keys. This will likely resolve itself once you re-run 
abctl local install
correctly after fixing the RDS connectivity. [Custom components issue] Problem 3: Docker pulling images failing
Copy code
Not continuing with pull after error: manifest unknown: manifest unknown
Some images failed to pull during the upgrade. This is a transient issue — re-running install will retry. [abctl install issue] Action Plan Step 1: Fix RDS security group to allow new EC2 IP on port 
5432
. Step 2: Verify RDS is reachable from EC2:
Copy code
nc -zv <your-rds-host> 5432
You should see 
Connection to <host> 5432 port [tcp/postgresql] succeeded!
Step 3: Re-run install:
Copy code
abctl local install --values values.yaml
Step 4: Monitor bootloader logs:
Copy code
kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig \
  -n airbyte-abctl logs airbyte-abctl-bootloader --tail=100 -f
The root cause is RDS connectivity after EC2 stop/start — fix that first and the other issues should resolve. [bootloader failed] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
p
@kapa.ai I fixed the DB connection issue but i have a new error now [ec2-user@ip-172-31-41-61 ~]$ abctl local install --values values.yaml INFO Using Kubernetes provider: Provider: kind Kubeconfig: /home/ec2-user/.airbyte/abctl/abctl.kubeconfig Context: kind-airbyte-abctl SUCCESS Found Docker installation: version 25.0.8 SUCCESS Existing cluster ‘airbyte-abctl’ found SUCCESS Cluster ‘airbyte-abctl’ validation complete INFO Patching image airbyte/db:1.7.0-17 INFO Pulling image airbyte/async-profiler:2.1.1 INFO Pulling image airbyte/bootloader:2.1.1 INFO Pulling image airbyte/connector-sidecar:2.1.1 INFO Pulling image airbyte/container-orchestrator:2.1.1 INFO Pulling image airbyte/cron:2.1.1 INFO Pulling image airbyte/db:1.7.0-17 INFO Pulling image airbyte/manifest-server:7.10.0 INFO Pulling image airbyte/server:2.1.1 INFO Pulling image airbyte/utils:2.1.1 INFO Pulling image airbyte/worker:2.1.1 INFO Pulling image airbyte/workload-api-server:2.1.1 INFO Pulling image airbyte/workload-init-container:2.1.1 INFO Pulling image airbyte/workload-launcher:2.1.1 INFO Pulling image temporalio/auto-setup:1.27.2 INFO Namespace ‘airbyte-abctl’ already exists INFO Persistent volume ‘airbyte-local-pv’ already exists INFO Persistent volume claim ‘airbyte-storage-pvc’ already exists INFO Persistent volume ‘airbyte-volume-db’ already exists INFO Persistent volume claim ‘airbyte-volume-db-airbyte-db-0’ already exists INFO Starting Helm Chart installation of ‘airbyte/airbyte’ (version: 2.1.1) INFO Starting Helm Chart installation of ‘airbyte/airbyte’ (version: 2.1.1) ERROR Failed to install airbyte/airbyte Helm Chart ERROR Unable to install Airbyte locally ERROR unable to install airbyte chart: unable to install helm: cannot patch “airbyte-abctl-server” with kind Deployment: Deployment.apps “airbyte-abctl-server” is invalid: spec.template.spec.containers[0].env[152].valueFrom: Invalid value: “”: may not be specified when 
value
is not empty
k
The error is clear:
Copy code
spec.template.spec.containers[0].env[152].valueFrom: Invalid value: "": 
may not be specified when `value` is not empty
This means one of your environment variables in the Helm chart has both 
value
and 
valueFrom
set at the same time, which Kubernetes does not allow. This is a known issue when upgrading between chart versions with a 
values.yaml
that has conflicting or duplicate env var definitions. [env var conflict issue] Most likely cause Your 
values.yaml
likely has an env var defined with a hardcoded 
value
that the new chart 
2.1.1
is also trying to set via 
valueFrom
(a ConfigMap or Secret reference). The two conflict. [env var conflict issue] Step 1: Identify which env var is conflicting Check what is at index 
[152]
in the server deployment:
Copy code
kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig \
  -n airbyte-abctl get deployment airbyte-abctl-server -o json | \
  jq '.spec.template.spec.containers[0].env[152]'
This will tell you exactly which env var is causing the conflict. Step 2: Check your values.yaml for duplicate/conflicting env vars Look for any env vars in your 
values.yaml
that you may have set with a hardcoded value that the chart also sets via ConfigMap. For example, something like:
Copy code
server:
  env_vars:
    SOME_VAR: "some-value"  # conflicts if chart also sets this via valueFrom
Remove any such overrides that are already managed by the chart itself. Step 3: Clean up the existing deployment and reinstall Since the deployment is in a conflicted state, force-delete it and let Helm recreate it cleanly:
Copy code
kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig \
  -n airbyte-abctl delete deployment airbyte-abctl-server
Then re-run:
Copy code
abctl local install --values values.yaml
Step 4: If the issue persists — full reinstall If the conflict keeps coming back, the safest option (since your data is in RDS) is a full uninstall and reinstall: [Upgrading abctl]
Copy code
abctl local uninstall
abctl local install --values values.yaml
This will not affect your RDS data — it only removes the Kubernetes workloads and Helm releases, not the external database. [External DB] Please share: 1. The output of the 
jq
command (which env var is at index 152), and 2. The relevant section of your 
values.yaml
So I can help identify the exact conflicting variable. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
5 Views
Open in Slack
PreviousNext
Powered by