https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> How can I install airbyte OSS on an...
# replication-ask-ai
g
@kapa.ai How can I install airbyte OSS on an EKS cluster? What are the best practices? do you have a guideline I can follow?
k
Installing Airbyte OSS on EKS Yes, Airbyte can be deployed on Kubernetes, including Amazon EKS. Here's a guideline based on the official documentation: Prerequisites Before starting, ensure you have the following ready: • An Amazon EKS cluster running on EC2 instances (note: EKS on Fargate is not supported) [Infrastructure Prerequisites] • Instances with at least 2 cores and 8 GB of RAM (memory-optimized instances like M7i/M7g are recommended) [Infrastructure Prerequisites] Required tools: • helm • kubectl • AWS CLI • eksctl Step-by-Step Guide 1. Configure kubectl to connect to your EKS cluster
Copy code
eksctl utils write-kubeconfig --cluster=$CLUSTER_NAME
kubectl config get-contexts
kubectl config use-context $EKS_CONTEXT
[Infrastructure Prerequisites] 2. Create a Kubernetes namespace for Airbyte
Copy code
kubectl create namespace airbyte
3. Add the Airbyte Helm Repository
Copy code
helm repo add airbyte https://airbytehq.github.io/helm-charts
helm repo update
helm search repo airbyte
[Installation Guide] 4. Install Airbyte via Helm Helm chart V1:
Copy code
helm install airbyte airbyte/airbyte \
  --namespace airbyte \
  --values ./values.yaml
Helm chart V2 (for newer versions, e.g., platform 2.0):
Copy code
# First, find the chart version matching your desired Airbyte version
helm search repo airbyte-v2 --versions

# Then install
helm install airbyte airbyte-v2/airbyte \
  --namespace airbyte-v2 \
  --values ./values.yaml \
  --version 2.0.18
[Installing Airbyte] Best Practices • Use multiple availability zones: Run your EKS cluster across 2 or more availability zones for resiliency. [Infrastructure Prerequisites] • Use memory-optimized instances: M7i or M7g instance types are recommended. • Use a dedicated namespace: Isolating Airbyte in its own namespace is considered good practice. [Installation Guide] • Customize via 
values.yaml
: Fill out a 
values.yaml
file for your specific configuration before running the Helm install. • Configure Ingress: After installation, set up an Ingress (e.g., Amazon ALB) to expose the Airbyte UI. [Installing Airbyte] • External components (recommended for production): Amazon RDS Postgres (dedicated database), Amazon S3 (log/state storage), and Amazon Secrets Manager (connector secrets). [Infrastructure Prerequisites] For the full Helm chart usage guide, refer to airbytehq.github.io/helm-charts. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
I want to use s3 for blob storage, and a db in rds for the aribyte_db. whcih values do I change. provide me with a values file
@kapa.ai how do i setup the values.yaml for postgres rds and s3?
k
Here's how to configure your 
values.yaml
for an external PostgreSQL RDS instance and S3 storage on EKS: 1. External PostgreSQL (RDS) First, create a Kubernetes secret with your database credentials:
Copy code
apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  database-user: "your-db-user"
  database-password: "your-db-password"
Then add the following to your `values.yaml`: Helm chart V1:
Copy code
postgresql:
  enabled: false

global:
  database:
    type: external
    secretName: "airbyte-config-secrets"
    host: "your-rds-endpoint.rds.amazonaws.com"
    port: "5432"
    database: "airbyte_db"
    userSecretKey: "database-user"
    passwordSecretKey: "database-password"
Helm chart V2 (note: the database name key changed from 
database
to 
name
):
Copy code
postgresql:
  enabled: false

global:
  database:
    type: external
    secretName: "airbyte-config-secrets"
    host: "your-rds-endpoint.rds.amazonaws.com"
    port: 5432
    name: "airbyte_db"  # key changed from "database" in V1
    userSecretKey: "database-user"
    passwordSecretKey: "database-password"
[External Database] 2. S3 Log/State Storage Using IAM credentials:
Copy code
global:
  storage:
    type: "S3"
    secretName: airbyte-config-secrets
    bucket:
      log: airbyte-bucket
      state: airbyte-bucket
      workloadOutput: airbyte-bucket
    s3:
      region: "us-east-1"
      authenticationType: credentials
      accessKeyId: ""       # Helm chart V2 only
      secretAccessKey: ""   # Helm chart V2 only
Using an IAM instance profile (recommended for EKS with IAM roles):
Copy code
global:
  storage:
    type: "S3"
    secretName: airbyte-config-secrets
    bucket:
      log: airbyte-bucket
      state: airbyte-bucket
      workloadOutput: airbyte-bucket
    s3:
      region: "us-east-1"
      authenticationType: instanceProfile
Set 
authenticationType
to 
instanceProfile
if your EKS nodes already have an IAM role with permissions to read/write the S3 bucket — this avoids managing static credentials. [State and Logging Storage] Combined Example (Helm chart V2)
Copy code
postgresql:
  enabled: false

global:
  database:
    type: external
    secretName: "airbyte-config-secrets"
    host: "your-rds-endpoint.rds.amazonaws.com"
    port: 5432
    name: "airbyte_db"
    userSecretKey: "database-user"
    passwordSecretKey: "database-password"

  storage:
    type: "S3"
    secretName: airbyte-config-secrets
    bucket:
      log: airbyte-bucket
      state: airbyte-bucket
      workloadOutput: airbyte-bucket
    s3:
      region: "us-east-1"
      authenticationType: instanceProfile
[External Database; State and Logging Storage] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
how do I know if i'm on v1 or v2 helm?
@kapa.ai how do I know if i'm on v1 or v2 helm?
k
You can check which Helm chart version you're currently running by using this command:
Copy code
helm list -n <NAMESPACE>
Replace 
<NAMESPACE>
with the namespace where Airbyte is installed (e.g., 
airbyte
or 
airbyte-v2
). [Upgrading Airbyte] The output will show the chart version. You can then cross-reference it: • Helm chart V1 uses the repository 
https://airbytehq.github.io/helm-charts
(added as 
airbyte
) and chart versions like 
1.x.x
(e.g., 
1.8.0
). [Add the Helm Repository] • Helm chart V2 uses the repository 
https://airbytehq.github.io/charts
(added as 
airbyte-v2
) and chart versions like 
2.x.x
(e.g., 
2.0.18
). [Add the Helm Repository] So if your 
helm list
output shows a chart version starting with 
1.
, you're on V1; if it starts with 
2.
, you're on V2. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai what's the config for s3 helm v1?
k
Here is the S3 configuration for Helm chart V1 in your `values.yaml`:
Copy code
global:
  storage:
    type: "S3"
    secretName: airbyte-config-secrets  # Name of your Kubernetes secret
    bucket:
      log: airbyte-bucket
      state: airbyte-bucket
      workloadOutput: airbyte-bucket
    s3:
      region: ""  # e.g. us-east-1
      authenticationType: credentials  # Use "credentials" or "instanceProfile"
[State and Logging Storage] You'll also need a Kubernetes secret with your S3 credentials (expected under the keys 
s3-access-key-id
and 
s3-secret-access-key
by default):
Copy code
apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  s3-access-key-id: "AKIAIOSFODNN7EXAMPLE"
  s3-secret-access-key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
[State and Logging Storage] > Tip: If your EKS nodes have an IAM role with the necessary S3 permissions, you can set 
authenticationType: instanceProfile
instead of 
credentials
to avoid managing static keys. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai the connection test with snowflake is taking a long time. It's asking 'me to wait a liitle bit more ' for over 5 minutes now
k
Based on the knowledge sources available, there are a few known causes for Snowflake connection tests hanging or taking a very long time: Possible Causes & Fixes 1. Network/Firewall Restrictions The 
destination-snowflake
connector may be attempting to connect to external endpoints (e.g., Azure Blob Storage staging endpoints like 
sdgp7dsfcb1stg.blob.core.windows.net
) during connection verification. If your network has firewall rules blocking these, the connection test will hang indefinitely. [destination-snowflake issue] Fix: Check your firewall rules and ensure the required external endpoints are allowed through. 2. Snowflake Network Policies Snowflake may be blocking the connection attempt based on IP restrictions. [Network policies] Fix: Run the following in Snowflake to check if a network policy is restricting access:
Copy code
-- Check account-level network policy
SHOW PARAMETERS LIKE 'network_policy' IN ACCOUNT;

-- Check user-level network policy
SHOW PARAMETERS LIKE 'network_policy' IN USER <username>;
If a network policy exists, make sure Airbyte's IP addresses are on the allowlist. 3. Incorrect Credentials or Permissions A missing or misconfigured database/schema can also cause the connection to hang or fail. [connection stopped working] Fix: Verify that the Snowflake user has the correct permissions, and that the database, schema, warehouse, and role are all correctly specified in your connector configuration. 4. Outdated Connector Version Using an outdated connector version has been noted as a potential cause of connection issues. [Snowflake connection issue] Fix: Check if your 
destination-snowflake
connector is up to date and upgrade if needed. If none of these resolve the issue, I'd recommend checking the Airbyte logs for more specific error messages, as they will give you a clearer picture of where exactly the connection is failing. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai there's no network policy setup
k
Since there's no Snowflake network policy in place, the most likely cause of your hanging connection test is a firewall/network restriction blocking the connector's external connection attempts. As noted in the open GitHub issue, the 
destination-snowflake
connector attempts to connect to external staging endpoints (e.g., Azure Blob Storage) during connection verification. If these endpoints are blocked by your network firewall, the connection test will hang indefinitely. [destination-snowflake issue] What to check: 1. Review your firewall/egress rules — The connector was observed trying to reach endpoints like 
*.blob.core.windows.net
(Azure Blob Storage). Ensure outbound traffic to Snowflake's staging endpoints is allowed from your AKS cluster. 2. Check Snowflake activity logs — Look at Snowflake's query history to see if any connection or staging activity is being attempted or blocked. 3. Verify connector configuration — Double-check that the database, schema, warehouse, and role are all correctly specified. Missing or incorrect values can also cause issues. [connection stopped working] 4. Check the Airbyte sync logs — The logs may show exactly which external endpoint the connector is trying to reach, which will help you identify the specific firewall rule needed. Since you're on AKS, you may need to work with your network/security team to allow egress to the required Snowflake staging endpoints. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai for my workload api server deployment, why would I have this error: Warning Unhealthy 9m47s (x2184 over 6h44m) kubelet spec.containers{airbyte-workload-api-server-container}: Startup probe failed: Get "http://172.31.71.91:8007/health/liveness": dial tcp 172.31.71.918007 connect: connection refused Normal Killing 8m47s (x75 over 6h42m) kubelet spec.containers{airbyte-workload-api-server-container}: Container airbyte-workload-api-server-container failed startup probe, will be restarted Warning BackOff 3m56s (x213 over 6h20m) kubelet spec.containers{airbyte-workload-api-server-container}: Back-off restarting failed container airbyte-workload-api-server-container in pod airbyte-workload-api-server-5c7855f585-p8qk8_airbyte(d296760d-ed56-4a6f-8f8b-accfd2513feb) Normal Pulled 20s (x77 over 6h44m) kubelet spec.containers{airbyte-workload-api-server-container}: Container image "airbyte/workload-api-server:2.0.1" already present on machine and can be accessed by the pod
k
The error indicates that the 
airbyte-workload-api-server
container is failing its startup probe because nothing is listening on port 
8007
— the container is crashing before it can start serving traffic, causing a restart loop. Likely Causes 1. Probe returning 404 (known issue with v2.0.x) There is a known issue reported with the 2.0.0/2.0.1 chart where the workload-api-server's liveness/readiness/startup probes return 404, causing the container to be killed before it fully starts. [Multiple issues with 2.0.0 chart] Workaround: Disable the startup/liveness/readiness probes temporarily in your `values.yaml`:
Copy code
workload-api-server:
  livenessProbe:
    enabled: false
  readinessProbe:
    enabled: false
2. Container crashing on startup due to misconfiguration The container may be crashing before it can bind to port 8007. A known cause is a missing environment variable — for example, 
STORAGE_BUCKET_AUDIT_LOGGING
— which causes the server pod to crash with a variable templating error. [Multiple issues with 2.0.0 chart] Check: Run the following to see the container logs before it crashes:
Copy code
kubectl logs -n airbyte <workload-api-server-pod-name> --previous
3. Connection to the workload API server failing from other components There is also a reported issue where the workload API server fails to connect internally (e.g., 
Failed to connect to airbyte-helm-workload-api-server-svc/10.x.x.x:8007
), which may indicate a service or networking misconfiguration. [Error installing via helm chart] Recommended Steps 1. Check the pod logs with 
kubectl logs ... --previous
to identify the root crash cause. 2. Verify your S3/storage configuration is complete (including 
STORAGE_BUCKET_AUDIT_LOGGING
if applicable). 3. Consider temporarily disabling the probes to allow the container to start and diagnose further. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai how do I forward the port to view the ui?
k
Based on your setup (EKS with Helm), here's how to port-forward to access the Airbyte UI: Helm chart V1:
Copy code
kubectl -n airbyte port-forward deployment/airbyte-server 8080:8001
Helm chart V2:
Copy code
kubectl -n airbyte-v2 port-forward deployment/airbyte-server 8080:8001
Then open your browser at: **http://127.0.0.1:8080** [Installation Guide] > Note: If you deployed to a custom namespace, replace 
airbyte
or 
airbyte-v2
with your actual namespace name. Port-forwarding requires keeping the terminal window open while you use the UI. If you want a more permanent solution instead of port-forwarding, you can set up an AWS ALB ingress — which is the recommended approach for EKS. [Ingress] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai can you provide me with a values.yaml file that avoids all issues and give a perfect deployment the first time?
k
Based on the conversation history (EKS, Helm chart V2, S3 with instance profile, external RDS), here is a consolidated 
values.yaml
that incorporates all the configurations discussed and best practices from the documentation:
Copy code
global:
  edition: community

  airbyteUrl: "https://your-airbyte-domain.example.com"  # Replace with your actual URL

  # External PostgreSQL (RDS)
  database:
    type: external
    secretName: "airbyte-config-secrets"
    host: "your-rds-endpoint.rds.amazonaws.com"  # Replace
    port: 5432
    name: "airbyte"  # Replace with your DB name
    userSecretKey: "database-user"
    passwordSecretKey: "database-password"

  # External S3 Storage (using IAM instance profile - recommended for EKS)
  storage:
    secretName: "airbyte-config-secrets"
    type: s3
    bucket:
      log: airbyte-bucket           # Replace with your bucket name
      state: airbyte-bucket
      workloadOutput: airbyte-bucket
      activityPayload: airbyte-bucket
    s3:
      region: "us-east-1"           # Replace with your region
      authenticationType: instanceProfile

# Disable internal Postgres
postgresql:
  enabled: false

# Disable internal Minio (since we're using S3)
minio:
  enabled: false
And the required Kubernetes secret:
Copy code
apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
  namespace: airbyte  # Match your deployment namespace
type: Opaque
stringData:
  database-user: "your-db-user"
  database-password: "your-db-password"
[External Database; Configuring External Logging] A few important notes: • The knowledge sources do not provide a single "complete" reference 
values.yaml
that guarantees a perfect deployment — the above is assembled from the documented configurations in the knowledge sources. • Make sure your EKS node IAM role has 
s3:GetObject
, 
s3:PutObject
, 
s3:DeleteObject
, and 
s3:ListBucket
permissions on your S3 bucket for 
instanceProfile
auth to work. • Replace all placeholder values (
your-rds-endpoint
, 
airbyte-bucket
, 
us-east-1
, etc.) with your actual values. • If you're on Self-Managed Enterprise, you'll also need to add 
edition: enterprise
, your license key secret, SSO config, and the 
auth
block as described in the Enterprise implementation guide. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai something that avoids liveliness probe issues
k
Based on the knowledge sources, the liveness/readiness probe configuration is available in the 
values.yaml
reference. To avoid the startup/liveness probe issues seen with the workload API server, you can disable or tune the probes in your `values.yaml`:
Copy code
global:
  edition: community

  airbyteUrl: "https://your-airbyte-domain.example.com"

  database:
    type: external
    secretName: "airbyte-config-secrets"
    host: "your-rds-endpoint.rds.amazonaws.com"
    port: 5432
    name: "airbyte"
    userSecretKey: "database-user"
    passwordSecretKey: "database-password"

  storage:
    secretName: "airbyte-config-secrets"
    type: s3
    bucket:
      log: airbyte-bucket
      state: airbyte-bucket
      workloadOutput: airbyte-bucket
      activityPayload: airbyte-bucket
    s3:
      region: "us-east-1"
      authenticationType: instanceProfile

postgresql:
  enabled: false

minio:
  enabled: false

# Disable probes on the workload API server to avoid startup failures
workloadApiServer:
  livenessProbe:
    enabled: false
  readinessProbe:
    enabled: false
[Values.yaml reference] > Important caveats: 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai on-snowflake-check-3741b443-5d8f-4eec-b183-61f073d417c0-0-qcnep is stuck in init
k
Based on the knowledge sources, a pod stuck in the 
init
state (specifically the init container phase) is a known issue in Kubernetes-based Airbyte deployments. The init container must complete before the main container starts, and if it hangs, the entire pod is stuck. Known Causes 1. Kubernetes cannot pull the connector image If the init container is waiting on image availability or network access to pull the image, it will hang. [All connectors fail test] Check:
Copy code
kubectl describe pod on-snowflake-check-3741b443-5d8f-4eec-b183-61f073d417c0-0-qcnep -n airbyte
Look for 
Events
at the bottom — it will tell you if it's stuck on 
Pulling image
or 
ErrImagePull
. 2. Timed out waiting for init pod to run There is a documented error where Airbyte times out waiting for the init container to become ready: [Temporal Scheduler issue]
Copy code
Timed out waiting for [300000] milliseconds for [Pod] with name:[destination-snowflake-check-...]
This typically happens when the Kubernetes cluster is under resource pressure or the node can't schedule the pod in time. 3. Network/firewall blocking the init container As discussed earlier in your case, the 
destination-snowflake
connector attempts external connections during verification. If the init container itself can't reach required endpoints, it may hang. [destination-snowflake issue] Steps to Diagnose
Copy code
# Check pod status and events
kubectl describe pod <pod-name> -n airbyte

# Check init container logs specifically
kubectl logs <pod-name> -n airbyte -c init

# Check node resource availability
kubectl get nodes
kubectl describe node <node-name>
Quick Fix If the pod is stuck, delete it to force a reschedule:
Copy code
kubectl delete pod <pod-name> -n airbyte
Airbyte will recreate it. If it keeps happening, the root cause is likely a resource or network issue on your EKS cluster. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai sh-4.2$ kubectl get pods -n airbyte -l airbyte=job-pod,job_type=check -w NAME READY STATUS RESTARTS AGE on-snowflake-check-0f8a7fb4-1d93-43cf-a036-3ab29eb48ebb-0-bfhca 0/2 Init:0/1 0 26s
k
The pod is stuck in 
Init:0/1
, meaning the init container (
airbyte/workload-init-container
) has not completed. Based on the knowledge sources, here are the most likely causes and steps to diagnose: Diagnose the Init Container
Copy code
# Check events and status
kubectl describe pod on-snowflake-check-0f8a7fb4-1d93-43cf-a036-3ab29eb48ebb-0-bfhca -n airbyte

# Check init container logs directly
kubectl logs on-snowflake-check-0f8a7fb4-1d93-43cf-a036-3ab29eb48ebb-0-bfhca -n airbyte -c init
Known Causes 1. Timed out waiting for init pod to run — Airbyte waits up to 300 seconds for the init container to become ready. If the cluster is under resource pressure or the node can't schedule it in time, it will hang and eventually fail with a timeout. [Temporal Scheduler issue] 2. Missing environment variables in the init container — A known issue where the init container does not receive required AWS credentials (e.g., 
AWS_ACCESS_KEY_ID
, 
AWS_SECRET_ACCESS_KEY
) that other containers do receive, causing it to fail silently. [EKS HTTP 500 issue] 3. Node architecture mismatch — If your EKS cluster uses Karpenter and provisions 
arm64
nodes, the 
airbyte/workload-init-container
image may not have an 
arm64
manifest, causing it to fail. You can force 
amd64
scheduling: [Failed to create pod]
Copy code
global:
  jobs:
    kube:
      nodeSelector:
        kubernetes.io/arch: amd64
4. Insufficient cluster resources — The init container may be pending because no node has enough CPU/memory to schedule it. [File Source stuck] Quick Fix Delete the stuck pod to force a reschedule:
Copy code
kubectl delete pod on-snowflake-check-0f8a7fb4-1d93-43cf-a036-3ab29eb48ebb-0-bfhca -n airbyte
The 
kubectl describe
output (especially the 
Events
section) will be the most useful thing to share if the issue persists. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai Give instructionsn as well as the values.yml to deploy airbyte with helm v2
k
Here are the instructions and a 
values.yaml
for deploying Airbyte with Helm chart V2, incorporating all the configurations discussed in this conversation (EKS, external RDS, S3 with instance profile, and probe fixes). Step 1: Add the Helm V2 Repository
Copy code
helm repo add airbyte-v2 https://airbytehq.github.io/charts
helm repo update
[Add the Helm Repository] Step 2: Create a Kubernetes Namespace
Copy code
kubectl create namespace airbyte-v2
Step 3: Create the Kubernetes Secret
Copy code
apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
  namespace: airbyte-v2
type: Opaque
stringData:
  database-user: "your-db-user"
  database-password: "your-db-password"
Apply it:
Copy code
kubectl apply -f secret.yaml
[External Database] Step 4: Create your values.yaml
Copy code
global:
  edition: community

  airbyteUrl: "https://your-airbyte-domain.example.com"  # Replace with your URL

  # External PostgreSQL (RDS)
  database:
    type: external
    secretName: "airbyte-config-secrets"
    host: "your-rds-endpoint.rds.amazonaws.com"  # Replace
    port: 5432
    name: "airbyte"          # Replace with your DB name (note: V2 uses "name", not "database")
    userSecretKey: "database-user"
    passwordSecretKey: "database-password"

  # External S3 Storage (IAM instance profile - recommended for EKS)
  storage:
    secretName: "airbyte-config-secrets"
    type: s3
    bucket:
      log: airbyte-bucket           # Replace with your bucket name
      state: airbyte-bucket
      workloadOutput: airbyte-bucket
      activityPayload: airbyte-bucket
    s3:
      region: "us-east-1"           # Replace with your region
      authenticationType: instanceProfile

  # Force amd64 scheduling to avoid architecture mismatch on init containers
  jobs:
    kube:
      nodeSelector:
        kubernetes.io/arch: amd64

# Disable internal Postgres
postgresql:
  enabled: false

# Disable internal Minio (using S3 instead)
minio:
  enabled: false

# Disable probes on workload API server to avoid startup failures
workloadApiServer:
  livenessProbe:
    enabled: false
  readinessProbe:
    enabled: false
[External Database; Values.yaml reference] Step 5: Find the Right Chart Version
Copy code
helm search repo airbyte-v2 --versions
This lists available chart versions and their corresponding Airbyte app versions. [Installing Airbyte] Step 6: Install Airbyte
Copy code
helm install airbyte airbyte-v2/airbyte \
  --namespace airbyte-v2 \
  --values ./values.yaml \
  --version 2.0.18   # Replace with your desired chart version
[Installing Airbyte] Key Notes • In Helm V2, the database name key is 
global.database.name
(not 
database
as in V1). [Upgrade to Helm chart V2] • 
instanceProfile
auth for S3 requires your EKS node IAM role to have the necessary S3 permissions. • Disabling probes is a workaround for known startup issues — re-enable them once your deployment is stable. [Values.yaml reference] • Replace all placeholder values before deploying. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai Does xmin in postgres db require write access?
k
No, the xmin replication method does not require write access. It only requires read-only (
SELECT
) access to the source tables. From the documentation, the xmin method is described as a read-only approach — in fact, the WAL disk consumption troubleshooting page explicitly contrasts it with solutions that require write access: > "The easiest way to fix this issue is to add one or more tables with high transaction volumes to the Airbyte publication... However, this may greatly increase disk consumption. If you do not want to increase disk consumption, use the following solutions, which require write access." This implies the xmin method itself does not require write access. [Postgres Troubleshooting] The standard setup for xmin only requires granting the Airbyte user read permissions:
Copy code
GRANT USAGE ON SCHEMA <schema_name> TO <user_name>;
GRANT SELECT ON ALL TABLES IN SCHEMA <schema_name> TO <user_name>;
ALTER DEFAULT PRIVILEGES IN SCHEMA <schema_name> GRANT SELECT ON TABLES TO <user_name>;
[Cloud SQL for PostgreSQL] Note that xmin does have other limitations to be aware of — it cannot detect row deletions, requires a full table scan on each sync, and can have issues with transaction ID (XID) wraparound. [Postgres Troubleshooting] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai trying to connect to an aurora db, this was the outpout: 2026-03-23 114326 INFO 2026-03-23 114326 info 2026-03-23 114330 INFO 2026-03-23 114330 info Connector exited, processing output 2026-03-23 114330 INFO 2026-03-23 114330 info Output file jobOutput.json found 2026-03-23 114330 INFO 2026-03-23 114330 info ----- START CHECK ----- 2026-03-23 114330 INFO 2026-03-23 114330 info 2026-03-23 114330 INFO 2026-03-23 114330 info Connector exited with exit code 0 2026-03-23 114330 INFO 2026-03-23 114330 info Reading messages from protocol version 0.2.0 2026-03-23 114330 INFO 2026-03-23 114330 info INFO main i.a.i.s.p.PostgresSource(main):711 starting source: class io.airbyte.integrations.source.postgres.PostgresSource 2026-03-23 114330 INFO 2026-03-23 114330 info INFO main i.a.c.i.b.IntegrationCliParser$Companion(parseOptions):144 integration args: {check=null, config=/config/connectionConfiguration.json} 2026-03-23 114330 INFO 2026-03-23 114330 info INFO main i.a.c.i.b.IntegrationRunner(runInternal):130 Running integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedSource 2026-03-23 114330 INFO 2026-03-23 114330 info INFO main i.a.c.i.b.IntegrationRunner(runInternal):131 Command: CHECK 2026-03-23 114330 INFO 2026-03-23 114330 info INFO main i.a.c.i.b.IntegrationRunner(runInternal):132 Integration config: IntegrationConfig{command=CHECK, configPath='/config/connectionConfiguration.json', catalogPath='null', statePath='null'} 2026-03-23 114330 WARN 2026-03-23 114330 warn WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword 2026-03-23 114330 WARN 2026-03-23 114330 warn WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword group - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword 2026-03-23 114330 WARN 2026-03-23 114330 warn WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword 2026-03-23 114330 WARN 2026-03-23 114330 warn WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword always_show - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword 2026-03-23 114330 WARN 2026-03-23 114330 warn WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword pattern_descriptor - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword 2026-03-23 114330 WARN 2026-03-23 114330 warn WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword 2026-03-23 114330 WARN 2026-03-23 114330 warn WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword display_type - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword 2026-03-23 114330 WARN 2026-03-23 114330 warn WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword min - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword 2026-03-23 114330 WARN 2026-03-23 114330 warn WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword max - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword 2026-03-23 114330 WARN 2026-03-23 114330 warn WARN main c.n.s.UnknownKeywordFactory(lambda$getKeyword$0):37 Unknown keyword groups - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword or if it should generate annotations AnnotationKeyword 2026-03-23 114330 INFO 2026-03-23 114330 info INFO main i.a.c.i.b.s.SshTunnel$Companion(getInstance):424 Starting connection with method: NO_TUNNEL 2026-03-23 114330 INFO 2026-03-23 114330 info INFO main i.a.i.s.p.PostgresUtils(isCdc):70 using CDC: false 2026-03-23 114330 INFO 2026-03-23 114330 info INFO main i.a.i.s.p.PostgresSource(toSslJdbcParamInternal):937 DISABLED toSslJdbcParam disable 2026-03-23 114330 INFO 2026-03-23 114330 info INFO main c.z.h.HikariDataSource(<init>):79 HikariPool-1 - Starting... 2026-03-23 114330 INFO 2026-03-23 114330 info INFO main c.z.h.HikariDataSource(<init>):81 HikariPool-1 - Start completed. 2026-03-23 114330 INFO 2026-03-23 114330 info INFO main c.z.h.HikariDataSource(close):349 HikariPool-1 - Shutdown initiated... 2026-03-23 114330 INFO 2026-03-23 114330 info INFO main c.z.h.HikariDataSource(close):351 HikariPool-1 - Shutdown completed. 2026-03-23 114330 INFO 2026-03-23 114330 info INFO main i.a.c.i.b.IntegrationRunner(runInternal):224 Completed integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedSource 2026-03-23 114330 INFO 2026-03-23 114330 info INFO main i.a.i.s.p.PostgresSource(main):713 completed source: class io.airbyte.integrations.source.postgres.PostgresSource 2026-03-23 114330 INFO 2026-03-23 114330 info Checking for optional control message... 2026-03-23 114330 INFO 2026-03-23 114330 info Optional control message not found. Skipping... 2026-03-23 114330 INFO 2026-03-23 114330 info Writing output of decd338e-5647-4c0b-adf4-da0e75f5a750_99b14412-35ae-463a-9173-e5b5ce0e9ad0_0_check to the doc store 2026-03-23 114330 INFO 2026-03-23 114330 info Marking workload decd338e-5647-4c0b-adf4-da0e75f5a750_99b14412-35ae-463a-9173-e5b5ce0e9ad0_0_check as successful 2026-03-23 114330 INFO 2026-03-23 114330 info 2026-03-23 114330 INFO 2026-03-23 114330 info Deliberately exiting process with code 0. 2026-03-23 114330 INFO 2026-03-23 114330 info ----- END CHECK ----- 2026-03-23 114330 INFO 2026-03-23 114330 info
k
Good news — your connection check to Aurora succeeded! The logs show: • 
HikariPool-1 - Start completed.
— the connection pool was established successfully • 
Completed integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedSource
— the check completed cleanly • 
Marking workload ... as successful
— Airbyte marked the check as successful • Exit code 
0
— no errors About the WARN messages The 
Unknown keyword order/group/airbyte_secret/multiline/...
warnings are harmless and appear in virtually every Airbyte connector run. They are related to JSON schema validation and do not affect functionality. Aurora-Specific Considerations Now that the connection check passes, keep in mind these known limitations when using AWS Aurora with Airbyte: • CDC caching layer: AWS Aurora has a CDC caching layer that is incompatible with Airbyte's CDC implementation. If you plan to use CDC, you must disable it by setting the rds.logical_wal_cache parameter to 
0
in your Aurora parameter group. [Postgres Troubleshooting] • Timeout errors: If you encounter timeout errors, set apg_write_forward.idle_session_timeout to 
1200000
(20 minutes) in the Aurora parameter group. [Postgres Troubleshooting] You're good to proceed with setting up your sync! 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by