https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT>, I have deployed Airbyte OSS in GKE...
# replication-ask-ai
g
@kapa.ai, I have deployed Airbyte OSS in GKE cluster. now i need to enable autoscaling , what is configuratiojn needed in values.yaml file ?
k
Airbyte’s Helm 
values.yaml
exposes Horizontal Pod Autoscaler (HPA) toggles per component (e.g., worker, workload-launcher, metrics, etc.). In the snippets you shared, each of these sections has an 
hpa.enabled
flag that is disabled by default. [[Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)] From the available docs, the only explicit configuration we can see is:
Copy code
yaml worker:   hpa:     enabled: false  workload-launcher:   hpa:     enabled: false  metrics:   hpa:     enabled: false
To enable autoscaling, you would set the relevant component’s HPA flag to 
true
in your 
values.yaml
, for example:
Copy code
yaml worker:   hpa:     enabled: true  workload-launcher:   hpa:     enabled: true
However, the knowledge sources do not document any additional HPA parameters (like min/max replicas, CPU utilization, etc.), nor do they show the full HPA schema for these sections. Because of that, I cannot reliably tell you which other keys (if any) are supported beyond 
enabled
. Given this limitation: - You can safely toggle 
hpa.enabled: true
for the components you want to autoscale as shown above. [[Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)] - For detailed HPA configuration (min/max replicas, metrics), the docs point you to the full 
charts/v2/airbyte/values.yaml
in GitHub, but its contents are not included in the provided knowledge, so I cannot quote or describe it further. [[Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)] If you need exact fields for HPA tuning, the current knowledge sources do not contain enough information; you’ll need to inspect the 
charts/v2/airbyte/values.yaml
file in the Airbyte repo directly.
g
@kapa.ai, what does replicaCount and maxNotifyWorkers: 5 represents?
k
A message from kapa.ai
replicaCount
and 
maxNotifyWorkers
are Helm chart values used to configure Airbyte components, but the docs you provided only partially explain them. ### 
replicaCount
For multiple components (e.g., 
worker
, 
workload-launcher
, 
metrics
, 
connector-rollout-worker
), 
replicaCount
is documented as: > “Number of \<component\> replicas” [[Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)] So: - 
replicaCount
= how many pod replicas of that component Kubernetes should run (when HPA is disabled). - Example:
Copy code
yaml worker:   replicaCount: 2
…means 2 worker pods. [[Scaling Airbyte](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte)] ### 
maxNotifyWorkers: 5
maxNotifyWorkers
appears in several sections with the comment: > 
## current no exist documentations
> 
maxNotifyWorkers: 5
[[Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)] The docs do not explain what 
maxNotifyWorkers
does (no description, no behavior details). Because of that, I cannot reliably say what it controls beyond that it is a numeric setting defaulting to 
5
. So: - 
replicaCount
is clearly “number of pod replicas” for that component. - For 
maxNotifyWorkers
, the knowledge sources do not contain enough information to describe its purpose or impact.
g
@kapa.ai, I have the below config, but still hpa is not enabled worker: enabled: true nodeSelector: cloud.google.com/gke-nodepool: airbyte-job-workloads-001 tolerations: - key: "nodetype" operator: "Equal" value: "airbyte-job-workloads" effect: "NoSchedule" resources: requests: cpu: 500m memory: 1Gi limits: cpu: 2000m memory: 4Gi hpa: enabled: true minReplicas: 2 maxReplicas: 6 targetCPUUtilizationPercentage: 70 targetMemoryUtilizationPercentage: 70
k
A message from kapa.ai
The knowledge sources don’t show the full HPA schema for the 
worker.hpa
block, nor do they describe how/when the chart actually creates an HPA object. They only show that an 
hpa.enabled
flag exists and defaults to 
false
for several components (worker, workload-launcher, metrics, connector-rollout-worker). [[Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)] Because of that, I cannot confirm that the extra keys you’re setting (
minReplicas
, 
maxReplicas
, 
targetCPUUtilizationPercentage
, 
targetMemoryUtilizationPercentage
) are recognized by the chart, or that 
hpa.enabled: true
alone is sufficient to create an HPA. From what is documented: - 
worker.hpa.enabled
is a boolean toggle, default 
false
. [[Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)] - There is no explicit documentation in the provided sources about: - required/optional HPA fields, - default min/max replicas, - which metrics are used, - or any prerequisites (e.g., metrics server). Given these gaps, here’s what you can reliably do from the docs: 1. Ensure 
worker.enabled: true
and 
worker.hpa.enabled: true
in 
values.yaml
(which you already have). [[Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)] 2. Verify that you’re using the chart version that actually supports 
worker.hpa
(the docs mention it for both 1.8 and 2.0, but the rendered templates are not included in the knowledge, so I can’t confirm behavior). [[1.8 values](https://docs.airbyte.com/platform/1.8/deploying-airbyte/values); [2.0 values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/values)] 3. After deploying, check in your cluster whether an HPA object was created:
Copy code
bash    kubectl get hpa -n <your-namespace>
The docs don’t describe this step, but it’s the only way to confirm from Kubernetes’ side. Because the chart’s HPA template and full 
values.yaml
schema are not present in the knowledge sources, I can’t say why HPA is not appearing in your case or whether your specific keys are correct for this chart. To debug further you’ll need to inspect the actual 
charts/v2/airbyte/templates
and 
charts/v2/airbyte/values.yaml
in the Airbyte repo, which are referenced but not included in the provided documentation. [[2.0 values](https://docs.airbyte.com/platform/2.0/deploying-airbyte/values)]
g
@kapa.ai, deployed OSS version of the Airbyte in GKE , but the UI is directly going to homepage without login screen . what configuration i need to have to get the login page?
k
A message from kapa.ai
Airbyte OSS authentication is controlled via the 
global.auth.enabled
flag in your Helm 
values.yaml
. By default it is disabled, which is why you’re going straight to the homepage without a login screen. [[Auth integration](https://docs.airbyte.com/platform/deploying-airbyte/integrations/authentication); [Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)] To get the login page, enable auth and (optionally) configure the instance admin:
Copy code
yaml global:   auth:     enabled: true     instanceAdmin:       secretName: "airbyte-config-secrets"   # or your secret       firstName: "Admin"       lastName: "User"       emailSecretKey: "instance-admin-email"       passwordSecretKey: "instance-admin-password"
Then: 1. Make sure the 
airbyte-auth-secrets
(or your chosen secret) exists, or let Airbyte generate credentials and read them with:
Copy code
bash    kubectl get secret airbyte-auth-secrets -n <YOUR_NAMESPACE> -o yaml
Decode 
instance-admin-password
with 
base64 -d
. [[Auth integration](https://docs.airbyte.com/platform/deploying-airbyte/integrations/authentication)] 2. Redeploy/upgrade the Helm release with the updated 
values.yaml
. After this, accessing the UI should show the login screen instead of going directly to the homepage.
g
@kapa.ai, is the below config correct?
Copy code
auth:
    # -- Whether auth is enabled
    enabled: true
    # -- Admin user configuration
    instanceAdmin:
      secretName: "airbyte-config-secrets"
      emailSecretKey: "instance-admin-email"
      passwordSecretKey: "instance-admin-password"
k
A message from kapa.ai
Yes, that config matches what’s documented for enabling auth via Helm values. From the 
values.yaml
reference, the relevant section is:
Copy code
yaml global:   auth:     enabled: false     instanceAdmin:       secretName: "airbyte-config-secrets"       firstName: ""       lastName: ""       emailSecretKey: "instance-admin-email"       passwordSecretKey: "instance-admin-password"
[[Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)] So your snippet is structurally correct, but make sure: 1. It is nested under 
global:
(i.e. 
global.auth
, not top-level 
auth
):
Copy code
yaml    global:      auth:        enabled: true        instanceAdmin:          secretName: "airbyte-config-secrets"          emailSecretKey: "instance-admin-email"          passwordSecretKey: "instance-admin-password"
2. The Kubernetes secret 
airbyte-config-secrets
exists in the same namespace and contains the keys: - 
instance-admin-email
- 
instance-admin-password
[[Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)] If those two conditions are met, this is the correct configuration to enable the login page.
g
@kapa.ai , now i am getting the login screen after enabling , but it is defaulting my email address from where it picks up the email ?
Copy code
global:
     auth:
       enabled: true
k
A message from kapa.ai
The knowledge sources don’t fully document how the initial email is chosen when you enable 
global.auth.enabled: true
without specifying 
instanceAdmin
settings, so I can’t say with certainty exactly where your specific email is coming from. What is documented: - When you enable auth via Helm, the recommended way to control the initial user is to set 
global.auth.instanceAdmin
and back it with a secret that contains the email and password. [[Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)] - The docs only explicitly describe controlling the password via the 
airbyte-auth-secrets
secret (
instance-admin-password
key). They do not describe how the email is chosen by default. [[Auth integration](https://docs.airbyte.com/platform/deploying-airbyte/integrations/authentication)] - In one discussion, a user who enabled auth later on found they had to manually update the 
email
field in the 
organization
table in the database to change the initial email. [[Initial user email](https://github.com/airbytehq/airbyte/discussions/72222)] Given that: - The docs do not state whether the default email is: - something you entered earlier in a setup flow, - a hardcoded value, - or derived from any environment variable. - They only show how to override it after the fact (by editing the DB) or by using 
instanceAdmin
+ secrets. To explicitly control the login email going forward, the documented options are: 1. Use 
instanceAdmin
with a secret that includes the email key:
Copy code
yaml    global:      auth:        enabled: true        instanceAdmin:          secretName: "airbyte-config-secrets"          emailSecretKey: "instance-admin-email"          passwordSecretKey: "instance-admin-password"
And in your secret:
Copy code
yaml    apiVersion: v1    kind: Secret    metadata:      name: airbyte-config-secrets    type: Opaque    stringData:      instance-admin-email: <mailto:you@example.com|you@example.com>      instance-admin-password: your-password
[[Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)] 2. If auth was enabled after initial deployment and you already have an organization, update the 
email
column in the 
organization
table in the Airbyte database, as described in the discussion. [[Initial user email](https://github.com/airbytehq/airbyte/discussions/72222)] Beyond that, the knowledge sources do not explain the internal logic that produced the email you’re currently seeing.
g
@kapa.ai, which table stores the login details ?
k
Airbyte stores login-related data across a couple of tables in the config database: 1. 
user
table – stores the core user profile, including the login email: ◦ 
email
(unique) – the user’s email address used for login. ◦ Other fields: 
id
, 
name
, 
status
, etc. [User table] 2. 
auth_user
table – links a 
user
to an authentication identity: ◦ 
user_id
– FK to 
user(id)
. ◦ 
auth_user_id
– ID of the authenticated user at the auth provider. ◦ 
auth_provider
– which auth provider is used. [Auth user table] The documentation does not describe where or how the password hash is stored, only the email and auth linkage as above. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai, how and when does the data inserted into the above tables ?
k
The knowledge sources only describe the structure of the 
auth_user
table, not the lifecycle of how/when rows are created, so I can’t give a complete, precise sequence of events. What is documented: • 
auth_user
is in the config database and links an internal 
user
record to an authenticated identity: it has 
user_id
(FK to 
user(id)
), 
auth_user_id
(ID of the authenticated user), and 
auth_provider
. [Auth user] • For the initial admin user when you enable built‑in auth, Airbyte uses the email/password configured via: ◦ 
global.auth.instanceAdmin
in 
values.yaml
, and ◦ the Kubernetes secret (e.g. 
airbyte-auth-secrets
/ 
airbyte-config-secrets
) that holds 
instance-admin-password
(and optionally the email). [Authentication; Values.yaml] • There is also a community discussion where a user changed the initial email by directly updating the 
email
field in the 
organization
table, implying that the initial organization/user records are created automatically when auth is enabled and Airbyte is first set up. [Initial user email] What is not explicitly documented: • The exact trigger for inserting rows into 
user
and 
auth_user
(e.g., “on first login”, “on first startup after enabling auth”, etc.). • The precise order in which 
user
, 
auth_user
, and 
organization
rows are created. • How password hashes are stored or updated. So, based on the docs: • We know 
auth_user
is populated as part of Airbyte’s authentication flow to associate an authenticated identity with a 
user
record. [Auth user] • We know the initial admin’s credentials come from the auth configuration and secrets when auth is enabled. [Authentication; Values.yaml] But the knowledge sources do not describe the detailed “how and when” insertion logic beyond that, so I can’t provide a more specific timeline without going beyond the available documentation. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai, can we insert the records manually into these tables?
k
The knowledge sources only document the schema of the 
auth_user
table and related config DB tables; they do not describe or endorse manually inserting records into them. [Auth user] Because of that, I cannot say it is safe or supported to manually insert rows into 
user
/ 
auth_user
(or related auth tables). The only documented direct DB change around auth is a community workaround where a user updated an existing email in the 
organization
table, not inserted new auth records. [Auth user] So, based on the available docs: • There is no explicit guidance that manual inserts into these tables are supported. • Doing so risks breaking internal constraints or leaving the system in an inconsistent state. If you need additional users or to change auth behavior, the documentation points to configuring auth via 
global.auth
and the associated secrets, not via direct inserts into the config DB. [Authentication] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai, i need to add additional user for login ,how can i do that
k
The knowledge sources don’t describe any supported way to create additional login users directly in Self‑Managed OSS via the UI or API. They only cover: • How to enable/disable built‑in auth and set the initial admin via 
global.auth.instanceAdmin
and Kubernetes secrets. [Values.yaml ref; Auth integration] • How to manage workspace members (invite users by email) in Cloud / Self‑Managed Enterprise, not OSS. [Workspaces] For OSS: • There is no documented procedure to: ◦ Create additional local login users, ◦ Or safely insert users directly into the config DB. Because of that, I can’t give you a reliable, supported method to add another login user in OSS beyond the single instance admin created at setup. If you need multi‑user access with proper user management, the documented path is to use Self‑Managed Enterprise or Cloud, where user and workspace management (inviting additional users) is supported in the UI. [Workspaces] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai, the organization table has my email id , can i updated it to a commin admin email , since there is no ability to add more users? do i need to change the user table also ?
k
You can update the email in the database, but the docs only show this as a workaround and do not fully describe all side effects, so treat it as an unsupported/manual change. What is documented: • The 
organization
table has an 
email
column (“Contact email for the organization”). [Org table] • A user who enabled auth later and wanted to change the login email did exactly what you’re proposing: they updated the 
email
field in the 
organization
table, and confirmed that this changed the login email. [Initial user email] From the available information: • Changing 
organization.email
to a common admin email is a known workaround and has been used successfully. [Initial user email] • The docs do not say you must also change anything in the 
user
table for this specific purpose, and the referenced workaround only mentions updating 
organization.email
, not 
user
. [Initial user email; User table] So, based on the docs: • Yes, you can update 
organization.email
to a shared admin email to change which email is used for login. • There is no explicit requirement in the documentation to also update the 
user
table for this to work, and the documented workaround did not do so. • This remains an internal DB modification, not an officially supported multi‑user mechanism, so do it carefully (backup DB first, test after change). 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai, with the below configuration in values.yaml file the bootloader is not coming up, but if i move secretName under instanceAdmin: then the bootloader is coming up but the secrets are not taken from "airbyte-config-secrets" secret and is failing with the error "Error: couldn't find key instance-admin-password in Secret edp-airbyte/airbyte-airbyte-secrets"
Copy code
auth:
    # -- Whether auth is enabled
    enabled: true
    secretName: "airbyte-config-secrets"
    # -- Admin user configuration
    instanceAdmin:    
      emailSecretKey: "instance-admin-email"
      passwordSecretKey: "instance-admin-password"
k
You’re running into two separate, but related, Helm/auth issues that are (unfortunately) known limitations/bugs in the current chart. 1. Where 
secretName
is actually supported In the documented 
values.yaml
schema, 
secretName
is only defined under 
global.auth.instanceAdmin
, not directly under `auth`:
Copy code
global:
  auth:
    enabled: false
    instanceAdmin:
      # -- Secret name where the instanceAdmin configuration is stored
      secretName: "airbyte-config-secrets"
      firstName: ""
      lastName: ""
      emailSecretKey: "instance-admin-email"
      passwordSecretKey: "instance-admin-password"
[Values ref] So this is the documented, supported structure:
Copy code
global:
  auth:
    enabled: true
    instanceAdmin:
      secretName: "airbyte-config-secrets"
      emailSecretKey: "instance-admin-email"
      passwordSecretKey: "instance-admin-password"
Putting 
secretName
at the same level as 
enabled
(as in your first example) is not part of the documented schema and explains why the bootloader won’t start in that case. 2. Why it still looks for 
airbyte-airbyte-secrets
/ 
airbyte-auth-secrets
 Even when you move 
secretName
under 
instanceAdmin
, you’re seeing: > 
couldn't find key instance-admin-password in Secret edp-airbyte/airbyte-airbyte-secrets
This matches a known bug in the Helm chart: • The chart ignores 
global.auth.instanceAdmin.secretName
and hard‑codes the secret name for the admin password to 
airbyte-auth-secrets
(or the release‑scoped equivalent like 
airbyte-airbyte-secrets
). [Helm auth bug; Helm secrets bug] • The bootloader template uses:
Copy code
- name: AB_INSTANCE_ADMIN_PASSWORD
      valueFrom:
        secretKeyRef:
          name: "airbyte-auth-secrets"  # hard-coded
          key: "web-instance-admin-password"
instead of using the 
secretName
from your values. [Helm secrets bug] So even though you correctly point 
instanceAdmin.secretName
to 
airbyte-config-secrets
, the bootloader still looks in the chart’s own secret (
airbyte-airbyte-secrets
/ 
airbyte-auth-secrets
) and fails if that secret doesn’t contain 
instance-admin-password
. [Helm auth bug] 3. What you can do with the current chart Given what’s in the docs and issues, the workarounds are: 1. Use the chart’s expected secret name Put your admin credentials into the secret the chart actually uses (e.g. 
airbyte-auth-secrets
or the release‑scoped 
airbyte-airbyte-secrets
), with the expected key names (e.g. 
instance-admin-password
). [Helm auth bug] 2. Override the bootloader env manually (documented workaround) In the open issue, they work around the hard‑coded secret by overriding 
AB_INSTANCE_ADMIN_PASSWORD
via 
airbyte-bootloader.extraEnv
and pointing it to `airbyte-config-secrets`:
Copy code
airbyte-bootloader:
     extraEnv:
       - name: AB_INSTANCE_ADMIN_PASSWORD
         valueFrom:
           secretKeyRef:
             name: "airbyte-config-secrets"
             key: "web-instance-admin-password"
[Helm secrets bug] You’d adapt the 
key
to match your own (
instance-admin-password
). Because of these chart bugs, your configuration:
Copy code
auth:
  enabled: true
  secretName: "airbyte-config-secrets"
  instanceAdmin:
    emailSecretKey: "instance-admin-email"
    passwordSecretKey: "instance-admin-password"
is: • Structurally unsupported (wrong nesting) per the docs. • Even when corrected to 
global.auth.instanceAdmin.secretName
, the chart currently does not honor that secret name for the password; it still expects its own secret, which is why you see the error against 
airbyte-airbyte-secrets
. [Values ref; Helm secrets bug] So, to get it working today, you’ll need to either: • Put the password into the secret the chart is actually using, or • Use the 
airbyte-bootloader.extraEnv
override workaround above. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
g
@kapa.ai, should i use "instance-admin-password" instead of "web-instance-admin-password" ?
Copy code
airbyte-bootloader:
     extraEnv:
       - name: AB_INSTANCE_ADMIN_PASSWORD
         valueFrom:
           secretKeyRef:
             name: "airbyte-config-secrets"
             key: "web-instance-admin-password"
k
You should use the *key name that actually exists in your secret*; the docs show 
instance-admin-password
as the standard key. In the authentication docs, when setting the password via a Kubernetes secret, the example uses: [Auth integration]
Copy code
apiVersion: v1
kind: Secret
metadata:
  name: airbyte-auth-secrets
type: Opaque
stringData:
  instance-admin-password: # password
So if your 
airbyte-config-secrets
secret is defined like this:
Copy code
stringData:
  instance-admin-password: my-password
then your override should be:
Copy code
airbyte-bootloader:
  extraEnv:
    - name: AB_INSTANCE_ADMIN_PASSWORD
      valueFrom:
        secretKeyRef:
          name: "airbyte-config-secrets"
          key: "instance-admin-password"
The 
web-instance-admin-password
key name comes from a user’s custom example in a GitHub issue, not from the official docs, so only use it if you actually created your secret with that exact key. [Helm secrets bug; Auth integration] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
5 Views
Open in Slack
PreviousNext
Powered by