Hi @kapa.ai, I am trying to collect the sync logs into Loki. In my case I added labels and annotations as below, to both the worker and server. While the logs for both worker and server are successfully collected, the

replication-job

pod logs are not. Seems to be the case where the the vector label is not being injected into the replication-job pods, which is similar to this github.com/airbytehq/airbyte/issues/33068 issue. I'm currently on airbyte 1.8.5; is there any workaround or solution to this?

# -- Add extra annotations to the worker pods
  podAnnotations:
    vector.dev/sinks: loki

  # -- Add extra labels to the worker pods
  podLabels:
    vector.dev/include: "true"

Thinking completed The reason your

replication-job

pods are not receiving those labels and annotations is because configuring

podAnnotations

and

podLabels

under

worker

or

server

only applies them to the static deployment pods (the worker and server deployments themselves), not to the ephemeral pods spawned dynamically to run jobs [Github Issue (closed): [compose/api] Error upgrading Airbyte helm chart from 0.63.11 to 1.3.0]. In Airbyte, job/workload pods (like

replication-job

pods) inherit their annotations and labels from the job configuration section. You can apply labels and annotations to these pods using either of the following approaches: Solution 1: Use global.jobs.kube in Helm values In your Helm

values.yaml

, define the labels and annotations under the

jobs.kube

block (or

global.jobs.kube

depending on your chart setup) [Github Issue (closed): [helm] Workload pods are missing the service account; Github Issue (closed): [compose/api] Error upgrading Airbyte helm chart from 0.63.11 to 1.3.0]:

global:
  jobs:
    kube:
      annotations:
        vector.dev/sinks: loki
      labels:
        vector.dev/include: "true"

Under the hood, these values populate the

JOB_KUBE_ANNOTATIONS

and

JOB_KUBE_LABELS

environment variables that Airbyte uses when generating the pod specs for sync/replication jobs [Github Issue (closed): [compose/api] Error upgrading Airbyte helm chart from 0.63.11 to 1.3.0]. Solution 2: Filter in Vector using Airbyte's default pod labels (Recommended) Airbyte automatically tags all workload/job pods with standard labels, including

airbyte=job-pod

and

job_type=sync

[Metadata.kt; Collect Logs from a Flex Data Plane]. Instead of relying on injecting custom labels into the pod definitions, you can configure Vector to collect logs directly using Airbyte's default pod label [Setting Up Log Collection]:

sources:
  kubernetes_logs:
    type: kubernetes_logs
    extra_label_selector: "airbyte=job-pod"

Note on log output: In sync/replication pods, stdout for the

source

and

destination

containers is empty because it is reserved for internal Airbyte protocol messages. The human-readable logs are emitted by the

orchestrator

container inside the replication pod [Container Reference]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai My bad I forgot to mention that I've already inserted the label and annotaiton in the jobs config

kube:
      ## JOB_KUBE_ANNOTATIONS
      # pod annotations of the sync job and the default pod annotations fallback for others jobs
      # -- key/value annotations applied to kube jobs
      annotations:
        vector.dev/sinks: loki

      ## JOB_KUBE_LABELS
      ## pod labels of the sync job and the default pod labels fallback for others jobs
      # -- key/value labels applied to kube jobs
      labels:
        vector.dev/include: "true"

But the ephemeral pods are still not being lablled correctly. I can't really make direct changes to the vector config so is there a way to make the first solution work?

Thinking completed The provided knowledge sources do not explicitly explain why custom

labels

or

annotations

set under

global.jobs.kube

fail to propagate to workload launcher-spawned ephemeral pods (

replication-job

), though known issues exist where properties passed through

JOB_KUBE_*

variables or Helm values are not applied to ephemeral workload pods [Github Issue (closed): [helm] Workload pods are missing the service account; Github Issue (open): Kube jobs annotations are not rendered correctly.]. Under Airbyte's workload-launching architecture, the component responsible for creating the dynamic

replication-job

pods is the Workload Launcher [Workload launching architecture; Github Issue (closed): workload-launcher false positive liveness probes under lost storage connection]. Here are a few workarounds and troubleshooting steps to make the first solution work: 1. Pass the environment variables directly to workload-launcher via extraEnv Job properties set in Helm are translated to environment variables in comma-separated

key=value

format (such as

JOB_KUBE_LABELS

and

JOB_KUBE_ANNOTATIONS

) [Jobs]. If the launcher isn't receiving them from the default ConfigMap, pass them explicitly to the launcher deployment via

extraEnv

[Update your values.yaml file; Github Issue (closed): [compose/api] Error upgrading Airbyte helm chart from 0.63.11 to 1.3.0]:

workload-launcher: # or workloadLauncher depending on your chart version
  extraEnv:
    - name: JOB_KUBE_LABELS
      value: "vector.dev/include=true"
    - name: JOB_KUBE_ANNOTATIONS
      value: "vector.dev/sinks=loki"

If your chart uses the legacy worker rather than the workload-launcher to start pods, apply the same

extraEnv

under

worker.extraEnv

as well [Github Issue (closed): [compose/api] Error upgrading Airbyte helm chart from 0.63.11 to 1.3.0]. 2. Verify environment variables inside the launcher pod Check whether the

workload-launcher

pod has picked up the variable definitions:

kubectl get pod -l app.kubernetes.io/name=workload-launcher -o yaml | grep -A2 JOB_KUBE_LABELS

Also check the ConfigMap (

airbyte-airbyte-env

or similar) to ensure the values are present [Github Issue (closed): [helm] Workload pods are missing the service account]. 3. Check for unsetting in launcher startup logs The workload launcher logs unsets on empty variables upon boot [Github Issue (open): Replication pod creation fails (KubeClientException) but UI/Public API still reports sync as successful]. Check the container logs:

kubectl logs -l app.kubernetes.io/name=workload-launcher --tail=100

Ensure it is not discarding or logging an error regarding

JOB_KUBE_LABELS

or

JOB_KUBE_ANNOTATIONS

during startup. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai The ConfigMap contains the label and annotation. Adding extraEnv to the workload launcher is not a good idea as my ArgoCD will see duplicates and throw an error. I can confirm that the workload launcher does contain the variable definitions

Thinking completed The knowledge sources do not provide an explicit in-application fix or Helm configuration option for why the

workload-launcher

fails to propagate

JOB_KUBE_LABELS

and

JOB_KUBE_ANNOTATIONS

to dynamically created replication pods, even when the environment variables are correctly present on the launcher pod. There are documented cases where configuration variables (such as

JOB_KUBE_NODE_SELECTORS

or

JOB_KUBE_SERVICEACCOUNT

) are present in both the ConfigMap and the launcher environment, yet fail to be applied to the dynamically launched workload pods [Github Issue (closed): [helm] Workload pods are missing the service account]. Additionally, formatting and rendering bugs around job annotations exist across several chart versions [Github Issue (open): Kube jobs annotations are not rendered correctly.]. If you cannot modify the Vector configuration and the launcher ignores these environment variables during pod creation, consider the following workarounds outside the Airbyte Helm chart: 1. Use a Kubernetes Mutating Admission Webhook (Kyverno or OPA Gatekeeper) If you have a policy engine running in your cluster, define a mutating rule that matches pods by name prefix (e.g.,

name: replication-job-*

) or existing labels (like

airbyte=job-pod

) within your Airbyte namespace and automatically injects: ◦ Label:

vector.dev/include: "true"

◦ Annotation:

vector.dev/sinks: loki

1. Check for Character Escaping or Formatting Conflicts Airbyte parses annotations and labels as comma-separated

key=value

pairs [Jobs-specific]. Although

vector.dev/include="true"

and

vector.dev/sinks=loki

appear standard, special characters (such as slashes

/

or periods

.

) in the key name can occasionally cause parsing issues within the pod template builder [Github Issue (open): Kube jobs annotations are not rendered correctly.]. Inspect the launcher logs during the

BUILD

and

LAUNCH

stages to see how the pod specification is constructed [Github Issue (closed): workload-launcher false positive liveness probes under lost storage connection; Github Issue (open): Replication pod creation fails (KubeClientException) but UI/Public API still reports sync as successful]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai can I know if this issue is being looked at at all? A lot folks seem to have encountered the same issue here github.com/airbytehq/airbyte/issues/33068

Thinking completed I'm sorry, but the provided knowledge sources do not contain information regarding GitHub issue #33068 or whether that specific issue is currently being actively investigated by the Airbyte team. The sources do show related open and tracked issues regarding pod creation and workload launcher propagation discrepancies (such as #72448 being escalated to internal on-call tracking [Github Issue (open): Replication pod creation fails (KubeClientException) but UI/Public API still reports sync as successful]), but they do not mention the current status, assignment, or roadmap priority for issue #33068. To check the latest status or bring visibility to it, commenting directly on GitHub issue #33068 with your Airbyte version and use case is the best way to prompt the Airbyte maintainers for an update [Github Issue (open): [helm] Cannot start replication pods when using a service account not named airbyte-admin (Airbyte OSS 1.3.1)]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.