Hey can someone help me if we can integrate opente...
# connector-contributions
k
Hey can someone help me if we can integrate opentelemetry to collect metrics from airbyte Open source version? As the official guide says its only available with self managed enterprise edition.
j
It looks like they removed metrics from Airbyte OSS, we lost all of our metrics.
u
Metrics are now an enterprise-only feature (read more here)
c
Is there any other recommend way to monitor self-hosted Airbyte deployment?
Kinda a unfortunate that metric is put behind a paywall
j
Airbyte webhooks and Kubernetes deployment monitoring like Datadog etc
c
@Joey Benamy do you directly POST to PagerDuty then?
j
We don't use PagerDuty but unless you need to send your webhook to multiple places then yeah you could just post it to pagerduty directly
Right now we just publish the webhooks to Slack because Datadog can't handle the webhooks directly
s
Hi @[DEPRECATED] Marcos Marx , in future would it be possible to mark in the release notes when a feature is being removed from OSS? I see here that it says 'Self-managed enterprise now generates a number of crucial metrics...', but there's no indication (that I can see), it is being removed from OSS. https://docs.airbyte.com/release_notes/v-1.5#opentelemetry-otel-metrics-monitoring This is a bit of a rug pull, which makes me very wary of doing future upgrades.
c
100% agree with @Steven Murphy... observability is kinda a necessity for self-hosted applications and then having it silently pulled from the source code. Even the console output still wrongly suggests that the metrics endpoint is being pushed every minute but that isn't the case as it has been put behind a paywall
s
For what it's worth, I did some helm chart hackery and deployed version 1.2.0 of the metrics deployment. That has given me back the
num_running_jobs
and
num_pending_jobs
metrics. The rest are still missing, but for me those were the critical ones that give an overall indication of how busy the system is at a given moment.
c
Is it just the metrics deployment that takes care of the opentelemetry metrics? Could you share your solution on running an older version for the metrics deployment please
s
I've not quite got it productionised yet, just got it running last night to prove the idea. Our setup looks like (before the hack):
airbyte-metrics
->
otel_collector
->
service monitor
->
promtheus
->
thanos
->
grafana
otel collector is an image
otel/opentelemetry-collector-contrib
which listens to the metrics being emitted by
airbyte-metrics
, exposes it as prometheus metrics which can then get scraped. Now what I've done is disable
airbyte-metrics
, swap in
airbyte-metrics-12
, the rest is unchanged. The deployment as it stands is below. Apologies it's very very messy as I was playing with various configs, not 100% yet what combination of env vars etc I strictly need yet From what I remember of looking at the metrics source code previously, it gets the metrics from the DB.
Copy code
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/instance: airbyte-metrics-12-metrics
    airbyte: airbyte-metrics-12-metrics
  name: airbyte-metrics-12-metrics
  namespace: mynamespace
spec:
  replicas: 1
  selector:
    matchLabels:
      airbyte: airbyte-metrics-12-metrics
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        airbyte: airbyte-metrics-12-metrics
    spec:
#      affinity:
#        nodeAffinity:
#          requiredDuringSchedulingIgnoredDuringExecution:
#            nodeSelectorTerms:
#            - matchExpressions:
#              - key: dedicated
#                operator: In
#                values:
#                - preemptible
#        podAntiAffinity:
#          requiredDuringSchedulingIgnoredDuringExecution:
#          - labelSelector:
#              matchExpressions:
#              - key: app.kubernetes.io/name
#                operator: In
#                values:
#                - metrics
#            topologyKey: kubernetes.io/hostname
      containers:
      - env:
        - name: AIRBYTE_VERSION
          valueFrom:
            configMapKeyRef:
              key: AIRBYTE_VERSION
              name: airbyte-airbyte-env
        - name: CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION
          valueFrom:
            configMapKeyRef:
              key: CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION
              name: airbyte-airbyte-env
        - name: METRIC_CLIENT
          value: "otel"
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "<http://otel-collector:4317>"
#        - name: MICROMETER_METRICS_ENABLED
#          value: "false"
        - name: MICROMETER_METRICS_OTLP_ENABLED
          value: "true"
        - name: MICROMETER_METRICS_STATSD_ENABLED
          value: "false"
        - name: MICROMETER_METRICS_STATSD_FLAVOR
          value: "datadog"
        - name: MICROMETER_METRICS_STEP
          value: "PT1M"
        - name: MICROMETER_METRICS_ENABLED
          value: "true"

        - name: DATABASE_HOST
          valueFrom:
            configMapKeyRef:
              key: DATABASE_HOST
              name: airbyte-airbyte-env
        - name: DATABASE_PORT
          valueFrom:
            configMapKeyRef:
              key: DATABASE_PORT
              name: airbyte-airbyte-env
        - name: DATABASE_DB
          valueFrom:
            configMapKeyRef:
              key: DATABASE_DB
              name: airbyte-airbyte-env
        - name: DATABASE_USER
          valueFrom:
            secretKeyRef:
              key: DATABASE_USER
              name: airbyte-airbyte-secrets
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: postgresql-password
              name: airbyte-cloudsql-postgres-secret
        - name: DATABASE_URL
          valueFrom:
            configMapKeyRef:
              key: DATABASE_URL
              name: airbyte-airbyte-env
        - name: CONTAINER_ORCHESTRATOR_SECRET_MOUNT_PATH
          value: /secrets/gcs-log-creds
        - name: CONTAINER_ORCHESTRATOR_SECRET_NAME
          value: airbyte-gcs-log-creds
        image: airbyte/metrics-reporter:1.2.0
        imagePullPolicy: IfNotPresent
        name: airbyte-metrics-container
      imagePullSecrets:
      - name: gcr-json-key
      restartPolicy: Always
      serviceAccountName: my-service-account
#      tolerations:
#      - effect: NoSchedule
#        key: dedicated
#        operator: Equal
#        value: preemptible
Metrics for running jobs is lower than previous patterns, still to determine the cause. Thinking: • Could be the metric is now inaccurate as using unsupported approach • Airbyte is now running more efficiently, so gets through jobs faster
c
Wouldn't it be sufficient to just overwrite the metric image with an older version? In the helm chart you should be able to just set metrics.image.tag
s
That would make too much sense 🙂 . I'll give it a shot
c
had another look, it should favor metrics.image.tag and fallback to global.image.tag
please let me know if it works 😄
s
Copy code
values:
    global:
      metrics:
        otlp:
          collectorEndpoint: "<http://otel-collector.mynamespace.svc.cluster.local:4317>"

    metrics:
      enabled: true
      image:
        tag: "1.2.0"

      extraEnv:
        - name: "METRIC_CLIENT"
          value: "otel"
here's the metric values that worked, had to add the extraEnv as that env var was there previously but not now. same result as making my own deployment, not full metrics but limited ones. good enough for now.
c
that is really good to know, thank you 🙂
z
Hi @Steven Murphy @Chris, I'm working on getting Airbyte metrics as well. Followed the instructions for the setup but still no metrics of any kind. Tried with gRPC and HTTP on the OTLP collector. I'm using Airbyte 1.5.1 OSS via Helm chart. Any clues to this mystery?
Copy code
airbyte:
  global:  
    metrics:
      enabled: true
      otlp:
        enabled: true
        collectorEndpoint: "<http://xxx:4317>"
      statsd:
        enabled: false
  metrics:
    enabled: true
    image:
      tag: "1.2.0"
    extraEnv:
      - name: "METRIC_CLIENT"
        value: "otel"
Here are logs from the Airbyte services which tells me it will publish metrics, but no data on our Grafana side. Separately, I've tested the OTLP collector (HTTP) via curl and confirmed working.
Copy code
2025-03-31 15:32:31,120 [main]	INFO	i.m.c.i.p.PushMeterRegistry(start):110 - Publishing metrics for OtlpMeterRegistry every 1m to <http://xxx:4317> with resource attributes {}
The following environment variables are set correctly.
Copy code
[airbyte@airbyte-metrics-79797fdbcd-4xtp4 app]$ env | grep -i -e 'MICROMETER_METRICS' -e 'METRIC_CLIENT' -e 'OTEL_COLLECTOR_ENDPOINT'
MICROMETER_METRICS_OTLP_ENABLED=true
OTEL_COLLECTOR_ENDPOINT=<http://xxx:4317>
MICROMETER_METRICS_STATSD_ENABLED=false
MICROMETER_METRICS_STATSD_FLAVOR=datadog
METRIC_CLIENT=otel
MICROMETER_METRICS_STEP=PT1M
MICROMETER_METRICS_ENABLED=true
s
Hi Zhi, ignoring Grafana for now, can you see metrics appearing on your collector?
z
Hi Steven, I've checked the logs on the collector but no indication of requests from Airbyte. Of course, it was setup by another team so logs may be limited. In the meantime, I'll reach out internally to see if logs can be exposed.
s
what is your collector? i use
otel/opentelemetry-collector-contrib:0.114.0
the logs for that are pretty quiet, they don't show the regular activity coming in. the config is below, it listens on grpc on port 4317 from airbyte. it exposes prometheus metrics on http port 8889 which can get scraped. i can view those metrics on
<http://mycollector:8889/metrics>
Copy code
receivers:
      otlp:
        protocols:
          grpc:
            endpoint: "0.0.0.0:4317"

    processors:
      batch:
        timeout: 5s
        send_batch_size: 50

    exporters:
      prometheus:
        endpoint: "0.0.0.0:8889"

    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheus]
once they get to that point, you can then conclude the extraction of metrics from airbyte is working, then focus on whatever process you need to do in order to get them into grafana
z
Nice. We use Grafana Alloy but the configurations looked similar. I'll check out /metrics to see if there is anything. Thanks.
s
i'm pretty sure we have grafana alloy in our stack somewhere as well 🙂. for this purpose, running my own lightweight collector helped to get the config right.
z
@Steven Murphy I was able to spin up a new otel collector with similar configurations. This allow me to received Airbyte metrics. Still need to check internally why Grafana Alloy setup did not work. Thanks. Here are the manifests for the setup.
Copy code
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: default
  labels:
    app: otel-collector
data:
  otel-collector-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: "0.0.0.0:4317"
          http:
            endpoint: "0.0.0.0:4318"
    exporters:
      prometheus:
        endpoint: "0.0.0.0:8889"
    service:
      pipelines:
        metrics:
          receivers: [otlp]
          exporters: [prometheus]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: default
  labels:
    app: otel-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
        - name: otel-collector
          image: otel/opentelemetry-collector-contrib:0.114.0
          args: ["--config=/conf/otel-collector-config.yaml"]
          ports:
            - containerPort: 4317 # gRPC
            - containerPort: 4318 # HTTP
          volumeMounts:
            - name: config-volume
              mountPath: /conf
      volumes:
        - name: config-volume
          configMap:
            name: otel-collector-config
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  namespace: default
  labels:
    app: otel-collector
spec:
  selector:
    app: otel-collector
  ports:
    - name: grpc
      port: 4317
      targetPort: 4317
    - name: http
      port: 4318
      targetPort: 4318
    - name: metrics
      port: 8889
      targetPort: 8889
---
apiVersion: <http://monitoring.coreos.com/v1|monitoring.coreos.com/v1>
kind: ServiceMonitor
metadata:
  name: otel-collector
  namespace: default
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: otel-collector
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics
Copy code
root@zhi-kdemo:/tools# curl <http://otel-collector.default.svc.cluster.local:8889/metrics>
...
num_active_conn_per_workspace_bucket{job="metrics-reporter",le="0"} 0
...
num_active_conn_per_workspace_sum{job="metrics-reporter"} 244
num_active_conn_per_workspace_count{job="metrics-reporter"} 244
...
num_running_jobs{job="metrics-reporter"} 1
c
@[DEPRECATED] Marcos Marx please check this thread 🙂
s
Nice one @Zhi Shen, that set up is broadly similar to mine
s
Hey all, I'm also trying to collect some metrics in the open source version. Does anyone has an exhaustive list of what all metrics we can collect in the open source version? Till now I only see: 1. est_num_metrics_emitted_by_reporter 2. num_abnormal_scheduled_syncs_last_day 3. num_active_conn_per_workspace 4. num_orphan_running_jobs 5. num_pending_jobs 6. num_running_jobs 7. num_total_scheduled_syncs_last_day 8. num_unusually_long_syncs 9. oldest_pending_job_age_secs 10. oldest_running_job_age_secs
c
One database migration in a recent version and using old version like 1.2 for metrics will break
Best that they undo the decision to put metrics behind a paywall
u
Hello folks I'm going to catch up all it was discussed here!
c
Any update 🙂 ?
Hi @[DEPRECATED] Marcos Marx 🙂 Was wondering if you had find the time to check this thread and provide us with feedback
Hi @[DEPRECATED] Marcos Marx. Hope all is well. Since this has been a while I was wondering if there was an update regarding this
m
Hi @[DEPRECATED] Marcos Marx, bumping this one too, to see if there were any developments on this 🫥
c
Hi @[DEPRECATED] Marcos Marx. Hope all is well. Since this has been a while I was wondering if there was an update regarding this
plus1 1