kk
03/21/2025, 3:52 PMJoey Benamy
03/21/2025, 3:55 PM[DEPRECATED] Marcos Marx
Chris
03/23/2025, 5:15 PMChris
03/23/2025, 5:16 PMJoey Benamy
03/23/2025, 6:17 PMChris
03/24/2025, 8:38 AMJoey Benamy
03/24/2025, 1:29 PMJoey Benamy
03/24/2025, 1:30 PMSteven Murphy
03/26/2025, 5:23 PMChris
03/27/2025, 8:12 AMSteven Murphy
03/27/2025, 9:45 AMnum_running_jobs and num_pending_jobs metrics. The rest are still missing, but for me those were the critical ones that give an overall indication of how busy the system is at a given moment.Chris
03/27/2025, 10:12 AMSteven Murphy
03/27/2025, 10:22 AMairbyte-metrics -> otel_collector -> service monitor -> promtheus -> thanos -> grafana
otel collector is an image otel/opentelemetry-collector-contrib which listens to the metrics being emitted by airbyte-metrics, exposes it as prometheus metrics which can then get scraped.
Now what I've done is disable airbyte-metrics, swap in airbyte-metrics-12, the rest is unchanged.
The deployment as it stands is below. Apologies it's very very messy as I was playing with various configs, not 100% yet what combination of env vars etc I strictly need yet
From what I remember of looking at the metrics source code previously, it gets the metrics from the DB.
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/instance: airbyte-metrics-12-metrics
airbyte: airbyte-metrics-12-metrics
name: airbyte-metrics-12-metrics
namespace: mynamespace
spec:
replicas: 1
selector:
matchLabels:
airbyte: airbyte-metrics-12-metrics
strategy:
type: Recreate
template:
metadata:
labels:
airbyte: airbyte-metrics-12-metrics
spec:
# affinity:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: dedicated
# operator: In
# values:
# - preemptible
# podAntiAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# - labelSelector:
# matchExpressions:
# - key: app.kubernetes.io/name
# operator: In
# values:
# - metrics
# topologyKey: kubernetes.io/hostname
containers:
- env:
- name: AIRBYTE_VERSION
valueFrom:
configMapKeyRef:
key: AIRBYTE_VERSION
name: airbyte-airbyte-env
- name: CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION
valueFrom:
configMapKeyRef:
key: CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION
name: airbyte-airbyte-env
- name: METRIC_CLIENT
value: "otel"
- name: OTEL_COLLECTOR_ENDPOINT
value: "<http://otel-collector:4317>"
# - name: MICROMETER_METRICS_ENABLED
# value: "false"
- name: MICROMETER_METRICS_OTLP_ENABLED
value: "true"
- name: MICROMETER_METRICS_STATSD_ENABLED
value: "false"
- name: MICROMETER_METRICS_STATSD_FLAVOR
value: "datadog"
- name: MICROMETER_METRICS_STEP
value: "PT1M"
- name: MICROMETER_METRICS_ENABLED
value: "true"
- name: DATABASE_HOST
valueFrom:
configMapKeyRef:
key: DATABASE_HOST
name: airbyte-airbyte-env
- name: DATABASE_PORT
valueFrom:
configMapKeyRef:
key: DATABASE_PORT
name: airbyte-airbyte-env
- name: DATABASE_DB
valueFrom:
configMapKeyRef:
key: DATABASE_DB
name: airbyte-airbyte-env
- name: DATABASE_USER
valueFrom:
secretKeyRef:
key: DATABASE_USER
name: airbyte-airbyte-secrets
- name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
key: postgresql-password
name: airbyte-cloudsql-postgres-secret
- name: DATABASE_URL
valueFrom:
configMapKeyRef:
key: DATABASE_URL
name: airbyte-airbyte-env
- name: CONTAINER_ORCHESTRATOR_SECRET_MOUNT_PATH
value: /secrets/gcs-log-creds
- name: CONTAINER_ORCHESTRATOR_SECRET_NAME
value: airbyte-gcs-log-creds
image: airbyte/metrics-reporter:1.2.0
imagePullPolicy: IfNotPresent
name: airbyte-metrics-container
imagePullSecrets:
- name: gcr-json-key
restartPolicy: Always
serviceAccountName: my-service-account
# tolerations:
# - effect: NoSchedule
# key: dedicated
# operator: Equal
# value: preemptibleSteven Murphy
03/27/2025, 10:52 AMChris
03/27/2025, 11:24 AMSteven Murphy
03/27/2025, 11:25 AMChris
03/27/2025, 11:32 AMChris
03/27/2025, 11:33 AMSteven Murphy
03/27/2025, 2:56 PMvalues:
global:
metrics:
otlp:
collectorEndpoint: "<http://otel-collector.mynamespace.svc.cluster.local:4317>"
metrics:
enabled: true
image:
tag: "1.2.0"
extraEnv:
- name: "METRIC_CLIENT"
value: "otel"
here's the metric values that worked, had to add the extraEnv as that env var was there previously but not now.
same result as making my own deployment, not full metrics but limited ones. good enough for now.Chris
03/27/2025, 8:33 PMZhi Shen
03/31/2025, 3:47 PMairbyte:
global:
metrics:
enabled: true
otlp:
enabled: true
collectorEndpoint: "<http://xxx:4317>"
statsd:
enabled: false
metrics:
enabled: true
image:
tag: "1.2.0"
extraEnv:
- name: "METRIC_CLIENT"
value: "otel"
Here are logs from the Airbyte services which tells me it will publish metrics, but no data on our Grafana side. Separately, I've tested the OTLP collector (HTTP) via curl and confirmed working.
2025-03-31 15:32:31,120 [main] INFO i.m.c.i.p.PushMeterRegistry(start):110 - Publishing metrics for OtlpMeterRegistry every 1m to <http://xxx:4317> with resource attributes {}
The following environment variables are set correctly.
[airbyte@airbyte-metrics-79797fdbcd-4xtp4 app]$ env | grep -i -e 'MICROMETER_METRICS' -e 'METRIC_CLIENT' -e 'OTEL_COLLECTOR_ENDPOINT'
MICROMETER_METRICS_OTLP_ENABLED=true
OTEL_COLLECTOR_ENDPOINT=<http://xxx:4317>
MICROMETER_METRICS_STATSD_ENABLED=false
MICROMETER_METRICS_STATSD_FLAVOR=datadog
METRIC_CLIENT=otel
MICROMETER_METRICS_STEP=PT1M
MICROMETER_METRICS_ENABLED=trueSteven Murphy
03/31/2025, 3:49 PMZhi Shen
03/31/2025, 3:52 PMSteven Murphy
03/31/2025, 3:54 PMotel/opentelemetry-collector-contrib:0.114.0
the logs for that are pretty quiet, they don't show the regular activity coming in. the config is below, it listens on grpc on port 4317 from airbyte. it exposes prometheus metrics on http port 8889 which can get scraped. i can view those metrics on <http://mycollector:8889/metrics>
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
processors:
batch:
timeout: 5s
send_batch_size: 50
exporters:
prometheus:
endpoint: "0.0.0.0:8889"
service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus]Steven Murphy
03/31/2025, 3:55 PMZhi Shen
03/31/2025, 3:57 PMSteven Murphy
03/31/2025, 3:58 PMZhi Shen
03/31/2025, 6:38 PMapiVersion: v1
kind: ConfigMap
metadata:
name: otel-collector-config
namespace: default
labels:
app: otel-collector
data:
otel-collector-config.yaml: |
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
exporters:
prometheus:
endpoint: "0.0.0.0:8889"
service:
pipelines:
metrics:
receivers: [otlp]
exporters: [prometheus]
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: otel-collector
namespace: default
labels:
app: otel-collector
spec:
replicas: 1
selector:
matchLabels:
app: otel-collector
template:
metadata:
labels:
app: otel-collector
spec:
containers:
- name: otel-collector
image: otel/opentelemetry-collector-contrib:0.114.0
args: ["--config=/conf/otel-collector-config.yaml"]
ports:
- containerPort: 4317 # gRPC
- containerPort: 4318 # HTTP
volumeMounts:
- name: config-volume
mountPath: /conf
volumes:
- name: config-volume
configMap:
name: otel-collector-config
---
apiVersion: v1
kind: Service
metadata:
name: otel-collector
namespace: default
labels:
app: otel-collector
spec:
selector:
app: otel-collector
ports:
- name: grpc
port: 4317
targetPort: 4317
- name: http
port: 4318
targetPort: 4318
- name: metrics
port: 8889
targetPort: 8889
---
apiVersion: <http://monitoring.coreos.com/v1|monitoring.coreos.com/v1>
kind: ServiceMonitor
metadata:
name: otel-collector
namespace: default
labels:
release: prometheus
spec:
selector:
matchLabels:
app: otel-collector
endpoints:
- port: metrics
interval: 30s
path: /metrics
root@zhi-kdemo:/tools# curl <http://otel-collector.default.svc.cluster.local:8889/metrics>
...
num_active_conn_per_workspace_bucket{job="metrics-reporter",le="0"} 0
...
num_active_conn_per_workspace_sum{job="metrics-reporter"} 244
num_active_conn_per_workspace_count{job="metrics-reporter"} 244
...
num_running_jobs{job="metrics-reporter"} 1Chris
04/01/2025, 7:29 AMSteven Murphy
04/01/2025, 8:59 AMShashank Wadhwa
04/02/2025, 9:34 AMChris
04/02/2025, 11:02 AMChris
04/02/2025, 11:03 AM[DEPRECATED] Marcos Marx
Chris
04/07/2025, 7:16 AMChris
04/09/2025, 7:45 AMChris
04/29/2025, 7:28 AMMonika Bednarz
08/13/2025, 12:21 AMChris
08/20/2025, 12:34 PM