Hey folks! Anyone encountered an issue with Airbyt...
# replication-ask-ai
w
Hey folks! Anyone encountered an issue with Airbyte sending large amount of custom metrics to the observability platform? I noticed that Airbyte is currently set to emit temporal-related metrics every 120 seconds (see
REPORT_INTERVAL_SECONDS
) but some metrics are sent more than the others. Is this expected or is there a misconfiguration at some place? cc. @Xiaohan Song (Airbyte) if you got any additional context since you updated the value before
k
A message from kapa.ai
Based on the information provided in the extracts, it's not clear if there's a misconfiguration causing some metrics to be sent more frequently than others. However, Airbyte does support exporting built-in metrics to Datadog or OpenTelemetry, and the frequency of these metrics might depend on the specific configuration and the nature of the metrics themselves. Airbyte also allows you to configure the metrics client through environment variables. For instance, you can set the
METRIC_CLIENT
to
datadog
or
otel
and define whether to publish metrics collected by the Metrics Reporter by setting
PUBLISH_METRICS
to
true
or
false
[source](https://docs.airbyte.com/operator-guides/configuring-airbyte). If you're using Datadog, you can set
DD_AGENT_HOST
to the IP where the Datadog agent is running and
DD_DOGSTATSD_PORT
to the port the Datadog agent is using [source](https://docs.airbyte.com/operator-guides/collecting-metrics). If you're using OpenTelemetry, you can set
OTEL_COLLECTOR_ENDPOINT
to the ip:port the OTEL metric client sends metrics to [source](https://docs.airbyte.com/operator-guides/configuring-airbyte). If you're still experiencing issues, it might be helpful to check the metrics Airbyte is sending. You can find a complete list of metrics Airbyte is sending [here](https://github.com/airbytehq/airbyte-platform/blob/main/airbyte-metrics/metrics-lib/src/main/java/io/airbyte/metrics/lib/OssMetricsRegistry.java) [source](https://airbytehq.slack.com/archives/C021JANJ6TY/p1678303133772099). If you're still having trouble, you might want to reach out to the Airbyte community on Slack or [create an issue](https://github.com/airbytehq/airbyte/issues) for further assistance.
w
Example of metrics collected in large (see top 5). Large amount of custom metrics are expensive so we are trying to see if the rate is configurable