Hey, just wondering what the state of pipeline monitoring support is in Airbyte - I guess I can ship the logs off, and parse the output - but I would ideally want to track such things as entities modified over time. runtimes or such via for example prometheus (I've seen a commit that removed some prometheus stuff and I've found datadog code in your repo as well, but it would be nice to have a statement of what the state+plan is 🙂 - thank you!)

Hi, great question! We are slowly adding metrics to Airbyte as we learn more while operating Cloud. In general our philosophy is to expose as many things as possible in OSS, so expect the list of available metrics to increase over time. You can see the list of available metrics today at: https://github.com/airbytehq/airbyte/blob/master/airbyte-metrics/lib/src/main/java/io/airbyte/metrics/lib/MetricsRegistry.java In terms of supported metric systems, we moved away from promtheus as it was inferior compared to the native DD client. We are currently looking into Open Telemetry, and it is highly likely we’ll be exposing things via OT in the next few months