Apache Flink

Happy New Year folks, has anyone tried using <https://opentelemetry.io> from Flink for monitoring?

Curious about this one as well. It's on my plans to get to work on Observability and Open Telemetry with Flink, but havent tried it yet

There’s an open ticket for adding opentelemetry support but it doesn’t look to have had much traction :disappointed:
<https://issues.apache.org/jira/browse/FLINK-22390>

Here's a project that uses the rest api to collect Flink metrics <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/flinkmetricsreceiver>

I've been looking into this as well. I think the most "bang-for-our-buck" at the moment would be to run an OTel collector alongside Flink and configure it to scrape from the existing Prometheus Exporter.

A new OTel reporter might be more interesting as people want more context added to metrics and a unified collection of other types of telemetry data (traces, logs, etc.).

How do we go about proposing a new feature for Flink and suggesting changes?

The docs have a nice guide to this: <https://flink.apache.org/contributing/contribute-code.html|https://flink.apache.org/contributing/contribute-code.html>
A FLIP would likely be necessary here, but in general start w/ a mailing list discussion 

Thanks for your comments folks, I'll dig into this topic further myself too. I hope we can get something good out of it.

Let's collaborate and jot down our ideas for tackling this project!

I'll initiate the conversation and circulate it on the mailing list soon.

I sent the email yesterday, and I would greatly appreciate your feedback.

I saw your email <@U03HLDK57V5>, let's see if there's any buzz around the topic.

Since the current metric system is pretty much baked into Flink this is O(infinity) complexity. And as you rightfully pointed out there's a lot to assess firts.

What's your motivation behind OpenTelemetry?