Hello team! Does anyone added distributing tracing...
# troubleshooting
t
Hello team! Does anyone added distributing tracing capabilities to their Flink pipelines, as-in reading Kafka records with tracing headers and writing to Kafka also with tracing headers while preserving the same
traceparent
? Currently, from the work I did it seems that it's very very hard to properly do it, unless you pass tracing metadata as part of the POJO that get's passed from operator to operator, but even in this case, you'll have to "activate" the trace in each operator of your pipeline. I found this to be not very DX friendly and very error prone. We're on Flink 1.18 if it's important. I'm looking for inspiration mainly on how to do it nicer and more DX friendly, so open for suggestions/ideas. Thanks
p
In the past I saw some folks creating a container pojo maintaining all the metadata. But to do so, they create a flavored process function that would only accept containerized POJOs. I achieved it in the past with KafkaStreams. But TLDR you'd have to platformize it in a sense. Are you willing to do something like this or planning something broader? I know datorios have something similar, but they've been on a radio silence lately, so I'm not sure how it's going.