Hi! :) First time here, will keep the intro brief...
# general
a
Hi! :) First time here, will keep the intro brief šŸ™‚ We're about to publish a little Druid extension as open source, thinking it might be useful to other people as well. Before we actually publish it though - would be nice if I can get some feedback from the community. You know - it's one thing to use internally a piece of code you've developed, it's completely different when sharing with other people šŸ™‚ Right! So, this extension is basically an InputFormat for OpenTelemetry data - lets you ingest OpenTelemetry into Druid. Over here we find Druid particularly useful for telemetry analytics, and instead of doing complex ingestion specs to process the OpenTelemetry protobufs - we just repurposed a bit of java code that already had. The result is an
otlp
InputFormat extension that flattens and preprocesses OTEL data for easier querying. You can find the pre-release [here.](https://github.com/mishmash-io/opentelemetry-server-embedded/tree/main/druid-otlp-format) I'm pretty sure there are a number of things that can be improved and would be great if I can get some feedback? P.S. Better docs are coming, but will take a bit of time :)
k
This is cool Lets add it to the apache druid source code itself. druid-contribs-extension would be a good place to start.
a
That's alright, personally I think a few things are necessary before that: • possibly a more convenient druid table schema - we'll also publish soon some Apache Superset charts and dashboards that we have around, but I'm still not 100% sure the druid table schemas are as easy to use as they can be • There are a couple of other concerns - like, for example - I don't fully understand schema discovery in druid... The OpenTelemetry signals (with some exceptions) are quite good and stable, so, ideally, I would like to make the otlp format work straight away through schema discovery (for ease of use)... but not sure yet we've actually managed that...
Other than concerns like the above - we're pretty happy to spend a bit of effort promoting the use of OpenTelemetry and Druid... and we don't insist on where the code would be ultimately...