Hi Team, is there any document detailing how to run the Overlord as a standalone service. we are not able to get the right executable and config to run the overlord. Or is there any config to run the overlord on a different port than the cordinator. Any help is highly appreciated. Thanks

Hello there, going trough the Basic cluster tunning guide Overlord/Coordinator's sections there is no mention of the Direct Memory buffers, does that means that it's safe to run without passing these flag

-XX:MaxDirectMemorySize

?

Hey everybody! Druid Summit is back and IN-PERSON this year – Tuesday, October 22, 2024 in Redwood City, California! This is your chance to join your fellow Druid community members, developers, and data professionals as they share experiences with real-time analytics solutions powered by the Apache Druid database – maybe even share your own! This year’s event will feature keynotes from Druid experts, educational sessions, interactive spaces, and real-world case studies of Druid in production. Want to speak at Druid Summit? Submit your 25 or 45 minute talk by August 5: https://bit.ly/45Qh4iv

Hello everyone, I wanted to know, if I want to backup druid, what exactly do I need to take backup? Deep Storage, metadata storage or segment cache?

Hello druid family, I am looking for the way to remove the duplicates from a datasource if it ingested duplicate rows. Is their anyway we can achieve in druid(version 25)

Hey, I seek help on working with Sketches in Druid. What the proper way of calculating a sum of sketches in my datasource? So far I am doing:

SELECT entity_id, SUM(HLL_SKETCH_ESTIMATE("hll_column")) FROM "test_hll_dataset" GROUP BY 1

But that is quite slow (takes over 20s to calculate); is there a more Druid-way to make this calculation?

Hi everyone, I am wondering if there is an option in Druid to be notified when a new data has been added to the data source? I can't find any information in docs, but maybe you have some knowledge?

Hi druid family, If the incoming events are getting thrown away or unparseable in druid for millions , then will druid close the task with unsuccessful status. If so what is limits of these data which druid will not close the task

👋 Introducing the migration guides for Apache Druid, developed to help you understand the latest and greatest features, and how to migrate to them when there are breaking changes. See the migration guides for the following topics: • Multi-value dimensions to arrays • Front-coded dictionary encoding • Byte-based memory limits on subqueries • SQL compliant null handling Learn more at https://druid.apache.org/docs/latest/release-info/migration-guide

Hi Team, I am working on a tool to streamline and improve documentation contributions for open source projects. I would love to get in touch with someone maintaining docs for Druid and discuss with them. You could read more about the tool here: https://ekline.atlassian.net/wiki/external/MDJmNWI3MjBkMzE4NGQyMzhlYjkxNjUwNDVhYjBlOGQ

Hey Druids. My team has been working on lots of druid related features. We've written a medium blog on Query Level Monitoring via Request Logging. It allows us to set up individual query level monitoring. Do check it out!

🚀 Introducing MIDAS: The Intelligent Autoscaler for Druid’s MiddleManager! 🚀 Hey Druid community! 🌟 Ever felt the pain of wrestling with scaling Druid’s MiddleManager service on Kubernetes? I did too—until I built MIDAS (Middle Manager Intelligent Dynamic Autoscaler)💡 You may have tried using HPA(Horizontal Pod Scaling) to autoscale your MiddleManagers, you might have faced issues like scaling down even when tasks are running, or spinning up more MiddleManagers than necessary. 😅 But with MIDAS, you can autoscale your MiddleManagers without any ingestion failures, ensuring that only the resources required to fulfill ingestion requests are used.🕺🏻 Using MIDAS we have cut our Druid infrastructure costs by 30%🤘🏻. Now, we can scale Druid’s MiddleManagers without worrying about unexpected ingestion failures, and we’re paying only for the resources we actually use🥳 Curious about how MIDAS can turn your chaotic infrastructure into a well-oiled machine? Check out my latest article on Druid Autoscaling: Medium 🔗 The best part? MIDAS is open-sourced! 🎉 Feel free to explore, contribute, or just marvel at how it can revolutionize your Druid deployments. Let’s make Druid even more powerful together! 💪. Repository 🔗 Looking forward to your thoughts and feedback! Feel free to connect with me on LinkedIn 🔗 Cheers, Rahul

Hey Druid folks While not quite docs, per se, I don't see a more appropriate channel to ask, so here goes: A couple of us PMC members on the Apache Superset project started a little podcast talking to folks from different DB teams. We'd love to have a PMC member or three from the Druid project over for a beer and some chat. DM if interested.

Hi, I am playing with the learn-druid repo, great stuff BTW, and I am a bit confused on why the port 8888 from the host keeps redirecting to the jupyter notebook instead of the Druid console? When I look at the compose mappings I don't see anything specific that would cause that, any ideas?

Hey all! Some notebook updates pushed to learn-druid this last week. Thanks @Hugh Evans @Charles Smith for the work on these 🙂 • Tiering • Async query on historical data • Async query on real-time data Thanks also to @Hugh Evans who's been working to spellcheck (!) and generally clean-up the notebooks behind-the-scenes.

Hey all, I'm playing around with single server druid and was wondering if it's possible later down the road to easily migrate to multi-server druid deployments with kafka ingestion (or any streaming ingestion)? I'm assuming the multi-server deployment will just populate the historicals or fetch from the deep storage and the ingestion task will read kafka offsets from the DB?

Hey Druid community! 🌟 I'm looking to implement MM-less in Druid 30. Could anyone share relevant links or documentation to get started? Thanks!

Hey team! Can I ask what is your take on Druid Operator? Is it mature for production use? I am thinking of migrating Druid from EC2 VM instances to Druid Operator as a means for better scalability and cutting down costs, as EC2 instances scaling is not so great. For example, having to scale from r7i.4xlarge to r7i.8xlarge at some point in time represents an almost double in cost while (I hope) with K8s and Druid Operator we can just reserve just some more memory. What is your experience and what do you think about that?

hello everyone ,i am a developer from china

Hi Druids! Hope you're all well. I had some questions for druid_processing_numThreads. We are seeing the average CPU for the historical is around 10-20, reaching a max of 30% during the day. The value for the druid_processing_numThreads currently is set to 20 for our 16core, 128Gb box. The docs recommend keeping it at : numcores - 1. But since we're in such under utilisation, I wanted to increase the number of threads to about 40. Was wanting to know, how others have the config for the same. If you have the following numbers kindly comment below with a one liner for your experience of the same. This would specifically be for the case of running separate servers for Druid, not single server deployments. Kindly add any of the following below : CPU cores druid_processing_numThreads Average CPU Utilisation P99 Query time Any major experience upsides/downsides. Thank you guys for your consideration and time.

Hey Team! I have a doubt on TopN queries. If I do a topN query with filters on a dimension to include only two values A and B. Let's say this dimension is a high cardinality. Does adding filters to only include A and B makes now makes sure that answer will always be 100% accurate. That being said, can I now keep the topN threshold value too as 2 and then too there would be no approximate results. So my question is does TopN first reads top K dimension values and then filter it, or filtering of dimension values from Dictionary happen first ? I hope my question makes sense

Hi All! I was reading the Druid documentation for segment loading and mmap. The docs say that : "The segment cache uses memory mapping*. The cache consumes memory from the underlying operating system so Historicals can hold parts of segment files in memory to increase query performance at the data level*". Followed by "At query time*, if the required part of a segment file is available in the memory mapped cache or "page cache", the Historical re-uses it and reads it directly from memory. If it is not in the memory-mapped cache, the Historical reads that part of the segment from disk. In this case, there is potential for new data to flush other segment data from memory.*" My understanding of this is that each segment is mmap-ed and that a % of each segment is available in RAM (Address space of historical). If the part of the segment required for query is in this page cache, it will get used. If not; the newly accessed data of the same segment will replace the old page cache of the same the segment. Is this a correct understanding? Does this mean that % of each segment cached using mmap is approximately equal for each segment? Does a frequently accessed segment not get an increased mmap-ed percentage? And by increasing the RAM the % of segment cacheable simply increases and hence request speeds are faster?

Resharing here :)

Hello team, Do you still not recommend enabling

dropExisting

for compaction config? Can I hear your experience with this option? I want to save our S3 cost but I hesitate enabling this because I don't want to loose any data.

Hi team, is there any way to set the query timeout in supervisor level. I need to set the query timeout in supervisor leve

Hey team, it seems like there may be some incorrect examples in the Druid documentation for handling SQL queries against Multi Value Dimensions. Doc reference here. Based on the documentation, assuming a dataset like this:

{"timestamp": "2011-01-12T00:00:00.000Z", "label": "row1", "tags": ["t1","t2","t3"]}
{"timestamp": "2011-01-13T00:00:00.000Z", "label": "row2", "tags": ["t3","t4","t5"]}
{"timestamp": "2011-01-14T00:00:00.000Z", "label": "row3", "tags": ["t5","t6","t7"]}
{"timestamp": "2011-01-14T00:00:00.000Z", "label": "row4", "tags": []}

A query of

SELECT label, tags
FROM "mvd_example_rollup"
WHERE tags = 't3'
GROUP BY 1,2

should return an exploded view of all tags within each MVD that got matched (filter applied pre-explosion):

{"label":"row1","tags":"t1"}
{"label":"row1","tags":"t2"}
{"label":"row1","tags":"t3"}
{"label":"row2","tags":"t3"}
{"label":"row2","tags":"t4"}
{"label":"row2","tags":"t5"}

However, in my V30 instance, the behavior I see is instead returning a result that suggests a post-explosion filter. The results I get returned are effectively as follows:

{"label":"row1","tags":"t3"}
{"label":"row2","tags":"t3"}

Are any gurus able to decipher if this is a simple documentation issue, or is my cluster behaving in an unintended way?

Hi, I was trying to use the configuration parameter "druid.server.loadSegmentsOnStartup" to avoid loading all segments from deep storage at every historical startup. I noticed that it is not documented, but I found this method on GitHub: https://github.com/apache/druid/blob/9df92230979342d6179c6f8ba94ee9efbfb4983c/server/src/main/java/org/apache/druid/server/coordination/SegmentBootstrapper.java#L172 Does anyone have experience with this?

Hi Team..im setting up druid emitter prometheus , does anyone has sample json for grafana dashboard and expersion ?

@Sivakumar Karthikesan try this dashboard.

🚀 Thinking of going nanoservices in production? It’s not just about smaller services—state management & orchestration are your real battlegrounds ⚔️ 🔍 Real-world challenges 🧠 Battle-tested solutions 🛠️ YAML workflows & rollback patterns 👇 Dive in: https://medium.com/@akshat111111/b4d6c5925b1d