java.lang.RuntimeException: java.io.FileNotFoundException: var/druid/processing/druid-groupBy-bf95eed8-402f-4543-97c0-e1c6f641ce0a_5385733b-af07-4a0a-aa39-eecc1c4518ae/00015496.tmp (Too many open files)

Increased u limit

This message was deleted.

what does ulimit -a show?

also you need to make sure ulimit is increased for the user running druid services

This message was deleted.

This message was deleted.

Hello everyone, I am new to druid. I have been upgrading druid on our system from version 24.0.1 to 25.0.0 and want to use http instead of zookeeper. Currently I am facing the issue that overlord won't work properly. My system: • Kubernetes cluster in the cloud • No zookeeper • Kubernetes extension All the druid components started up and are running through except the component overlord. Here are the variables that I have set: ### K8S without Zookeeper # druid.zk.service.host=localhost # druid.zk.paths.base=/druid druid.zk.service.enabled=false druid.serverview.type=http druid.coordinator.loadqueuepeon.type=http druid.coordinator.loadqueuepeon.http.batchSize=1 druid.indexer.runner.type=httpRemote druid.announcer.type=http druid.discovery.type=k8s druid.discovery.k8s.clusterIdentifier=druid druid.discovery.k8s.podNameEnvKey=POD_NAME druid.discovery.k8s.podNamespaceEnvKey=POD_NAMESPACE And in the attachment you will find the log that I see. Also in the log line 86 the parameter "druid.indexer.runner.type" is still "remote" rather than "httpRemote". Is this the cause or have I missed something? How to fix this issue?

you need to add druid-kubernetes-extensions to extension load list

This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.

What can be the reason

This message was deleted.

This message was deleted.

This may be useful .. some pre-processing may be required - https://blog.hellmar-becker.de/2023/01/08/druid-data-cookbook-modeling-flag-lists/

Facing this error while using theta sketch but all other aggregates - sum are working fine ..org.apache.druid.query.QueryException: Not enough merge buffer memory to execute this query. Try enabling disk spilling by setting druid.query.groupBy.maxOnDiskStorage to an amount of bytes available on your machine for on-disk scratch files. Or, if you have additional off-heap memory available

I have tried adjusting the following values in broker/runtime.properties & historical/runtime.properties

druid.query.groupBy.maxOnDiskStorage=20000000000 druid.query.groupBy.maxMergingDictionarySize=20000000000 druid.processing.buffer.sizeBytes=2000MiB druid.processing.numMergeBuffers=4

This message was deleted.

Is it possible to use helm chart to deploy a druid cluster with druid-kubernetes-extensions enabled (zk disabled) ? I tried with below settings in values.yaml, but pods are not coming to ready status.

druid_zk.service_enabled: false 
  druid_serverview_type: http 
  druid_coordinator_loadqueuepeon_type: http 
  druid_indexer_runner_type: httpRemote 
  druid_discovery_type: k8s
  druid_discovery_k8s_clusterIdentifier : druid
  druid_discovery_k8s_podNameEnvKey : POD_NAME
  druid_discovery_k8s_podNamespaceEnvKey : POD_NAMESPACE

Hi Druid Experts I have the following query regarding task logs Due to some exceptions , if task fails, task logs doesn't get uploaded to s3, and it remains in the mm node Can someone please help me understand, when these task logs are cleaned up? from the mm node Or how can we get rid of it

Can I use hers to store metadata and other data in Druid

Hers? I don’t understand

I mean hdfs

Sorry for autocorrect

I am not seeing any detailed documentation on same