hello Everyone! I've joined this slack to seek some guidance on Druid Indexer service 🙂 tl;dr - shall we consider Indexer Service to be production ready and to be used on large scale deployments? We considered using Indexer Service because our current mm + peon approach has following problem: We are relying on Druid's MiddleManager (MM) + Peon setup for data ingestion. While this approach has served us well, it comes with a significant limitation— inefficient resource allocation during ingestion tasks. Specifically: - Each ingestion task is assigned its own JVM (Java Virtual Machine), with a single configuration setting controlling the JVM size. - We must set the JVM size to accommodate the largest ingestion tasks. This results in an over-allocation of resources for smaller tasks that don't require nearly as much memory. - Consequently, this leads to under-utilization of our available resources, reducing the total number of ingestion tasks we can run simultaneously. I saw a post by @Benjamin Hopp here that triggered an idea we might not want to go with Indexer Service at all. thanks in advance for any suggestions! 🙏

Hi Michael, Afaik Ben is correct ... the focus on ingestion process optimization has turned towards K8s jobs and middle-manager-less ingestion, and away from the indexer service. That being said, you can tune your indexing tasks a bit per Supervisor if you want, e.g.:

"context": {
    "druid.indexer.fork.property.druid.processing.numThreads": "8",
    "druid.indexer.fork.property.druid.processing.numMergeBuffers": "8",
    "druid.indexer.fork.property.druid.server.http.numThreads": "80",
    "druid.indexer.runner.javaOptsArray": [
      "-Xmx8g",
      "-Xms8g"
    ]
  },

> - Each ingestion task is assigned its own JVM (Java Virtual Machine), with a single configuration setting controlling the JVM size. > - We must set the JVM size to accommodate the largest ingestion tasks. This results in an over-allocation of resources for smaller tasks that don't require nearly as much memory. > - Consequently, this leads to under-utilization of our available resources, reducing the total number of ingestion tasks we can run simultaneously. The process that John mentioned is the best way to handle this. By configuring the resources at the supervisor, you can allocate proper resources per JVM, allowing you to more efficiently use the machines.

thank You @John Kowtko and @Benjamin Hopp for clarifications! And supervisor seems to be perfect fit for streaming ingestion (like from kafka) from what I've learned. Do we have similar flexibility in terms of resource utilization for ingesting from other sources (like HDFS etc.)? btw. Indexer Service is still marked as "experimental" (for a long time) - do You think eventually it will be discontinued complately/removed from Redis?

Yes, the same properties can be used in batch indexing. I suspect that the indexer will be removed at some point, however there are a couple very large users of it right now, so it probably will be a while before everyone migrates to middle-manager ingestion.

thank You - that clarifies and helps a lot!

Just to chip in, we are using indexer in production for a very long time and it's working great for us. The biggest advantage is sharing the query compute power for kafka intake. Using

equalDistributionWithCategorySpec

to route all

index_kafka

to indexers and the rest to middle managers.