https://linen.dev logo
Join Slack
Powered by
# general
  • j
    We did rolling restarts yesterday and today. In short; a busy broker took 2.5 hours to restart and a non-busy broker took 4 minutes. Is there any optimization that can be made during broker start sequence to avoid needing to wait hours for startup? We have 3 query nodes. All three are behind a LB but one of the nodes got hard-coded by a busy client. An obvious solution is to have all clients use the LB but there must be something else we can do to improve the startup time of a broker who got pinned by a client.
    r
    • 2
    • 7
  • v
    Hello Druid Experts, Has anyone performed benchmarks specifically for the “Virtual Storage” feature in Druid version 35? I’m interested in understanding the query latency metrics when Virtual Storage is enabled. Also, since this feature is marked as experimental in version 35, is it expected to be production-ready in the next release?
    c
    • 2
    • 6
  • l
    Is anyone using the druid-kubernetes-extensions in production? Its still labelled experimental, although there's an issue to take it out of that status (from 2022 😬). It doesn't look like its under active development based on the history.
    k
    • 2
    • 2
  • a
    Hi all, I am planning an upgrade from Druid 29.0.1 to 30.0.1. Normally we double capacity and replace historical nodes, but currently we can’t provision extra EC2 instances, and decommissioning/replacing historicals one by one would be very time-consuming (each node has ~10–28 TB of local segment cache on instance-store NVMe). Since this is an application-level upgrade, we’re considering an 
    in-place upgrade
     for Historicals: • stop the Historical process • clean up tmp directories and old Druid 29.x binaries • keep the local 
    segmentCache
    directory(data dir) intact • install Druid 30.x and restart the service The expectation is that on startup the Historical will re-announce and reuse existing local segments (no deep-storage re-downloads), so downtime per node is ~10–15 minutes. We have replicas, so no query downtime is expected. Is this approach supported for major version upgrades? Are there any known issues to watch out for (segment compatibility, tmp directories, map state, or other on-disk artifacts) when reusing the same instance and segment cache across 29.x -> 30.x? Any guidance or confirmation would be appreciated. Thanks!
    r
    • 2
    • 2
  • a
    Hi all, We are fine tuning few druid properties and had few doubts when going through docs, could someone please clarify 1. Query processing buffers/threads • 
    druid.processing.buffer.sizeBytes
    • 
    druid.processing.numMergeBuffers
    • 
    druid.processing.numThreads
    I want to understand how are these used when processing the druid queries, couldn't get much clarity from docs. Could anyone please explain these? I see there are formulas in the docs, but wanted to understand the logic behind them. For example, when we increase 
    numThreads
    , more query processing and row scanning happens in parallel, but at some point groupBy/aggregation phases require merge buffers. If merge buffers are insufficient, does query processing block waiting for a merge buffer? What are the key factors to decide appropriate values for these? https://druid.apache.org/docs/latest/configuration/#historical 2. Heap vs direct memory allocation Here, https://druid.apache.org/docs/30.0.1/configuration/#segmentwriteoutmediumfactory it mentions that for MiddleManagers, heap and direct memory should be the same size. Why is that requirement there? Also, does this recommendation apply to Historicals as well? Thanks
  • p
    One detail. it was Druid + Superset +AI project. And chatbot prepared links to Superset dashboards. Druid MCP server is close to it but not the thing that I'm trying to find. Still can't find those videos...
  • p
    Hello Druid Community, Do we know if there is workaround for this issue of Incorrect row signature order ? https://github.com/apache/druid/issues/18437
    k
    r
    a
    • 4
    • 4
  • j
    Do we have any news about this or in general about Druid working on Graviton Instance (Arm64 arc) (very old issue talking about it https://github.com/apache/druid/issues/11820)
  • a
    Hi when checking on consumer properties for idempotency fix for Druid 30.0.1v, I noticed that by default, 
    isolation.level
    is set to 
    read_committed
    https://druid.apache.org/docs/30.0.1/ingestion/kafka-ingestion/#consumer-properties But in latest version, I see default value is set to 
    read_uncommitted
    . Is there any reason why default value is changed. I believe 
    read_commited
    provides correct behaviour when producer uses transactions/idempotency and has no negative impact when it's not https://druid.apache.org/docs/latest/ingestion/kafka-ingestion/#consumer-properties
    g
    • 2
    • 3
  • p
    Hi! I have a question about parallelism in MSQ ingestion. My general understanding is that there are 2 levels of parallelism: number of tasks for given query (maxNumTasks) and number of processing threads in each task (maxThreads). There are some details I am not sure about though.
    When reading external data, EXTERN can read multiple files in parallel across different worker tasks. However, EXTERN does not split individual files across multiple worker tasks
    - docs - We should have number of tasks between 2 (1 controller + 1 worker, minimal parallelism) and number of files we are reading (maximum parallelism). - Question 1: If we have 1 task with 4 threads can it read 4 files at once? - Question 2: If we are reading 10 files and writing 20 segments - can segment writing also take advantage of parallelism? Shouldwe have 10 tasks each with 2 threads then?
    Worker tasks run single-threaded, which also determines the maximum number of processors on the server that can contribute towards multi-stage queries.
    - docs - Question 3: Does this relate to default of maxThreads = 1? Or does it mean something else? - Question 4: I am confused. There 3 different mentions of a default thread count for a worker and they give slight different answers. Here we have mention of "single-threaded", in code there a default of 1 thread, here we have "Not set (use default thread count)" (does it mean 1 or 
    druid.processing.numThreads
    value?)
    Indexing service tasks of type 
    query_worker
    that execute a query. There can be multiple worker tasks per query. Internally, the tasks process items in parallel using their processing pools (up to 
    druid.processing.numThreads
    of execution parallelism within a worker task).
    - docs - We can have more than 1 thread per task if we want to parallelize some stage of a query. This can be set in query context (capped at 
    druid.processing.numThreads
    config value set for Peon that picks up the task). - Question 5: What is the benefit of higher 
    druid.processing.numThreads
    ? What query stages can be parallelized? If I am doing simple ingestion (few files, simple select with no transformations) would I benefit from higher thread count at all?
    👀 1
  • d
    Hi, I am looking into load shedding, and right now am not using query laning on the broker, I noticed an odd behavior in testing where it seems when both 
    druid.server.http.numThreads
    and 
    druid.query.scheduler.numThreads
    , it seems the former enforces when 429s will be returned (enableRequestLimit=true here), but I found in the code this enableRequestQueuing that is not documented that when we set to false manually (by default its true), we get the 
    scheduler.numThreads
    being used as the limit after which 429s are returned. I just wanted to know: 1. Why is this flag not documented, and why is it intended to be removed? I am thinking we would want to enforce limit at the scheduler number because we would always want some capacity of threads avaialble for health checks and to return a 429 to the router - but do let me know if I'm missing something here with how load shedding can be done on the broker 2. On load shedding, would we want the historicals to return a 429, or is that a bad practice? Wondering if that could cause the overall query to fail at the broker or even the broker response to be incomplete
    ☝️ 1
  • p
    Is there a way to ingest specific column as metric in MSQ ingestion without enabling the rollup (so without group by clause)? I cannot come up with MSQ equivalent of:
    Copy code
    {
  "type": "index_parallel",
  "spec": {
    "dataSchema": {
      "dataSource": "test_2r_1d_1m_no_rollup",
      "timestampSpec": {
        "column": "ts",
        "format": "auto"
      },
      "dimensionsSpec": {
        "dimensions": [
          "category"
        ]
      },
      "metricsSpec": [
        {
          "type": "longSum",
          "name": "sum_val",
          "fieldName": "val"
        }
      ],
      "granularitySpec": {
        "queryGranularity": "hour",
        "rollup": false
      }
    },
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "inline",
        "data": "{\"ts\":\"2026-04-15T10:00:00Z\", \"category\":\"A\", \"val\":10}\n{\"ts\":\"2026-04-15T10:00:00Z\", \"category\":\"A\", \"val\":5}"
      },
      "inputFormat": {
        "type": "json"
      }
    }
  }
}
    b
    • 2
    • 6
  • p
    @Gian Merlino What was the original intent of SinkQuerySegmentWalker.CONTEXT_SKIP_INCREMENTAL_SEGMENT query context parameter in query context that seems to skip the incremental segments during query processing. This can be useful to us to reduce query times by avoiding the serial row scan of those segments. I see it is currently undocumented and hence my question.
    g
    • 2
    • 3
  • a
    Hi team, few questions on Kafka supervisor behaviour. If anyone can provide pointers on this it will be helpful. Thanks: 1. With 
    stopTaskCount=2
    , 
    taskCount=4
    and 
    replicas=2
    , does 
    stopTaskCount
    stop 2 task groups (all replicas included), or 2 individual tasks(staggering the two replicas)? 2. With same config as above, will both replicas of a task group start at the same time? During 
    intermediatePersistPeriod
    flushes, will both hit persist roughly simultaneously? If one peon is blocked on merge buffer acquisition, can the 
    broker
    use the other replica's peon to serve the query, or does it wait for both?
    g
    a
    • 3
    • 14
  • a
    Hi, is anybody building an MCP for Druid? What use case are you building it for? Asking so I can get some idea of where to start. Thanks!
    k
    s
    • 3
    • 3
  • j
    Has anyone had any success gleaming useful information out of the request logs? We've begun emitting them into a Druid datasource but now parsing them will be our next challenge...
    k
    • 2
    • 3
  • j
    Related to the above, is there documentation of the schema for the request logs?
    g
    • 2
    • 3
  • s
    guys the druid operator helm chart has been disappeared. I can’t find it. Used to be 
    helm repo add datainfrahq <https://datainfrahq.github.io/druid-operator>
    but now there is nowhere to be found. ANY IDEA ANYONE?
    k
    s
    +2
    • 5
    • 20
  • h
    Hi all — anyone using Druid Projections in production (32+)? Simple use case: a projection on 
    (metric_name, resource_id)
    to act as a presence check so the broker can skip full segment scans when no data exists for that pair. Curious about real-world experience — ingestion overhead, backfill on existing segments, whether the planner reliably picks the projection for this "is there data" shape. Thanks!
    • 1
    • 1
  • c
    Hi, We're excited to host three OSA Community meetups over the next few weeks: San Francisco, Seattle, Los Angeles. If you're local, come hang out with us. Meet fellow engineers, learn something new, and hear how teams are building real-time AI and analytics systems at scale. Food and drinks are on us!
  • l
    Hi all, does anyone have any experience ingesting databricks unity catalog based delta tables? I was the extension details but its not clear if its only delta open source that is supported. And if it does work, how is the performance in comparison to ingesting a days worth of parquet files? Thanks
    a
    • 2
    • 2
  • v
    Hi we are on Druid version 33. Trying to setup SSO with Microsoft Entra ID using Pac4j. Getting this exception. HTTP ERROR 500 org.pac4j.core.exception.TechnicalException: com.nimbusds.jose.proc.BadJOSEException: Signed JWT rejected: Another algorithm expected, or no matching key(s) found URI: /druid-ext/druid-pac4j/callback STATUS: 500 MESSAGE: org.pac4j.core.exception.TechnicalException: com.nimbusds.jose.proc.BadJOSEException: Signed JWT rejected: Another algorithm expected, or no matching key(s) found SERVLET: org.eclipse.jetty.servlet.DefaultServlet-41bdaa81 CAUSED BY: org.pac4j.core.exception.TechnicalException: com.nimbusds.jose.proc.BadJOSEException: Signed JWT rejected: Another algorithm expected, or no matching key(s) found CAUSED BY: com.nimbusds.jose.proc.BadJOSEException: Signed JWT rejected: Another algorithm expected, or no matching key(s) found
    • 1
    • 3
  • s
    Dear Druid-folks, this might be a little bit offtopic but I need some feedback on the druid-mcp-server: I've prepared the 2.0.0 Release github.com/iunera/druid-mcp-server/pull/25 to optimize Druid-mcps-server with small local LLM (running in Ypipe) and I'm asking myself which tools you want in which profile when you are using the druid.mcp-server in certain scenarios. This are the current tools / profile mappings github.com/iunera/druid-mcp-server/blob/…/application-ops.yaml (query + ingestions + retention etc) github.com/iunera/druid-mcp-server/blob/…/application-health.yaml (druid-doctor, health etc) github.com/iunera/druid-mcp-server/blob/…/application-query.yaml (query only, read only) github.com/iunera/druid-mcp-server/blob/…/application-permissions.yaml (modifing basic security extensions users, roles, permissions) here is the mapping to the actual druid rest api github.com/iunera/druid-mcp-server/tree/v2#… If you don't want to spam here, write me a DM or a github issue. Looking forward to get some feedback.
    👍 1
  • p
    Hi! Is there any estimated date of Druid v38 release?
    👀 1
    v
    k
    • 3
    • 4
  • a
    Hi, is there any reason why no docs show up for druid 31.0.2 version, druid.apache.org/docs/31.0.2/design. I do see druid release version here on git, github.com/apache/druid/releases#…. Thanks
  • p
    Hello All, We are hiring for Job Title: Senior Engineer with Druid Location: Irving, TX/ Dallas TX (REMOTE) Job Type: 12+ months Contract Work Arrangement: REMOTE , please email your updated resume to puja.k@abodetechzone.com. for more details please check the attachment, any referral is much appreciated .
  • p
    Hello All, We are hiring for Job Title: Senior Engineer with Druid Location: Irving, TX/ Dallas TX (REMOTE) Job Type: 12+ months Contract Work Arrangement: REMOTE , please email your updated resume to puja.k@abodetechzone.com. for more details please check the attachment, any referral is much appreciated . (edited)
  • v
    Hi, After the initial plan to start druid 38 release branch on july13, there has been no information after that. Is there any estimated date of Druid v38 release?
    🙌 1
    g
    • 2
    • 1
  • a
    Hi Druids! We are hosting Druid Office Hours on Thursday, September 24, 2026, from 830–930 a.m. PT. Please use this link to join: meet.google.com/xvj-hxgs-rkf For more details, please see the mailing list thread: lists.apache.org/thread/… Looking forward to seeing some of y’all there! 😄
  • j
    I'm currently trying to fix an OOM on my Brokers. What I've discovered is that the documentation on ByteBuffers is incorrect. It says:
    Copy code
    Direct Memory: (druid.processing.numMergeBuffers + 1) * druid.processing.buffer.sizeBytes
    But the actual limit is:
    Copy code
    (druid.server.http.numThreads + druid.processing.numMergeBuffers + 1) * druid.processing.buffer.sizeBytes
    Obviously this can get quite large because we want a large number of HTTP threads. However, an HTTP thread only grabs a buffer when it is processing intermediate results, which happens rarely or in bursts. In practice I haven't seen much more than 9 of these at a time (with http threads at 60). The key problem I see here is that there is no ability to cap the buffer pool since the SimplePool never rejects a request. Thoughts on this?
    j
    • 2
    • 2