Apache Druid

You can check the ~kill~ compaction task logs in the ingestion tab to see whats happening or what <@U03R5H3B750> said ^-^

`dropExisting`  is still an beta feature,  if you want to delete unused segments, you should use the kill task option :

<https://druid.apache.org/docs/latest/operations/clean-metadata-store.html#segment-records-and-segments-in-deep-storage-kill-task>

Also to replace segment (which i believe is your expectation) There is functionality which could restrict the replace, Please find below example from guide which explains the scenario where the original segment will not be replaced:

```Consider an existing segment with an interval of 2020-01-01 to 2021-01-01 and YEAR segmentGranularity. You want to overwrite the whole interval of 2020-01-01 to 2021-01-01 with new data using the finer segmentGranularity of MONTH. If the replacement data does not have a record within every months from 2020-01-01 to 2021-01-01 Druid cannot drop the original YEAR segment even if it does include all the replacement data. Set dropExisting to true in this case to replace the original segment at YEAR segmentGranularity since you no longer need it.```

<@U03R5H3B750> if this is still a beta feature then it won't work and I shall choose kill task request rather then this. Thanks !

Yes, We could say that, Beta features are something which is not tested up to the extent, so we could test it in our test environment but should not use in dev/prod, as it could provide some surprise.

Ok so that means it is released but not tested properly if I am not understanding wrong?

Yes, released as a Beta Feature with the warning:
```WARNING: dropExisting in ioConfig is a beta feature.```

<@U04NGQF4DV2>
I think you meant <@U03R5H3B750> :slightly_smiling_face:

If you have Imply Cluster, your answers could be best answered over a support case.

Generally we look for below parameters to login to mysql:
```druid.metadata.storage.connector.connectURI
druid.metadata.storage.connector.user
druid.metadata.storage.connector.password```

not sure how Prometheus scaping works but maybe the emissionPeriod is not getting set correctly? When the request completes, are you seeing any metrics though?

```
[2023-04-12T10:23:37.159Z] INFO main CliBroker:105 * druid.metadata.storage.type: postgresql
[2023-04-12T10:23:37.159Z] INFO main CliBroker:105 * druid.monitoring.emissionPeriod: PT10S
[2023-04-12T10:23:37.159Z] INFO main CliBroker:105 * druid.monitoring.monitors: ```
we are seeing this in broker log that emissionPeriod is set properly

the request is served by prometheus server so its hard to say what is going on. what you can do (if its happening consistently), is to take thread dumps 10 seconds apart after you issue the wget. Those thread dumps may offer more clues as to whats blocking the request.

Hi Team,

i have a strange issue:
below both aueries are same in second query i just added the count() to Query1. but Query1 returns results but Quert 2 doesnot return any results.
  Query1 :
  SELECT num from (
  SELECT num , MAX(__time ) AS lastruntime, Current_timestamp - INTERVAL '4' HOUR as currentTime
  FROM "XXX" WHERE eventName = 'HEARTBEAT' AND num !=''
  GROUP BY num ) WHERE TIMESTAMPDIFF(MINUTE, lastruntime, currentTime) &gt; 1

   Quey2:
  SELECT count(num) as mynumber from (
  SELECT num , MAX(__time ) AS lastruntime, Current_timestamp - INTERVAL '4' HOUR as currentTime
  FROM "XXX" WHERE eventName = 'HEARTBEAT' AND num !=''
  GROUP BY num ) WHERE TIMESTAMPDIFF(MINUTE, lastruntime, currentTime) &gt; 1

  Can someone pls help here

Yes. You can change the retention period. If you had not yet run a kill task, segments older than 3 months that were marked as unused are still recoverable. Take a look at this blog post: <https://imply.io/blog/apache-druid-recovering-dropped-segments/>

Is the metadata database responsive when this happens?

How many segments are there in the datasource?

Yes metadata base is accessible. we have 3770 segments

It is a timeout exception, is the query different somehow when this happens?

Scan queries works but  TOPN queries are failing and will be in this state until restarting the broker

Hmmm, sounds like a leak of some kind. Can you get a JVM dump?

Unfortunately our pods doesn't have lib for JVM dump.  I will update here

even installing latest version at /usr/local/opt/curl/bin/curl, it still have the same issue.

never mind, I found that I should use the url (rest endpoint) as `druid/coordinator/v1/datasources/events_transactions/markUnused`

That said, from the time I issued the command till the time I saw the log in coordinator with the line `Dropping uneeded segment ...`, it took more than half an hour. The coordinator period we set is only 2 minutes. Any idea why it takes so long? <@U0411DE2SS0>

no, I don't know the details of how it schedules dropping segments.

yeah in your original question you were sending you requests to the web console html route and getting back the web console html

this is a query timeout...likely because you don't enough resources in the historical or the broker. Can you paste your query here?

It’s failing for all query .taking a lot of time

After restarting query server it started working 

This might be a recent regression introduced in the merge pool. In the meantime, can you disable the parallel merge by setting the following property in the broker’s runtime to see if the problem persists. (This is a workaround however)
`druid.query.default.context.enableParallelMerge=false`

out of curiosity what java version are you using?

i have a hypothesis that this getting stuck behavior seems related to java versions; in my tests java 8 and java 20 were ok, but 11 and 17 might have some problems

and am proposing disabling parallel merge for these java versions by default until we can figure out exactly what is going on <https://github.com/apache/druid/pull/14130>

<@U030C8H59T8> what does the replication look like for this?

i'm on java 17 , going to run

```while true ; do mvn test -pl processing -Dtest=ParallelMergeCombiningSequenceTest ; done```
nothing in my short loop so far - but i'll leave it running for a while

(ps: are you on ARM or x86?)

I think there’s a real issue here but only shows up if the initialise gets blocked , and it doesn’t stop the query from timing out 

thanks for looking, i haven’t had a chance to get back to this yet

Getting this error while querying data from Druid

Hi, The data is ingested from the file `wikiticker-2015-09-12-sampled.json.gz` with the segment granularity DAY, so you ended up with 1 segment.
If you have any other files that can be ingested into the same data source, you could do that.
Also, if you change the `segmentGranularity` to Hour, then you will see more than 1 segment as well

<@U030LNJJ13M> so as you suggested I changed segment granularity to hour still could see only 1 seg when loaded the data

```{
  "type": "index_parallel",
  "spec": {
    "dataSchema": {
      "dataSource": "wikipedia7",
      "dimensionsSpec": {
        "dimensions": [
          "channel",
          "cityName",
          "comment",
          "countryIsoCode",
          "countryName",
          "isAnonymous",
          "isMinor",
          "isNew",
          "isRobot",
          "isUnpatrolled",
          "metroCode",
          "namespace",
          "page",
          "regionIsoCode",
          "regionName",
          "user",
          {
            "name": "added",
            "type": "long"
          },
          {
            "name": "deleted",
            "type": "long"
          },
          {
            "name": "delta",
            "type": "long"
          }
        ]
      },
      "timestampSpec": {
        "column": "time",
        "format": "iso"
      },
      "metricsSpec": [],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "hour",
        "queryGranularity": "none",
        "intervals": [
          "2015-09-12/2015-09-13"
        ],
        "rollup": false
      }
    },
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "local",
        "baseDir": "quickstart/tutorial/",
        "filter": "wikiticker-2015-09-12-sampled.json.gz"
      },
      "inputFormat": {
        "type": "json"
      },
      "appendToExisting": false
    },
    "tuningConfig": {
      "type": "index_parallel",
      "partitionsSpec": {
        "type": "dynamic"
      },
      "maxRowsInMemory": 25000
    }
  }
}```

can you share the task log for the new run with HOUR granularity here <@U04NGQF4DV2>

image.png

below is payload
```{
  "type": "index_parallel",
  "id": "index_parallel_wikipedia7_phpajhdc_2023-04-11T06:14:24.020Z",
  "groupId": "index_parallel_wikipedia7_phpajhdc_2023-04-11T06:14:24.020Z",
  "resource": {
    "availabilityGroup": "index_parallel_wikipedia7_phpajhdc_2023-04-11T06:14:24.020Z",
    "requiredCapacity": 1
  },
  "spec": {
    "dataSchema": {
      "dataSource": "wikipedia7",
      "timestampSpec": {
        "column": "time",
        "format": "iso",
        "missingValue": null
      },
      "dimensionsSpec": {
        "dimensions": [
          {
            "type": "string",
            "name": "channel",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "cityName",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "comment",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "countryIsoCode",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "countryName",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "isAnonymous",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "isMinor",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "isNew",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "isRobot",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "isUnpatrolled",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "metroCode",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "namespace",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "page",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "regionIsoCode",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "regionName",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "user",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "long",
            "name": "added",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": false
          },
          {
            "type": "long",
            "name": "deleted",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": false
          },
          {
            "type": "long",
            "name": "delta",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": false
          }
        ],
        "dimensionExclusions": [
          "__time",
          "time"
        ]
      },
      "metricsSpec": [],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "HOUR",
        "queryGranularity": {
          "type": "none"
        },
        "rollup": false,
        "intervals": [
          "2015-09-12T00:00:00.000Z/2015-09-13T00:00:00.000Z"
        ]
      },
      "transformSpec": {
        "filter": null,
        "transforms": []
      }
    },
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "local",
        "baseDir": "/opt/druid/quickstart/tutorial",
        "filter": "wikiticker-2015-09-12-sampled.json.gz",
        "files": []
      },
      "inputFormat": {
        "type": "json",
        "flattenSpec": {
          "useFieldDiscovery": true,
          "fields": []
        },
        "featureSpec": {}
      },
      "appendToExisting": false,
      "dropExisting": false
    },
    "tuningConfig": {
      "type": "index_parallel",
      "maxRowsPerSegment": 5000000,
      "appendableIndexSpec": {
        "type": "onheap"
      },
      "maxRowsInMemory": 25000,
      "maxBytesInMemory": 0,
      "skipBytesInMemoryOverheadCheck": false,
      "maxTotalRows": null,
      "numShards": null,
      "splitHintSpec": null,
      "partitionsSpec": {
        "type": "dynamic",
        "maxRowsPerSegment": 5000000,
        "maxTotalRows": null
      },
      "indexSpec": {
        "bitmap": {
          "type": "roaring",
          "compressRunOnSerialization": true
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs",
        "segmentLoader": null
      },
      "indexSpecForIntermediatePersists": {
        "bitmap": {
          "type": "roaring",
          "compressRunOnSerialization": true
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs",
        "segmentLoader": null
      },
      "maxPendingPersists": 0,
      "forceGuaranteedRollup": false,
      "reportParseExceptions": false,
      "pushTimeout": 0,
      "segmentWriteOutMediumFactory": null,
      "maxNumConcurrentSubTasks": 1,
      "maxRetry": 3,
      "taskStatusCheckPeriodMs": 1000,
      "chatHandlerTimeout": "PT10S",
      "chatHandlerNumRetries": 5,
      "maxNumSegmentsToMerge": 100,
      "totalNumMergeTasks": 10,
      "logParseExceptions": false,
      "maxParseExceptions": 2147483647,
      "maxSavedParseExceptions": 0,
      "maxColumnsToMerge": -1,
      "awaitSegmentAvailabilityTimeoutMillis": 0,
      "partitionDimensions": []
    }
  },
  "context": {
    "forceTimeChunkLock": true,
    "useLineageBasedSegmentAllocation": true
  },
  "dataSource": "wikipedia7"
}```

Seems like you have an issue with reading the service logs

I ran the ingestion for the same file. here is the payload

```{
  "type": "index_parallel",
  "spec": {
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "local",
        "baseDir": "/Users/hemanthpatil/Downloads/imply/imply-2023.01.1-LTS/dist/druid/quickstart/tutorial",
        "filter": "wikiticker-2015-09-12-sampled.json"
      },
      "inputFormat": {
        "type": "json"
      }
    },
    "tuningConfig": {
      "type": "index_parallel",
      "partitionsSpec": {
        "type": "dynamic",
        "maxRowsPerSegment": 5000000
      }
    },
    "dataSchema": {
      "dataSource": "wikiticker-2015-09-12-sampled",
      "timestampSpec": {
        "column": "time",
        "format": "iso"
      },
      "dimensionsSpec": {
        "dimensions": [
          "channel",
          "cityName",
          "comment",
          "countryIsoCode",
          "countryName",
          "isAnonymous",
          "isMinor",
          "isNew",
          "isRobot",
          "isUnpatrolled",
          "metroCode",
          "namespace",
          "page",
          "regionIsoCode",
          "regionName",
          "user",
          {
            "type": "long",
            "name": "delta"
          },
          {
            "type": "long",
            "name": "added"
          },
          {
            "type": "long",
            "name": "deleted"
          }
        ]
      },
      "granularitySpec": {
        "queryGranularity": "none",
        "rollup": false,
        "segmentGranularity": "hour"
      }
    }
  }
}```

In the logs, we can see how many segments got created

`2023-04-11T06:27:18,664 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Published [24] segments`

If you see the data in the file `wikiticker-2015-09-12-sampled.json` , the data starts from `2015-09-12T00:46:58.771Z`

Ohh
the stpes that I followed in druid to load data were
click load data -&gt; click other -&gt; click submit task -&gt; past the json data and load
is that fine?

I am running the same spec you pasted but only the path of the file is changed

It ran fine with 24 segments again
```2023-04-11T06:34:07,631 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Published [24] segments```

Can you retry again with the new Datasource or can you access the indexing logs location and check the task logs

it's working now. I just deleted few old data source which were causing some trouble it's fine now

also what is correct approache if I want to delete a data source completely from druid UI itself?

1. Mark them unused for all the segments from the console for that data source.
2. Post that, we can select `Show unused`  from the data source page on the top and run the kill task which will remove the segments and MetaData related data permanently

<https://druid.apache.org/docs/latest/tutorials/tutorial-delete-data.html#how-to-permanently-delete-data>

Ok <@U030LNJJ13M> Thank you so much for all the help.
How did you approach learning druid if you ok to share?

There are some cool trainings available which we can go through to learn druid and there is always the druid documentation.

<https://learn.imply.io/page/learn-apache-druid>

Fyi when ingesting in append mode you are not allowed to mix segment granularity, i.e. you cannot ingestion/append using HOUR granularity if the existing time chunk that you are ingesting into has existing segments with DAY granularity.

Now you are ingesting with append=FALSE, that is overwrite mode, which means you are intending to overwrite the existing segment(s) with the data you are ingesting.

This still may be problematic because the existing segments has DAY time boundaries, yet  when you first start ingesting the data with HOUR granularity it will want to lock only one HOUR time chunk at a time ... this may not mix properly.

So, as you found out, you had to completely remove that DAY of data from the datasource in order to ingest with HOUR granularity into that day.

At least that is my educated guess as to what you were experiencing here, Dinesh.

There is also a dropExisting=TRUE setting, and I believe and interval you can specify to go with that, that may be an option here to remove (i.e. hide using a tombstone segment) the entire day when you start ingested at the HOUR level.

Someone please correct me if I didn't get this quite right :slightly_smiling_face:

Thanks.  John

Hey <@U04C4593VDY> I have understood clearly what you said and it make completely sense to me. Also I removed completely old data source and then ingested with hour granularity, so there wasn't any mix.
Thanks !
Dinesh

Screenshot 2023-04-11 at 6.09.35 PM.png

The error seems harmless. What likely happened here is that k8s pod watcher thread got interrupted as the task was shutting itself down. We could probably handle the interrupedException better. can you file a github issue for this?

I don’t know the answer to your question, but what errors were you seeing on the broker? was there a specific one?

The rolling update order is supposed to be historical first, brokers, then coordinator last.

<https://druid.apache.org/docs/latest/operations/rolling-updates.html>

Not that that was what happened here. I've also had errors where unready historicals are polled by a coordinator and their dependencies on each other prevent either from coming up, so I had to kill the historicals to get the coordinator to start.

<@U03B1M35MHR> I was in a hurry to recover the cluster that I didn’t save the logs, but I will do that next time.

<@U04P4A4J3JP> Ah, this is good to know. I had forgotten the order of restarts.

Furthermore, it was the worst type of exception. It failed the query, logs the error, but did not crash the daemons.

Had it crashed the daemons, Kubernetes would have restarted them and the issue would have gone away on its own.

While pickup my son home, I had further questions.

If… historical has a non-zero chance of bringing down broker during rolling restart, then it doesn’t matter of the order of operation, no?

And this behavior is especially bad for large clusters. Imagine a cluster having 500-2000 historical daemons being rolling restarted. The downtime will be very long.

Coming to this a little late, but, a couple comments—

1) the update order from the docs is very important! Typically the Broker / Historical protocols work by the Historical speaking both old and new, and Broker only speaking new. So in a rolling update, Historicals have to be updated first, or else Broker tries to speak new protocol to old Historicals, which has unpredictable results. It can crash, cause query errors, cause incorrect results (due to, for example, old Historicals not understanding a new and important parameter), etc

2) when you say "historical has a non-zero chance of bringing down broker during rolling restart" what kind of issue did you see? this generally should not happen, so if it does happen I'd like to understand it so we can fix it / prevent it from happening again

Hi Gian!
I think your 1) answered our issues. Broker must have been speaking new protocol to old Historical. Causing every single query to error out.

I do see some logs from coordinator which says this.

```Task [index_kafka_events-bf6b07c8-68b5-422d-9230-c588905cab92_462dbbce4c0a00f_ocjdaiic] paused successfully```


but i am not able to understand why and when coordinator will ask task to be paused.

Just a thought, Was the coordinator stable at that time, Was their any message related to Leader election/frequent change in leader which could cause this. Should we just restart the Master nodes and  check if this could make a stable Leader and see how this comes out.

how does  coordinator instability causes tasks to be paused ? can you explain a bit ? I will check whether coordinator was stable or not.

OK, Sorry, i thought you saw the above messages in coordinator log, Generally If the leader election has some issue electing the leader the tasks/supervisors went to waiting state, and this is overlord leader election.

supervisor run should be task of overlord, so could you check overlord log and overlord stability

or  we may check check any connectivity issue with kafka , what error you  see in task logs .

Hi, This compaction task will take segments from the data source which is mentioned in the spec and will try to compact them based on the HOUR granularity

If the data source has multiple segments for the interval you have provided, then it will consider those segments and compact them

okay <@U030LNJJ13M>
so let's say my granularity is HOUR
and the interval I have provided is 2015-09-12/2015-09/13 which is basically one day interval.
so if I am not understanding  wrong whicever segment is coming in this interval it will run compaction on hourly basis on those segments is that correct?

<@U04NGQF4DV2> If your data ingested is of HOUR granularity, then the compaction task will consider one hour of interval and tries to compact the segments within that interval.
For example, `2015-09-12T00:00:00.000Z` and `2015-09-12T01:00:00.000Z`  will be one interval and if you have multiple segments within this the task will compact them. Same for the rest of the intervals within the DAY interval you have provided in the spec

<@U030LNJJ13M> ok got it.  if we specify the interval in compaction then what it will do ? the part you mentioned is about ingestion, so for this particular time interval we have if we ingest the data it will partition them based on the granularity and create segments. but if we mention same interval while making a compaction request what exactly this interval do. do we even need this interval now as the ingestion has happened based on the granularity and interval ?

Yes, the interval is needed in the compaction spec for it know what all it would need to cover.
Compaction tasks read an existing set of segments for a given time interval and combine the data into a new "compacted" set of segments

<https://druid.apache.org/docs/latest/data-management/compaction.html>

Okay <@U030LNJJ13M> got it.
Also dropExisting = true while running compaction to drop already existing segment copies. it's not working any comment on this?

we might had discussion on this previosly I am not sure

Can you check the coordinator logs once the compaction is done for these messages
```org.apache.druid.server.coordinator.duty.UnloadUnusedSegments - Dropping uneeded segment```

Did the compaction tasks Published any segments?

so what it did basically created  a new compacted copy and kept it in the same segment

this are the segments i have on hourly basis

each time it is creating a new compacted copy

the below SS is something present in first segment having time 00:00:00 on above pic

Are you checking the segment-cache or deep storage segments?

yes, so deep storage will have them unless we run the kill tasks

ok that i have done already.
my task it do achieve the same while runnning the compaction itself that's why I used dropExisting = true
isn't it possible?

I dont think that is possible on the compaction task to remove the segments from Deep storage

I believe the replace referred here is for the segments-cache which is the historical nodes

```You can set dropExisting flag in the ioConfig to true if you want the ingestion task to replace all existing segments that start and end within the intervals for your granularitySpec. This applies whether or not the new data covers all existing segments. dropExisting only applies when appendToExisting is false and the granularitySpec contains an interval. WARNING: this functionality is still in beta.```

Well I also believe that for ingestion it’s gonna work but for compaction i also have doubts 

<@U04NGQF4DV2> If you set `dropExisting` in the compaction or ingestion spec, it wont remove the segments from the deep storage. I did a test on this

Okay <@U030LNJJ13M> 
How can we validate that it was excuted but didn’t work as expected?

I noticed that after a heavy query completes there's a period of time where query times are slower and identical (shows as a flat horizontal line on query time graphs), almost as if the cluster is recovering. Any idea why? I was thinking maybe it's garbage collection because the heap size is too big, or maybe just time for different segments to get loaded again into memory.

I think sometimes it's either a communication problem - MM didn't get the notice, or, if I'm not mistaken, if MM decides to end the task for some reason, it tells it to shut down, task stops and says "success - I did what I was told".

I implemented something like this for our cluster using the load status endpoint on the coordinator. the first endpoint in this section returns a json map of the data source name to the percentage available <https://druid.apache.org/docs/latest/operations/api-reference.html#segment-loading>

:point_up_2: ... as far as I know

`GET /druid/coordinator/v1/loadstatus`

on the above doc page referenced by Ryan is the standard method of determing if all segments are queryable

what does the context path based routing mean?

for example we can access druid ui on <http://host>:port   but i need to route incoming requests to  <http://host>:port/druid

Druid can't do such routing. Though you have something like nginx in front of druid to do this kind of routing

That can be done with Nginx ingress or any other http proxy technologies.

seems like that the task wasn't able to connect to coordinator

Thanks, we had some restarts of our zookeeper... as Zk store some informations on the Cluster status maybe it is the root cause

What are the procedure to do after a restart/crash of ZK ? Need to restart Druid as well or not ?

druid will stabilize once the zk is stable as well. If you continue to see issues, you could restart just the master nodes.

I am trying to ingest 800 gb parquet data to Druid

Hi <@U04GEHS4TRU>, is your 800 GB spanning multiple days or months?  It might be better to ingest a smaller number of time slices at a time.  Additionally, please use the sql-based ingestion capability (<https://druid.apache.org/docs/latest/multi-stage-query/index.html>) and, if possible, the most recent version of Druid.

Agree with FJ, however we ingest terabytes of data per day into druid using the druid MM processes (Peons), we normally break the file into smaller chunks in GCS (Googles Cloud Storage) to parallelize the data load across our MM's.  We further modify the maxRowsInMemory/maxBytesInMemory configuration to a smaller number if we are running into OOM issues (This really depends on the number of columns, row size etc.... usually 1/6th the jvm heap is a reasonable starting point ).   Id recommend if possible to further partition the 800GB file into smaller chunks if that is possible and depending on the number of peons in your cluster.

I don't think you need to restart.  Did you wait for current compactions to complete and new ones to start?  Also, I'm not sure how maxNumConcurrentSubtasks comes in to play.  I tried figuring it out from <https://support.imply.io/hc/en-us/articles/360046597493-How-to-determine-the-number-of-slots-required-for-auto-compaction-|this KB>, but I'm still confused atm.

request logs may help, they include response time and timeout info <https://druid.apache.org/docs/latest/operations/request-logging.html>

I m using debezium as a producer in kafka and connecting the broker to druid. My entire system freeze when i start the ingestion anyone done this before?

we have seen that when leadership changes too frequently, there are lot many writes to the zookeeper. So its possible that is happening in your cluster. You may want to troubleshoot your master nodes and check they are not causing this.

Ok thanks do you know how this leadership is done ? Who choose the leader ?

overlord/coordinator processes use zk to decide one leader

Have you turned on the automatic cleaning job on ZK? By default it is not turned on for some unknown reason. The smallest frequency is 1 hour.

Hello <@U03149ALHN3> do you mean the purge ? Yes with
```ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL=1
ZOOKEEPER_AUTOPURGE_SNAP_RETAIN_COUNT=3```
After having again increase the disk size (to 50Gb) it is ok. My guess is that as we had a lot of instabilities on the cluster (historical restarted very often), that generate a lot of log on Zookeeper (at each restart it seems that each Historical stores all segments location on ZK)... so we reached very easily the limit

you could also move some stuff from ZK to HTTP if you haven't already, such as segment announcing, task assignment, etc

<@U031A9D19NU> I m runnning druid Kafka debezium on docker compose earlier druid was working fine but suddenly in normal ingestion also the console is getting freezed i can’t check the logs too

this seems like a bug to me, thanks for reporting it and the clear repro query!

<@U053NJ4GL1F> - thanks again for this clear jira.  Devs are taking a look and hope to get a fix for this regression into 26.0.  We appreciate you finding it and reporting it with details!

Thanks <@U0411DE2SS0>, let me know if I could be anyhow of an assistance!

once 26.0.0-rc1 is out (you can check the <mailto:dev@druid.apache.org|dev@druid.apache.org> mailing list for info) it is always helpful if you download it and verify it works for you!

it should be available soon, typically is up for 72 hours prior to release for people to do various validations

There’s an online course that takes you through a cluster configuration at <https://learn.imply.io|https://learn.imply.io> . I think it is part of the Druid Basic course. 

Thank you for the response will check it out and get back on the same 

<@U030K4UM3H7> I have gone through the cluster setup but I feel the instruction is not that details on multiple configuration, if there is a detailed documentation 

Single node configuration is very detailed 

If there is any further docs kindly share 

What configuration are you trying to do? This is a good write up <https://medium.com/@pushkarjoshi0410/druid-cluster-setup-6123bb921376|https://medium.com/@pushkarjoshi0410/druid-cluster-setup-6123bb921376>

With Druid 25.0 druid you should take a look at the `bin/start-druid --help` utility which allows you to just select the total memory you want to use and which services to start. So you can use it to deploy different services on different nodes and have the script figure out how to split the memory among those services. You'll still need to configure deep storage and metadata database.

But is this a standalone setup when I run start-Druid ? 

What should I do if I want to separate the master , data and query server 

You can run it with -s and list the services to start. On the master servers you can start coordinator and overlord on the data node you can start middle manager and historical and on the query node broker and router. Take a look at the output from the —help option it has quite a few examples.

Hi <@U030K4UM3H7> good day! I have used start-druid to start the processes in 3 different servers but while starting historical I am getting below error 


```druid.segmentCache.locations needs to be set in historical ```

```ERROR [main] org.apache.druid.cli.CliHistorical-Error when starting up.
com.google. inject .ProvisionExcept ion: Unable to provision, see the following errors:
Failing
1) druid.segmentCache. locations must be set on historicals.
at org.apache.druid.guice.StorageNodeModule.getDataNodeServiceStorageNodeModule.java:95)(viamodules:c at org.apache.druid.guice.StorageNodeModule.getDataNodeServiceStorageNodeModule.java:95)(viamodules:c while locating org.apache.druid.discovery.DataNodeService
at org.apache. druid.cli. ServerRunnable.b indAnnouncer (ServerRunnable. java: 80) (via modules: com.google. inj at org.apache.druid.cli.ServerRunnable.bindAnnouncer(ServerRunnable.java:80)(viamodules:com.google.inj while locating org.apache.druid.cli.ServerRunnable$DiscoverySideEffectsProviderschild```

Locations point to a list of paths where you will be storing the local cache for segments on the historicals, each historical must have some locally mounted storage for the segment cache, that path is what you want to point to.

<@U052HT3AK5F> were you able to resolve this?

<@U030K4UM3H7> no I was not able to resolve it, the historical properties file does have the location with size in mb but historical logs still throws the error 


```druid.segmentCache.locations=[{"path":"/druid/var/druid/segment-cache","maxSize":1000000}]```

Thank you <@U030K4UM3H7> <@U04RWBRG0VA> for your support, I was able to create the cluster but not using the cluster folder properties but using auto folder and starting processes on different drivers

Where did it put the segment cache folder? Were you able to configure it to your mount point?

I'll do a quick test for a historical and report back...

I tested changing the conf/druid/auto/historical/runtime.properties with:
```# Segment storage
druid.segmentCache.locations=[{"path":"/tmp/var/druid/segment-cache","maxSize":"300g"}]```
and started the historical service with:
```bin/start-druid -m 4g -s historical
[Tue Apr 18 08:27:33 2023] Starting Apache Druid.
[Tue Apr 18 08:27:33 2023] Open <http://localhost:8888/> or <http://Sergio-Ferragut.home:8888/> in your browser to access the web console.
[Tue Apr 18 08:27:33 2023] Or, if you have enabled TLS, use https on port 9088.
[Tue Apr 18 08:27:33 2023] Starting services with log directory [/Users/sergioferragut/druid/apache-druid-26.0.0-SNAPSHOT/log].
[Tue Apr 18 08:27:33 2023] Running command[historical]: bin/run-druid historical /Users/sergioferragut/druid/apache-druid-26.0.0-SNAPSHOT/conf/druid/auto '-Xms1310m -Xmx1310m -XX:MaxDirectMemorySize=1965m'```
and it started fine and created the segment cache path under /tmp. What was the last error you were getting? Could it be permissions on the mount?

How the start command is picking up the cluster properties file 

I thought by default it picks up auto direct Tory 

correct, that's why I update the runtime properties in the auto directory.

But my issue happened in cluster directory 

And during start-druid command I used -c and provided config path 

I tried with the cluster directory and got a different error. It could not find the _common folder. Can you share your startup command, so I can reproduce? Also, I'm not sure if the cluster config folders are meant to be used with the start-druid. I'm trying to figure that out.

Ya yes  but the auto folder works fine, so we should be good as of now

start-druid -s=historical -c=
```/druid/apache-druid-26.0.0-SNAPSHOT/conf/druid/cluster```

<@U030K4UM3H7> this is the command I used 


Hi team, I called POST /druid/coordinator/v1/config with nodes to be decommissioned. The coordinator log shows the node `has 0 left to load, 0 left to drop, 0 bytes queued`. But when /segment/count metrics still shows there are thousands segments in the node. I also checked `sys.server_segments` table, the segment number didn’t go down to 0 but stuck to thousands as well.
Is it expected?

```2023-04-14T14:07:09,900 WARN [NodeRoleWatcher[COORDINATOR]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher - Ignored event type [CONNECTION_SUSPENDED] for node watcher of role [coordinator].
2023-04-14T14:07:09,900 ERROR [main-EventThread] org.apache.curator.framework.imps.CuratorFrameworkImpl - Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[zookeeper-3.5.9.jar:3.5.9]
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:885) ~[curator-framework-5.4.0.jar:5.4.0]
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:677) ~[curator-framework-5.4.0.jar:5.4.0]
	at org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152) ~[curator-framework-5.4.0.jar:5.4.0]
	at org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:187) ~[curator-framework-5.4.0.jar:5.4.0]
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:630) ~[zookeeper-3.5.9.jar:3.5.9]
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) ~[zookeeper-3.5.9.jar:3.5.9]```
and then
```org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[zookeeper-3.5.9.jar:3.5.9]
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:885) ~[curator-framework-5.4.0.jar:5.4.0]
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:677) ~[curator-framework-5.4.0.jar:5.4.0]
	at org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152) ~[curator-framework-5.4.0.jar:5.4.0]
	at org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:187) ~[curator-framework-5.4.0.jar:5.4.0]
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:630) ~[zookeeper-3.5.9.jar:3.5.9]
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) ~[zookeeper-3.5.9.jar:3.5.9]```
and then
```2023-04-14T14:07:09,995 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS.
org.apache.druid.query.QueryInterruptedException: null
	at org.apache.druid.query.BaseQuery.checkInterrupted(BaseQuery.java:53) ~[druid-processing-25.0.0.jar:25.0.0]
	at org.apache.druid.segment.QueryableIndexCursorSequenceBuilder$QueryableIndexCursor.advance(QueryableIndexCursorSequenceBuilder.java:448) ~[druid-processing-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.input.DruidSegmentReader$IntermediateRowFromCursorIterator.next(DruidSegmentReader.java:326) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.input.DruidSegmentReader$IntermediateRowFromCursorIterator.next(DruidSegmentReader.java:279) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.java.util.common.guava.BaseSequence$1.next(BaseSequence.java:116) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.java.util.common.guava.ConcatSequence$2.next(ConcatSequence.java:133) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.input.DruidSegmentReader$1.next(DruidSegmentReader.java:203) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.input.DruidSegmentReader$1.next(DruidSegmentReader.java:190) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.java.util.common.parsers.CloseableIteratorWithMetadata$1.next(CloseableIteratorWithMetadata.java:77) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.data.input.IntermediateRowParsingReader$1.hasNext(IntermediateRowParsingReader.java:69) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.java.util.common.parsers.CloseableIterator$2.findNextIteratorIfNecessary(CloseableIterator.java:74) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.java.util.common.parsers.CloseableIterator$2.next(CloseableIterator.java:108) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.java.util.common.parsers.CloseableIterator$1.next(CloseableIterator.java:52) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.common.task.FilteringCloseableInputRowIterator.hasNext(FilteringCloseableInputRowIterator.java:68) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.data.input.HandlingInputRowIterator.hasNext(HandlingInputRowIterator.java:63) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.common.task.InputSourceProcessor.process(InputSourceProcessor.java:96) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:922) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:526) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runSequential(ParallelIndexSupervisorTask.java:1199) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runTask(ParallelIndexSupervisorTask.java:532) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.common.task.CompactionTask.runTask(CompactionTask.java:507) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1623) ~[?:?]
Caused by: java.lang.InterruptedException```


looks like MM lost connection to ZK and then failed?

ZK is healthy and I don’t see any errors in ZK logs.

It looks like coordinator is flapping, Did you start a new version of the coordinator before terminating the old one during upgrade ?

this business

```Encountered exception in BUILD_SEGMENTS.
org.apache.druid.query.QueryInterruptedException: null```
is a thread getting interrupted, probably because the task overall was failing/canceled for some other reason

this one seems sketchier

```org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss```

i wonder if it is related to upgrading ZK / Curator or if it is a coincidence or if it is some third thing…

Yeah… that was the reason I asked, could it be related to Curator upgrade?

I'm not sure… how often does this `KeeperException$ConnectionLossException` happen roughly? Do you see it anywhere other than compaction tasks?

If it's a Curator issue you'd probably see it sporadically across a bunch of servers, as there isn't really anything special about tasks

If it's only compaction tasks it might be something else going on? Maybe JVM pauses (GC pauses perhaps? Virtualization steal?) that lead to connectionloss

I haven’t seen this error again since 3 days ago. We did fix incorrect -Xmx on MM that specialized in compaction. So maybe the OO exception is what causing this.