Apache Druid

but sometimes, the query planner is not that smart, see:

image.png

&lt; left is my old query &gt;
Right is the same query, but with dim = CAST(? as VARCHAR)

I was using 16 cores (4 servers), now they all sum up are only using 25% of 1 core

Screenshot 2022-12-12 at 11.11.59 PM.png

:rolling_on_the_floor_laughing::rolling_on_the_floor_laughing::rolling_on_the_floor_laughing:

Hi Sheng Wang,

Below should delete the unused segments and hence the old folder.

`druid.coordinator.kill.on`: When true, enables the Coordinator to submit a kill task for unused segments, which deletes them completely from metadata store and from deep storage. Only applies to the specified datasources in the dynamic configuration parameter killDataSourceWhitelist. If killDataSourceWhitelist is not set or empty, then kill tasks can be submitted for all datasources.
`druid.coordinator.kill.period`: Defines the frequency in ISO 8601 format for the cleanup job to check for and delete eligible segments. Defaults to P1D. Must be greater than druid.coordinator.period.indexingPeriod.
`druid.coordinator.kill.durationToRetain`: Defines the retention period in ISO 8601 format after creation that segments become eligible for deletion.
`druid.coordinator.kill.maxSegments`: Defines the maximum number of segments to delete per kill task.

More details in below link: <https://docs.imply.io/latest/druid/operations/clean-metadata-store/#segment-records-and-segments-in-deep-storage-kill-task>

hi，Bibek Sahoo.Thanks for your replying!
Our druid version is  0.17, dose 0.17 has the params you mentioned?

<https://druid.apache.org/docs/0.22.0/operations/clean-metadata-store.html>

I will check for the 0.17 release doc and update you

0.17 does not have this feature of auto kill, it got introduced in 0.22 as per above doc.

Can you try running the kill task manually as per the example given in below link, the link is for 0.17:
<https://druid.apache.org/docs/0.17.0/tutorials/tutorial-delete-data.html#run-a-kill-task>

we have use the kill task to delete the interval segment not in the retention setting.
for example: datasource A’s retention is 3day, we will submit kill task every day to delete the segment before 3days in hdfs.

in this reindex case, the interval is still used, we just want to delete the old version folder.But the kill task will delete the interval folder, it will result in the used version folder also be deleted.

for example:
in hdfs , 20221210T040000.000Z_20221210T050000.000Z interval has two version folders.
(1)
<hdfs://ip>:port/druid/segments/datasourceA/20221210T040000.000Z_20221210T050000.000Z/2022-12-10T03_59_30.758Z
(2)
<hdfs://ip>:port/druid/segments/datasourceA/20221210T040000.000Z_20221210T050000.000Z/2022-12-10T10_22_33.385Z

kill task will delete the folder:
<hdfs://ip>:port/druid/segments/datasourceA/20221210T040000.000Z_20221210T050000.000Z
but
<hdfs://ip>:port/druid/segments/datasourceA/20221210T040000.000Z_20221210T050000.000Z/2022-12-10T10_22_33.385Z
still used.

Kill task should remove only the UNUSED old segments and the new ones should not be affected.

If you open the Druid console and look at the status tile it will show you the Druid version

Imply has two version lines: monthly releases that are versioned `&lt;year&gt;.&lt;month&gt;.&lt;revision&gt;-iap` and LTS releases that are versioned `&lt;year&gt;.01.&lt;revision&gt;-lts-iap` . These correspond to the latest released Druid from that date + important patches and fixes brought forward from apache master

Hi <@U04EA3QR78T> more details on ingesting data from Kafka can be found here:
<https://druid.apache.org/docs/latest/tutorials/tutorial-kafka.html>

you can take two approaches….one is to publish to kafka from nifi na dthen ingest into druid. The other is to write out files and do a batch ingest to druid

<@U03D7RPAT0A> - Can you share an example of the query used?

did I read that right you are on Druid 0.9.1 ?

<@U030W50CNAC> yeah you read right , we are using very older and in process to backfill druid data .

<@U03S0JXTM4N> Sharing example logs , threshold : 5000 and fromnext: true . Using while loop and exit from while loop once we get empty result set but we don't get empty result set because getting same paging identifiers repeatedly .

test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z
test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1

```pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 2708, 'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1': 2290}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 5451, 'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1': 4547}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 9388, 'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1': 5610}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 10421, 'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1': 9577}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 15421}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1': 4999}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 4999}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1': 4547, 'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 5451}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds>
Response SELECT Query: Datasource Name:test-ds , Query API Resonse Code:200
pagingIdentifiers for next Query: {'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z': 9388, 'test-ds_2021-10-13T00:00:00.000Z_2021-10-13T08:01:00.000Z_2022-02-07T05:09:25.413Z_1': 5610}
Requesting SELECT query: <http://test:8080/druid/v2/test-ds> ```

I have not used the `select` query in a while - it was replaced with `scan` at some point

have you tried not using `fromNext` and just incrementing the identifiers by 1 yourself?

also are you just exporting all the data without a filter - if so you could also use the dump segment tool (just wanted to mention)

yes i tried using fromnext:false and increment offset manually by code but it does not work in case of multi-partitions . i have designed complete backfill logic to pull druid data using backfill framework which runs on workflow engine and scalable , we backfilled around 50 TB raw data using this framework .  facing an issue in one of the corner case .

yes, I remember that when Plywood (the toolkit I built for querying Druid) used `select` (it long doesn't anymore) it used to do the increment client side <https://github.com/implydata/plywood/blob/d2ff8dff936baaadbc93396e8c65b39486fe837e/src/external/druidExternal.ts#L447>

I wonder why the multi-partition stuff does not work for you, did you make sure to increment all the counters like the code :point_up: does?

yes , few folks also faced similar issues but those were closed due to inactivity . <https://github.com/apache/druid/issues/4849>

Also fromnext:true it increment offset automatically and take cares internally and it should work with multi partitions but not sure why it's happening with multi-partitions . logs shared above .

<@U030W50CNAC> <@U03S0JXTM4N> thanks !
I added a fix at code level .
```    global paging
    paging_check={} 
      
        for key, value in paging.items():
                            if not paging_check:
                                paging_check.update(paging)
                            if key not in paging_check:
                                paging_check[key] = value
                            if value &gt; paging_check[key] and paging_check[key] != -1:
                                paging_check[key] = value
                            if value &lt; paging_check[key]:
                                paging_check[key] = -1
                        <http://logger.info|logger.info>("Custom hashmap {}".format(paging_check))
                        for key1, value1 in paging_check.items():
                            if value1 == -1:
                                for key1 in list(paging.keys()):
                                    del paging[key1]
                            else:
                                paging[key1] = value1
                            <http://logger.info|logger.info>(
                                "pagingIdentifiers for next Query through backfill code: {0}".format(paging))

                        if all(v == -1 for v in paging_check.values()):
                            events = []```

first check if the late data is either ignored on the ingestion task (there's a parameter for that) or that it's not put in the topic in the first place

also check if there's any auto-compaction enabled, because that could also try to compact the segments and uses a lock (it has lower priority than the indexing, but can lock out the batch ingestion)

instead of overwrite, you can also queue the old data, and insert into the topic once a day

an then let the auto-compation do the merging

:studio_microphone: Hey team! We’re having a drop-in session for anyone who’d like to come and say hello :slightly_smiling_face: Details here <https://www.meetup.com/apache-druid-london/events/290021436/>

Another Virtual Meetup today at 12 PST, in this session we'll review Apache Druid 24.0, its nested JSON capabilities and SQL based ingestion. Register and join us at: <https://www.meetup.com/druidio/events/290241696/>

you could drop a specific segment manually and then append a new segment in its place but in general the answer is "no". All the reindexing operations are on a per time-chunk level

there are a lot of reasons for it, one of them being that after manipulating data for a customer_id it might be advantageous to repartition data within the time chunk in question

Hi Niranjan Sridhara,

You may follow below guide:<https://druid.apache.org/docs/latest/ingestion/native-batch-input-sources.html#s3-input-source>

More details on S3 extention: <https://druid.apache.org/docs/latest/development/extensions-core/s3.html>

Also you can try ingesting from  druid console, load data&gt;S3  . It will generate  spec at the end and you can edit as per req with help of above docs.

Hi Abdel - <https://druid.apache.org/docs/latest/multi-stage-query/index.html|SQL inserts> are new, and may not be available in PyDruid yet

Thanks for your answer @Kyle !

I have tried to use the API directly with the following query:
```INSERT INTO d1_ingest VALUES ('2019/08/30T00:00:00' , 'st0' , 0.086852, 0.256154, 0.353368, 0.264800, 0.340716, 0.888801, 0.098555, 0.990292, 0.006500, 0.197772, 0.056110, 0.711823, 0.113245, 0.203696, 0.749044, 0.637718, 0.197120, 0.396791, 0.868563, 0.503080, 0.403444, 0.050145, 0.434476, 0.649876, 0.001515, 0.970305, 0.473175, 0.563040, 0.310380, 0.273118, 0.551110, 0.308798, 0.783165, 0.950974, 0.926163, 0.802391, 0.226309, 0.432790, 0.114836, 0.347638, 0.432519, 0.838368, 0.638415, 0.944477, 0.735955, 0.419511, 0.152170, 0.333158, 0.566972, 0.028693, 0.491042, 0.604122, 0.764282, 0.577141, 0.267561, 0.432286, 0.157174, 0.851707, 0.213233, 0.095803, 0.982957, 0.841460, 0.533001, 0.090531, 0.386728, 0.363502, 0.281768, 0.431700, 0.852836, 0.398835, 0.760999, 0.160066, 0.475869, 0.028385, 0.111324, 0.948984, 0.575755, 0.741031, 0.625195, 0.057176, 0.590370, 0.498445, 0.879716, 0.717900, 0.597301, 0.596217, 0.407540, 0.750143, 0.736843, 0.264539, 0.457388, 0.054152, 0.486477, 0.525835, 0.204051, 0.694023, 0.558147, 0.668027, 0.166661, 0.745691) PARTITIONED BY DAY ```
But this gives an error too:
```Error: Plan validation failed
sqlOuterLimit cannot be provided with INSERT.
org.apache.calcite.tools.ValidationException```
Any ideas of why is that? Any ideas on how to perform small continuous remote inserts?

Thanks!

I think this would look more like:
```REPLACE INTO "d1_ingest" OVERWRITE ALL
WITH "ext" AS (SELECT *
FROM TABLE(
  EXTERN(
    '{"type":"inline","data":"''2019/08/30T00:00:00'' , ''st0'' , 0.086852, 0.256154, 0.353368, 0.264800, 0.340716, 0.888801, 0.098555, 0.990292, 0.006500, 0.197772, 0.056110, 0.711823, 0.113245, 0.203696, 0.749044, 0.637718, 0.197120, 0.396791, 0.868563, 0.503080, 0.403444, 0.050145, 0.434476, 0.649876, 0.001515, 0.970305, 0.473175, 0.563040, 0.310380, 0.273118, 0.551110, 0.308798, 0.783165, 0.950974, 0.926163, 0.802391, 0.226309, 0.432790, 0.114836, 0.347638, 0.432519, 0.838368, 0.638415, 0.944477, 0.735955, 0.419511, 0.152170, 0.333158, 0.566972, 0.028693, 0.491042, 0.604122, 0.764282, 0.577141, 0.267561, 0.432286, 0.157174, 0.851707, 0.213233, 0.095803, 0.982957, 0.841460, 0.533001, 0.090531, 0.386728, 0.363502, 0.281768, 0.431700, 0.852836, 0.398835, 0.760999, 0.160066, 0.475869, 0.028385, 0.111324, 0.948984, 0.575755, 0.741031, 0.625195, 0.057176, 0.590370, 0.498445, 0.879716, 0.717900, 0.597301, 0.596217, 0.407540, 0.750143, 0.736843, 0.264539, 0.457388, 0.054152, 0.486477, 0.525835, 0.204051, 0.694023, 0.558147, 0.668027, 0.166661, 0.745691"}',
    '{"type":"csv","findColumnsFromHeader":false,"columns":["column001","column002","column003","column004","column005","column006","column007","column008","column009","column010","column011","column012","column013","column014","column015","column016","column017","column018","column019","column020","column021","column022","column023","column024","column025","column026","column027","column028","column029","column030","column031","column032","column033","column034","column035","column036","column037","column038","column039","column040","column041","column042","column043","column044","column045","column046","column047","column048","column049","column050","column051","column052","column053","column054","column055","column056","column057","column058","column059","column060","column061","column062","column063","column064","column065","column066","column067","column068","column069","column070","column071","column072","column073","column074","column075","column076","column077","column078","column079","column080","column081","column082","column083","column084","column085","column086","column087","column088","column089","column090","column091","column092","column093","column094","column095","column096","column097","column098","column099","column100","column101","column102"]}',
    '[{"name":"column001","type":"string"},{"name":"column002","type":"string"},{"name":"column003","type":"double"},{"name":"column004","type":"double"},{"name":"column005","type":"double"},{"name":"column006","type":"double"},{"name":"column007","type":"double"},{"name":"column008","type":"double"},{"name":"column009","type":"double"},{"name":"column010","type":"double"},{"name":"column011","type":"double"},{"name":"column012","type":"double"},{"name":"column013","type":"double"},{"name":"column014","type":"double"},{"name":"column015","type":"double"},{"name":"column016","type":"double"},{"name":"column017","type":"double"},{"name":"column018","type":"double"},{"name":"column019","type":"double"},{"name":"column020","type":"double"},{"name":"column021","type":"double"},{"name":"column022","type":"double"},{"name":"column023","type":"double"},{"name":"column024","type":"double"},{"name":"column025","type":"double"},{"name":"column026","type":"double"},{"name":"column027","type":"double"},{"name":"column028","type":"double"},{"name":"column029","type":"double"},{"name":"column030","type":"double"},{"name":"column031","type":"double"},{"name":"column032","type":"double"},{"name":"column033","type":"double"},{"name":"column034","type":"double"},{"name":"column035","type":"double"},{"name":"column036","type":"double"},{"name":"column037","type":"double"},{"name":"column038","type":"double"},{"name":"column039","type":"double"},{"name":"column040","type":"double"},{"name":"column041","type":"double"},{"name":"column042","type":"double"},{"name":"column043","type":"double"},{"name":"column044","type":"double"},{"name":"column045","type":"double"},{"name":"column046","type":"double"},{"name":"column047","type":"double"},{"name":"column048","type":"double"},{"name":"column049","type":"double"},{"name":"column050","type":"double"},{"name":"column051","type":"double"},{"name":"column052","type":"double"},{"name":"column053","type":"double"},{"name":"column054","type":"double"},{"name":"column055","type":"double"},{"name":"column056","type":"double"},{"name":"column057","type":"double"},{"name":"column058","type":"double"},{"name":"column059","type":"double"},{"name":"column060","type":"double"},{"name":"column061","type":"double"},{"name":"column062","type":"double"},{"name":"column063","type":"double"},{"name":"column064","type":"double"},{"name":"column065","type":"double"},{"name":"column066","type":"double"},{"name":"column067","type":"double"},{"name":"column068","type":"double"},{"name":"column069","type":"double"},{"name":"column070","type":"double"},{"name":"column071","type":"double"},{"name":"column072","type":"double"},{"name":"column073","type":"double"},{"name":"column074","type":"double"},{"name":"column075","type":"double"},{"name":"column076","type":"double"},{"name":"column077","type":"double"},{"name":"column078","type":"double"},{"name":"column079","type":"double"},{"name":"column080","type":"double"},{"name":"column081","type":"double"},{"name":"column082","type":"double"},{"name":"column083","type":"double"},{"name":"column084","type":"double"},{"name":"column085","type":"double"},{"name":"column086","type":"double"},{"name":"column087","type":"double"},{"name":"column088","type":"double"},{"name":"column089","type":"double"},{"name":"column090","type":"double"},{"name":"column091","type":"double"},{"name":"column092","type":"double"},{"name":"column093","type":"double"},{"name":"column094","type":"double"},{"name":"column095","type":"double"},{"name":"column096","type":"double"},{"name":"column097","type":"double"},{"name":"column098","type":"double"},{"name":"column099","type":"double"},{"name":"column100","type":"double"},{"name":"column101","type":"double"},{"name":"column102","type":"double"}]'
  )
))
SELECT
  "column001",
  "column002",
  "column003",
  "column004",
  "column005",
  "column006",
  "column007",
  "column008",
  "column009",
  "column010",
  "column011",
  "column012",
  "column013",
  "column014",
  "column015",
  "column016",
  "column017",
  "column018",
  "column019",
  "column020",
  "column021",
  "column022",
  "column023",
  "column024",
  "column025",
  "column026",
  "column027",
  "column028",
  "column029",
  "column030",
  "column031",
  "column032",
  "column033",
  "column034",
  "column035",
  "column036",
  "column037",
  "column038",
  "column039",
  "column040",
  "column041",
  "column042",
  "column043",
  "column044",
  "column045",
  "column046",
  "column047",
  "column048",
  "column049",
  "column050",
  "column051",
  "column052",
  "column053",
  "column054",
  "column055",
  "column056",
  "column057",
  "column058",
  "column059",
  "column060",
  "column061",
  "column062",
  "column063",
  "column064",
  "column065",
  "column066",
  "column067",
  "column068",
  "column069",
  "column070",
  "column071",
  "column072",
  "column073",
  "column074",
  "column075",
  "column076",
  "column077",
  "column078",
  "column079",
  "column080",
  "column081",
  "column082",
  "column083",
  "column084",
  "column085",
  "column086",
  "column087",
  "column088",
  "column089",
  "column090",
  "column091",
  "column092",
  "column093",
  "column094",
  "column095",
  "column096",
  "column097",
  "column098",
  "column099",
  "column100",
  "column101",
  "column102"
FROM "ext"
PARTITIONED BY ALL```

That worked, thanks <@U030MBK46BD>!!
However, inserting this one row took 11 seconds. I am looking to do dynamic frequent insertions (as small batches). Is there a better version of this query to allow faster insertions?
Thanks a lot!

Here are the insertion execution details:

Not really, my use-case involves raw insertions (ideally but not necessarily in SQL) without additional tools. I basically need to send small collected data to a Druid server with a fixed rate (e.g. 100 rows/sec)

for something small like this, it might make sense to use the old-fashioned ingestion spec. This would reduce the need to start a controller _and_ worker task and might be quicker because of this

I see! To my understanding, ingestion spec requires storing the data to a file and running the ingestion spec while referring to the data file. I basically need to push new data from the memory to Druid from a client machine on a second basis. This is why Pydruid with the INSERT INTO query was my first move.

Do you think ingestion spec is still usable for my use-case?

inline_native.json

Got it! I just don’t get how would I then run the ingestion spec remotely from the client without copying the data into the Druid machine?

if u don't want add kafka to the stack, you can try using redpanda, it should work

100row/s is a too few to batch, I think the overhead of a task for each second is too much

The INSERT and REPLACE functionality is only available via the task based SQL API. The task based SQL API is located on the endpoint `/druid/v2/sql/task` (vs the native based SQL API that is on `/druid/v2/sql/`). In the console you can select which engine (API) you use from this drop down:

The task based API (<https://druid.apache.org/docs/latest/multi-stage-query/api.html>) is async, so you issue the query, get a taskId back and then have to monitor the status of that task. The console does it for you

note also that the console by default has an auto engine selector mode that will automatically pick what engine to use for your query (based on the contents of the query), but you can also explicitly select one

It is not recommended to load 1 or only a small number of rows into Druid using batch ingestion (INSERT SQL or `index_parallel`). This is exactly what streaming ingestion is for so set up a Kafka topic, make a Supervisor that will connect to that topic and then publish your events to that topic one at a time. Batch ingestion in Druid is built with the idea of loading a large amount of data all at once.

Thanks <@U030MBK46BD>, <@U038XPQFJHM> and <@U030W50CNAC>!!

That sounds like it should work fine. Let us know how it goes. If you are using the Druid Helm chart to deploy, you should disable its ZK deployment and make sure you reference your zookeeper instance in the Druid config. Or you could just use the ZK in that helm chart instead.

I was not aware of this helm chart with in druid repo. Let me try that then . Thanks

If you have questions about Apache Druid, want to discuss a use case, or want to hear about others using Druid, <https://www.meetup.com/austin-apache-druid-meetup-by-imply/events/290102009?utm_medium=referral&amp;utm_campaign=share-btn_savedevents_share_modal&amp;utm_source=link|come to hour Office Hours Meetup> at 10AM PST today. It's an informal format in which we take on the subjects as they come. Experts are more than welcome, we love when you share your experience.

This represents the UTC date time value for epoch `Long.MAX_VALUE` in java (9223372036854775807). Druid by default uses an `ETERNITY` time interval filter on the queries which corresponds to `[Long.MIN_VALUE, Long.MAX_VALUE]` epoch range.

One clarification, the epoch values are actually `Long.MAX_VALUE / 2` and `Long.MIN_VALUE / 2` instead of `Long.MAX_VALUE` and `Long.MIN_VALUE`
It is so because the interval duration is needed to be storable in a java long - (`Long.MAX_VALUE - Long.MIN_VAUE` can't come in a java long but `Long.MAX_VALUE / 2 - Long.MIN_VALUE / 2` ) can

Thank you <@U031A2GTWEM> for a very comprehensive answer!

The web console is just a static webapp served from the router node. You can secure your Druid by following these instructions: <https://druid.apache.org/docs/latest/operations/security-overview.html>

the console (being just another API) will adopt whatever you enable to secure all your APIs.

Hello Druids!

The Apache Druid team is proud to announce the release of *Apache Druid 24.0.2*.
Druid is a high-performance analytics data store for event-driven data.

Apache Druid 24.0.2 is a bug fix release that fixes an issue with Hadoop ingestion with a custom extension config. The details can be found here: <https://github.com/apache/druid/milestone/49?closed=1>

Source and binary distributions can be downloaded from:
<https://druid.apache.org/downloads.html>

Release notes are at:
<https://github.com/apache/druid/releases/tag/druid-24.0.2>

Thanks to all the contributors to this release!

Sounds like something might have changed on the kafka side, or network side?   Is the 404 for a subtask log, or the main task log?  Sometimes subtasks don't write a log before they die, I think, so you get a 404.  But main task should have a log, normally.

When I do 'kill' the sub-tasks (tasks in ingestion using web-console interface) again it's running.

if kafka side got changed means (it's shouldn't run when I the kill the PENDING sub-task ) right ?

I'm not sure.  I looked up some other examples of people getting the same error, and they were due to peon heap sizes.  In the middle managers, does runtime.properties have an `-Xmx1 setting?  Is there room (in available RAM) to try increasing it and restarting the MM processes?

<@U0318CX1NH4> would you please consider creating a #release channel so that information about releases doesn't get lost in all the rest of the activity here in <#C0303FDCZEZ|> ?

actually nevermind on the copy paste... I posted a link there and pinned the latest release announcement to this channel

How do you know it's getting the data?  Could be many things, maybe coordinator http timeout settings, idk.  Any details in overlord or coordinator logs if you search for the task name?

you are right, I don't know. So I created a supervisor spec using the wizard initially, and from that it connected to the kafka stream, got some data and I went followed the steps to create a supervisor specs... so yeah, I can't assume its getting the data, I think I can say the connectivity is there at least. So the supervisor is running, kicking off these peon tasks periodically.. and they have their own logs which seem to have no errors in. These are the indexing logs I am talking about. They do however have this warning...

```2022-12-22T14:36:42,721 WARN [main] org.apache.druid.java.util.metrics.AllocationMetricCollectors - Cannot initialize org.apache.druid.java.util.metrics.AllocationMetricCollector
java.lang.reflect.InaccessibleObjectException: Unable to make public long[] com.sun.management.internal.HotSpotThreadImpl.getThreadAllocatedBytes(long[]) accessible: module jdk.management does not "exports com.sun.management.internal" to unnamed module @27ae2fd0
        at java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:340) ~[?:?]
        at java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280) ~[?:?]
        at java.lang.reflect.Method.checkCanSetAccessible(Method.java:198) ~[?:?]
        at java.lang.reflect.Method.setAccessible(Method.java:192) ~[?:?]
        at org.apache.druid.java.util.metrics.AllocationMetricCollectors.&lt;clinit&gt;(AllocationMetricCollectors.java:41) ~[druid-core-24.0.1.jar:24.0.1]
        at org.apache.druid.java.util.metrics.JvmMonitor.&lt;init&gt;(JvmMonitor.java:67) ~[druid-core-24.0.1.jar:24.0.1]
        at org.apache.druid.java.util.metrics.JvmMonitor.&lt;init&gt;(JvmMonitor.java:59) ~[druid-core-24.0.1.jar:24.0.1]
        at org.apache.druid.server.metrics.MetricsModule.getJvmMonitor(MetricsModule.java:148) ~[druid-server-24.0.1.jar:24.0.1]
        at org.apache.druid.server.metrics.MetricsModule$$FastClassByGuice$$99ddce1b.invoke(&lt;generated&gt;) ~[druid-server-24.0.1.jar:24.0.1]
        at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264) ~[guice-4.1.0.jar:?]
        at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401) ~[guice-4.1.0.jar:?]```
there are however these errors I've not managed to get to the bottom of...

```2022-12-22T14:38:52,228 ERROR [main-EventThread] org.apache.curator.framework.imps.EnsembleTracker - Invalid config event received: {server.2=w3-druid-zk-03-preprod:2888:3888:participant, server.1=w3-druid-zk-02-preprod:2888:3888:participant, server.0=0.0.0.0:2888:3888:participant, version=0}
2022-12-22T14:38:52,229 INFO [main-EventThread] org.apache.curator.framework.imps.EnsembleTracker - New config event received: {server.2=w3-druid-zk-03-preprod:2888:3888:participant, server.1=w3-druid-zk-02-preprod:2888:3888:participant, server.0=0.0.0.0:2888:3888:participant, version=0}
2022-12-22T14:38:52,229 ERROR [main-EventThread] org.apache.curator.framework.imps.EnsembleTracker - Invalid config event received: {server.2=w3-druid-zk-03-preprod:2888:3888:participant, server.1=w3-druid-zk-02-preprod:2888:3888:participant, server.0=0.0.0.0:2888:3888:participant, version=0}```
I think the zookeeper config is supposed to have 0.0.0.0 for the current node... I don't think the cluster forms if its not set to that.

Interesting.  If you follow the wizard, does the working config match this:

```{server.2=w3-druid-zk-03-preprod:2888:3888:participant, server.1=w3-druid-zk-02-preprod:2888:3888:participant, server.0=0.0.0.0:2888:3888:participant, version=0}```


Is this cluster running on a single machine?

<@U030W50CNAC> its on 5 machines, each of them redhat openshift vms. The druid doctor returns successfully. <@U0411DE2SS0> not sure what exactly you mean. I am thinking the zookeeper connection is correct mainly based of the fact that the cluster can form at all.

I see this in the task logs if that helps
```{
  "id": "index_kafka_sn-content-events_f6e229d741032ab_cifepkmi",
  "groupId": "index_kafka_sn-content-events",
  "type": "index_kafka",
  "createdTime": "2022-12-23T23:04:52.620Z",
  "queueInsertionTime": "1970-01-01T00:00:00.000Z",
  "statusCode": "FAILED",
  "status": "FAILED",
  "runnerStatusCode": "RUNNING",
  "duration": -1,
  "location": {
    "host": "w3-druid-data-01-preprod",
    "port": 8100,
    "tlsPort": -1
  },
  "dataSource": "sn-content-events",
  "errorMsg": "Task [index_kafka_sn-content-events_f6e229d741032ab_cifepkmi] failed to return start time, killing t..."
}```

Getting to this after 2 months (I saw a link to the thread recently), so I'm not sure if this is still relevant :slightly_smiling_face:

But… I wonder what version you're using? In 25.0 the code that checks start times (and all other RPC stuff) was rewritten to be more robust (<https://github.com/apache/druid/pull/13354>)

So, if you're using an earlier version, I would suggest upgrading

If it's still happening on 25.0+, then we should look into it more deeply

Hi,
1. What is your version ?
2. This is an `Info/WARN` warning - are the ingestion tasks running ?
3. `connection timed out` - can you check connectivity to your Kafka broker?

Also check your kafka listener. If you are using ip address for broker but if the advertised listener is returning host names and the dns is not correct then timeout will happen during metadata fetch

<@U04CGQ1P68Z>
1. My Kafka Version: `2.6.1`
2. Yes the ingestion tasks are running but without actually consuming any data from the kafka topic. I can see the reported lag for my topic though in the supervisor status
3. The other supervisor tasks run fine, but sometimes cause this issue and have to restart the entire cluster

<@U03BJAMQ4H5> My kafka runs on MSK and I am using the DNS directly

Attaching my supervisor status:
```{
  "dataSource": "conversation-analytics-events-v2",
  "stream": "conversations-analytics-event-v2",
  "partitions": 2,
  "replicas": 1,
  "durationSeconds": 3600,
  "activeTasks": [
    {
      "id": "index_kafka_conversation-analytics-events-v2_896fb93ac9e434b_fommhjii",
      "startingOffsets": {
        "0": 124556,
        "1": 124659
      },
      "startTime": "2022-12-26T09:08:54.117Z",
      "remainingSeconds": 1594,
      "type": "ACTIVE",
      "currentOffsets": {
        "0": 124556,
        "1": 124659
      },
      "lag": {
        "0": 8,
        "1": 7
      }
    }
  ],
  "publishingTasks": [],
  "latestOffsets": {
    "0": 124564,
    "1": 124666
  },
  "minimumLag": {
    "0": 8,
    "1": 7
  },
  "aggregateLag": 15,
  "offsetsLastUpdated": "2022-12-26T09:42:09.846Z",
  "suspended": false,
  "healthy": true,
  "state": "RUNNING",
  "detailedState": "RUNNING",
  "recentErrors": [
    {
      "timestamp": "2022-12-26T09:03:08.759Z",
      "exceptionClass": "java.util.concurrent.ExecutionException",
      "message": "org.apache.druid.indexing.common.IndexTaskClient$TaskNotRunnableException: Aborting request because task [index_kafka_conversation-analytics-events-v2_d36b77f9038cc00_mjlehghd] is not runnable",
      "streamException": false
    }
  ]
}```

Hi Chandra, I set up `24.0.2` and I wonder if you can provide me with a credentials on accessing your MSK with a dummy datasource. I am not familiar with MSK. If you could provide the
1. `topic`
2. `consumerProperties`
Thanks !

<@U04FKDX12ET> what is your advertised listener in listener config?

<@U04CGQ1P68Z> Its just AWS managed Kafka Services. Can we catchup over a quick huddle and debug instead please?

<@U03BJAMQ4H5> I am advertising 8091 port and yes both kafka and my middlemanager/peon are in the same VPC

For reference, attaching my supervisor config;
```{
  "type": "kafka",
  "spec": {
    "dataSchema": {
      "dataSource": "analytics-conversation-transformed-event",
      "timestampSpec": null,
      "dimensionsSpec": null,
      "metricsSpec": [],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "HOUR",
        "queryGranularity": {
          "type": "none"
        },
        "rollup": true,
        "intervals": []
      },
      "transformSpec": {
        "filter": null,
        "transforms": []
      },
      "parser": {
        "type": "string",
        "parseSpec": {
          "format": "json",
          "timestampSpec": {
            "column": "timestamp",
            "format": "auto"
          },
          "dimensionsSpec": {
            "dimensions": [
              "botId",
              "botReferenceId",
              "recordId",
              "customFields",
              "subject",
              "firstIntent",
              "conversationId",
              "uuid",
              "messageId",
              "exchangeType",
              "env",
              "falsePositive",
              "interactionMode",
              "visitorMessageSource",
              "keywords",
              "classificationStatus",
              "visitorMessageTime",
              "timestamp",
              "channel",
              "locale",
              "supportedLanguage",
              "exchangeCategory",
              "exchangeSubCategory",
              "skillId",
              "intentId",
              "intent",
              "intentCategory",
              "previousIntentId",
              "mappedTopic",
              "mappedCategory",
              "inboundMessageText",
              "botMessage",
              "ticketStatus",
              "currentState",
              "escalationReason",
              "feedback",
              "customerFeedback",
              "agentFeedback",
              "sendStatus",
              "customFieldsJson",
              "shortAnswer",
              "longAnswer",
              "question",
              "nodeType",
              "masterIntentId",
              "kbAnswer",
              "kbAnswerQuestion",
              "kbAnswerShortAnswer",
              "kbAnswerLongAnswer",
              "kbArticles",
              "masterIntent",
              "feedbackRemarks",
              "exchangeDuration",
              "narrowingOption1",
              "narrowingOption3",
              "narrowingOption2",
              "narrowingOption4",
              "narrowingOption6",
              "narrowingOption7",
              "narrowingOption8",
              "narrowingOption9",
              "narrowingOption10",
              "suggestionsOption1",
              "narrowingOption5",
              "suggestionsOption2",
              "suggestionsOption3",
              "suggestionsOption4",
              "suggestionsOption5",
              "suggestionsOption6",
              "suggestionsOption7",
              "suggestionsOption8",
              "suggestionsOption9",
              "suggestionsOption10",
              "menuOption1",
              "menuOption2",
              "menuOption3",
              "menuOption4",
              "menuOption5",
              "menuOption6",
              "menuOption7",
              "switchType",
              "fromIntentId",
              "menuOption8",
              "menuOption9",
              "menuOption10"
            ],
            "dimensionExclusions": [],
            "spatialDimensions": []
          }
        }
      }
    },
    "ioConfig": {
      "topic": "analytics-conversation-transformed-event",
      "inputFormat": null,
      "replicas": 1,
      "taskCount": 1,
      "taskDuration": "PT3600S",
      "consumerProperties": {
        "bootstrap.servers": "host1:9092,host2:9092"
      },
      "pollTimeout": 100000,
      "startDelay": "PT5S",
      "period": "PT30S",
      "useEarliestOffset": false,
      "completionTimeout": "PT1800S",
      "lateMessageRejectionPeriod": null,
      "earlyMessageRejectionPeriod": null,
      "lateMessageRejectionStartDateTime": null,
      "stream": "analytics-conversation-transformed-event",
      "useEarliestSequenceNumber": false,
      "autoscalerConfig": null,
      "type": "kafka"
    },
    "tuningConfig": {
      "type": "kafka",
      "appendableIndexSpec": {
        "type": "onheap"
      },
      "maxRowsInMemory": 500,
      "maxBytesInMemory": 0,
      "skipBytesInMemoryOverheadCheck": false,
      "maxRowsPerSegment": 50000,
      "maxTotalRows": null,
      "intermediatePersistPeriod": "PT2M",
      "maxPendingPersists": 0,
      "indexSpec": {
        "bitmap": {
          "type": "roaring",
          "compressRunOnSerialization": true
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs",
        "segmentLoader": null
      },
      "indexSpecForIntermediatePersists": {
        "bitmap": {
          "type": "roaring",
          "compressRunOnSerialization": true
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs",
        "segmentLoader": null
      },
      "reportParseExceptions": false,
      "handoffConditionTimeout": 0,
      "resetOffsetAutomatically": false,
      "segmentWriteOutMediumFactory": null,
      "workerThreads": 10,
      "chatThreads": 10,
      "chatRetries": 8,
      "httpTimeout": "PT10S",
      "shutdownTimeout": "PT80S",
      "offsetFetchPeriod": "PT30S",
      "intermediateHandoffPeriod": "PT10M",
      "logParseExceptions": false,
      "maxParseExceptions": 2147483647,
      "maxSavedParseExceptions": 0,
      "skipSequenceNumberAvailabilityCheck": false,
      "repartitionTransitionDuration": "PT120S"
    }
  }
}```

For additional information, I am running my entire druid setup on AWS Elastic Container Services and I am not setting the `druid.host` property

This looks to me an coordinator problem which generally run on 8081. (the plaint text version of it).
My guess is to check the coordinator logs first and to see whats happening there.
If nothing strikes out, check nw connectivity between peons, overlord -&gt; coordinator.

This is for the first exception.

For the `index_kafka_conversation-analytics-events-v2_d36b77f9038cc00_mjlehghd` not found
• Go to the overlord
• Grep for `index_kafka_conversation-analytics-events-v2_d36b77f9038cc00_mjlehghd` in the logs and see which MM this task is assinged.
• Go to the mm and see whats happening with `index_kafka_conversation-analytics-events-v2_d36b77f9038cc00_mjlehghd`.

The <https://druid.apache.org/docs/latest/configuration/index.html#metrics-monitors|metrics monitors> configured determine the extent of output metrics being emitted

I believe you read the docs on *druid.emitter.prometheus.dimensionMapPath*
```JSON file defining the Prometheus metric type, desired dimensions, help text, and conversionFactor for every Druid metric.```

<https://druid.apache.org/docs/latest/operations/metrics.html#ingestion-metrics>

`ingest/count` - ingest metrics i am not able to get
in code i can see this class `org.apache.druid.indexing.common.statsTaskRealtimeMetricsMonitor`
when i add it to `druid.monitoring.monitors=["org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor"]`
getting this error

```1 error
        at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:72)
        at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:62)
        at org.apache.druid.cli.Main.main(Main.java:113)
Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Could not find a suitable constructor in org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor.class(TaskRealtimeMetricsMonitor.java:44)
  while locating org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor
  at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:99) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
  at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:99) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
  while locating org.apache.druid.java.util.metrics.MonitorScheduler
  at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:84) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
  while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness) ```


<@U04CGQ1P68Z> thanks for the info
what is the use of this file
*`druid.emitter.prometheus.dimensionMapPath=/etc/druid/conf/_common/defaultMetricDimensions.json`*  ?
is it to restrict the emission of metrics

and by default are all metrics emitted?

as defined in docs:
```JSON file defining the Prometheus metric type, desired dimensions, help text, and conversionFactor for every Druid metric.```
It does not seem to restrict the emission of metrics

actuallu without *defaultMetricDimensions.json* this file also metrics are emmited so i am asking

As I said, I don’t believe this file restricts any emission of metrics

Your error could be due to some Kafka ingestion formats. Could you validate that ?

in my case i have only batch ingestion. nothing for kafka

while debugging i saw as the error sugest this metrics monitor doesn't have `'constructor annotated with @Inject or a zero-argument constructor that is not private'`

other monitors have @Inject on top of public constructure

why do you need this then ? `org.apache.druid.indexing.common.statsTaskRealtimeMetricsMonitor`

Hi, the best way is to export to a GCS bucket and run an `index_parallel`  job to load it in or use MSQE
<https://druid.apache.org/docs/latest/development/extensions-core/google.html>

from documentation:
```When the segment lock is used, a task locks individual segments instead of the entire time chunk. As a result, two or more tasks can create segments for the same time chunk of the same data source simultaneously if they are reading different segments. For example, a Kafka indexing task and a compaction task can always write segments into the same time chunk of the same data source simultaneously. The reason for this is because a Kafka indexing task always appends new segments, while a compaction task always overwrites existing segments. The segments created with the segment locking have the same major version and a higher minor version.

The segment locking is still experimental. It could have unknown bugs which potentially lead to incorrect query results.```
<https://druid.apache.org/docs/latest/ingestion/tasks.html#locking>

I think there are still issues with it.  Usual approach is to use a combination of rejecting data that's too late, and not compacting data for some period (eg, a 1 day offset).