Apache Druid

not that i know of, although on the query side, the SQL API is pretty simple: just the one endpoint and relatively straightforward request/response format

not sure if it is designed to go back and readjust everything after a change like that, <@U035FNDU6HH> do you know?

Auto compction will go back and change the partitioning of existing segments (starting with the latest segment and working it way to older segments) 

You don’t need to set dropExisting. dropExisting is designed for when you change the segmentGranularity and there is gap in the data (for example, change from Month granularity to day granularity and you don’t have data on every day of the month thus preventing the original month segment to be dropped)

Auto compaction is designed to replace existing segment by default 

Thanks <@U035FNDU6HH> That explains why I don't see older data not compacted.  I have slots available however I don't see compaction tasks running more often as there many old segments to compact

That may also depends on the slot allocated to auto compaction

To change the slots allocated to auto compaction: POST to /druid/coordinator/v1/config/compaction/taskslots?ratio={someRatio}&amp;max={someMaxSlots}

The actual max number of compaction tasks is min(max, ratio * total task slots)

We have 10 solts total, and 4  are used for for ingestion  and our compaction spec is    `"compactionTaskSlotRatio":0.4,`
   `"maxCompactionTaskSlots":5,`
   `"useAutoScaleSlots":false`

Are you seeing less than 4 slots being used in each cycle? What is the configured maxNumConcurrentSubTasks in the auto compaction spec?

I mean I see compaction tasks finish in around~5 mins and I dont see any compaction tasks next ~30 mins

I see 4 slots being used when compaction runs

Issue is not the number of slots but frequency of these tasks run

Ah. That is because tasks are issued during a run of the auto compaction duty. The duty is set to run at druid.coordinator.period.indexingPeriod

This is How often to send compact/merge/conversion tasks to the indexing service. And is default to 30 mins

You can change this config to a lower value but note that this config also control the frequency of some other coordinator duty

Depending on what your druid version is, you can see <https://github.com/apache/druid/pull/12263/|https://github.com/apache/druid/pull/12263/> to control the frequency of only the auto compaction run

<@U035FNDU6HH> Correct me I am wrong, Looks like Auto compaction cannot catchup old data compaction even when we have slots available because of the  druid.coordinator.period.indexingPeriod is set to 30mins. Changing this field might cause other coordinator tasks run frequently

Regarding catch up, that depends on how much incoming data you have and how much slots you have per run. Regarding the second point, the PR linked above solves that concern

The nominations for the 2022 Datanami Readers’ Choice Awards are open, and here's your chance to nominate Apache Druid for:

*Best Data and AI Product or Technology: Analytics Database*

Nominations are accepted from virtually anyone connected with the Big Data community who is a Datanami reader. Nominations will close on September 20, 2022 at 11:59 PM.

Make your voice heard – help tell the story of Big Data in 2022 by submitting your nominations for the 2022 Datanami Readers’ Choice Awards:

<https://www.surveymonkey.com/r/HSZGZD3>

from docs:

&gt; If the number of bins is specified instead of split points, the interval between the minimum and maximum values is divided into the given number of equally-spaced bins.
meaning, those 5 bins are equally-spaced from min to max: you can use the `0` and `1` quantile to get the min and max and do some math to figure out what the cut-points are for those 5 bins

it would be more convenient if this math was done for you and returned in the result; that'd be a nice enhancement to the function

example btw: if min is 0 and max is 10 and numBins is 5, then the bins are 0-2, 2-4, 4-6, 6-8, 8-10

Is there any existing function that returns the bins passing the min/max and number of bins?
Also, ideally the middle of the range can be used as the bin name something that is readily useful for plotting to a bar chart.

Thanks.

&gt; Is there any existing function that returns the bins passing the min/max and number of bins?
not that I know of — AFAIK you do need to compute it yourself (something like `binSize = (max - min) / numBins` and then `binEndPoint[i] = min + binSize * i`)

&gt; Also, ideally the middle of the range can be used as the bin name something that is readily useful for plotting to a bar chart.
i am not sure what you mean by this?

Does this thread help <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1661254230059209> ?

I don't have an answer offhand, but I'm curious: how many bytes would you _like_ to be able to fit in an individual column?

do you mean a single value in the column?

absolute hard limit for string length is 2GB since their length is stored as signed int :wink:

but you're likely gonna run into issues before that: for example, length of string dictionary in each segment is _also_ stored as signed int, and all strings for a given segment need to fit within the dictionary

so if you really had a 2GB string then the segment would need to have just a single row in it

with all storage parameters set to defaults (like segment size, etc) it's safe to have average string length be about a few hundred bytes

btw in next release we're introducing JSON columns, which can safely have much bigger values. if the reason you want to have a big value in a single column is that it's representing some JSON-like data structure, then the new JSON column feature is what you want

We're looking to have something on the order of a megabyte.  I am going to generate a test data set and see if we encounter performance bottlenecks that we need to work through.  Overall data volume here will be low compare to other datasources so I know we can adhere to the overall segment size restrictions. I'll let you all know what I learn.  Thanks for the help!

with a MB my main advice is lower your target segment size; the thing you'll run into is the total size of all strings in any given column must be &lt; 2GB/segment, so with 1MB/value, that's a max of 2000 rows. so a target of 1000–1500 would be good

<@U031F9Q5D1P>, could you please elaborate? Do you want to apply different compaction configs to different intervals of the datasource?

Hello <@U0318CX1NH4>,
For example, after 1 month, change queryGranularity and segmentGranularity to HOUR / HOUR, after 3 months change granularities to DAY  / DAY, after 6 months, remove some dimensions

Hmm, that would be interesting. No, we don't have that capability yet. You can always trigger manual compaction jobs apart from the auto-compaction.

You could also try to maintain rolled-up datasources in conjunction with the main one. Say, a datasource which ingests from the same source but always has granularity DAY or MONTH. Queries that require rolled-up data would have to be fired against that datasource. Would that be a viable option for you?

I’m currently (druid 0.22.0) using cronjobs running on k8s to deal with compaction (starting index_parallel tasks), but since we can change granularities in compaction tasks in 0.23.0, I wanted to use it

And I was just wondering if I could only use druid compaction tasks to do it.

&gt; using cronjobs running on k8s to deal with compaction (starting index_parallel tasks)
Ah, okay, so you were essentially re-indexing your data.

&gt; And I was just wondering if I could only use druid compaction tasks to do it.
Yeah, you can fire _manual compaction_ to do it in the newer versions as you can now update the granularity in compaction. But unfortunately, you would still need the cron job if you want different granularities for different intervals, because auto-compaction allows just a single config (and a single target granularity).

Let us know if you find a better solution!

Thanks a lot for the clarification!
Yep, will do!

I made this exact question a couple of months ago. I was told it was on the pipeline, but no updates for now: <https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1656524271091779>

looks like a good match based on this post: <https://imply.io/blog/an-end-to-end-streaming-analytics-stack-for-network-telemetry-data/|https://imply.io/blog/an-end-to-end-streaming-analytics-stack-for-network-telemetry-data/>

Are you based in Austin, TX? Next Thurs. 9/22, the Austin Apache Kafka user group is hosting an in-person meetup. Darin Briskman will be talking about <https://www.linkedin.com/feed/hashtag/?keywords=apachedruid&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6975845516190334976|#ApacheDruid> and Implementing Streaming Analytics at Scale. Register here: <https://www.meetup.com/austin-apache-kafka-meetup-stream-data-platform/events/288078510/>

not in the same datasource (table); altho you can always have two datasources of different granularity for the same underlying data

hmm, so the problem we are trying to solve is based on timezone

we are using HOUR Granularity(saved in GMT), which works perfectly for timezones whose  offset is exactly by the whole hour like PST ,EST etc. but that breaks when when converting to timezones whose offset is not by the whole hour like IST (+5:30), Any recommendations on that.

in a situation like that i would generally choose to index at 15m granularity across the board

(there are some other TZs with :45 offsets)

true 15m is the answer,  but that kind of explodes the data size (X4)

it sometimes doesn't, due to not all tuples appearing in all 15m intervals: i often find that going from hour to 15m increases data size by &lt;2x

if you see all tuples at all times then it would be 4x

yeah thats problem for us our data is like every minute or so

in this case it may make sense to index one datasource at hour and one at 15m, and use the 15m one only for certain timezones

ok let me check on that , thanks for the quick response

Kyle, my colleague at Imply, wrote a great series of blog posts about Laning and Prioritization.  The first [<https://medium.com/@kyle.hoondert/query-laning-in-apache-druid-80cf138806f3>] describes  the strategies, and this one describes observing (threshold) priority manipulation in action. <https://medium.com/@kyle.hoondert/observing-threshold-based-query-prioritization-in-apache-druid-7f362670b296>

One quick way: if I expect to allow 10 queries in the low lane, I've fired more than that and seen the rejection message

hi <@U031TAPB45S> thank you so much for answer my quesiton!. I have another question about laning. Instead of error out of the laning. Can we use lanes as queues? We don’t want to error out the requests coming into druid.

<@U031TAPB45S> I am trying this config on my local quick-start machine but only getting 429 errors instead of queuing up
```druid.server.http.numThreads=10
druid.query.scheduler.numThreads=10
druid.server.http.enableRequestLimit=false
druid.query.scheduler.laning.strategy=hilo
druid.query.scheduler.laning.maxLowPercent=20```

Hi <@U03FA8G5DDF> - one side-effect of enabling laning is we turn off queueing.

image.png

From <https://druid.apache.org/docs/latest/configuration/index.html#query-prioritization-and-laning>

&gt; __time can only support milli seconds
Can you `format` your `startedAt` field with `nano`?

<https://druid.apache.org/docs/latest/ingestion/ingestion-spec.html#timestampspec>

```"timestampSpec": {
  "column": "startedAt",
  "format": "nano"
}```

Screen Shot 2022-09-14 at 4.25.29 PM.png

Screen Shot 2022-09-14 at 4.25.11 PM.png

<@U030AQ8AR8E> We already mapped to nano.  In the attached screen shots you could see that nano seconds are dropped even when `startedAt` field with `nano`

Also document says _Treat `__time` as a *millisecond* timestamp: the number of milliseconds since Jan 1, 1970 at midnight UTC._  

I donot understand how the group by condition avoids sorting.

maybe if you could share both the queries and the ingestion spec we could help out a bit more.

<@U0315MQKEHK>  I meant for sorting on "__time"  "group by" is not needed. However it is mandator for sorting on any other column. If __time holds nano seconds then, sorting can be done "__time".
```If __time preserves nano seconds I can use this query which returns in 200ms

SELECT id, duration 
from View
order by __time DESC
LIMIT 100 OFFSET 0



Now I had use this which is taking ~20s
SELECT id, max(duration )
from View
GROUP BY id
order by min(startedAt) DESC
LIMIT 100 OFFSET 0```

as to the performance degradation for your new query… i guess it's because you have a GROUP BY in the second but not the first, which in turn i guess is because druid cannot sort by anything other than `__time` without a GROUP BY

there is an issue <https://github.com/apache/druid/issues/12958> about adding this feature with an accompanying PR, so stay tuned!

one suggestion i have in the meantime is to add a `__time` filter to your second query

to reduce the # of rows that must be scanned to figure out the first few `startedAt`

<@U0344FW86DD> Thanks will wait for the PR

Hey you!  Yeah.. you... Druid user!  Are you also a Presto user?  If so,  I am sure you have an excellent story to tell about what you are doing with the best data projects on the planet, Druid and Presto!  How about sharing it with the world??  The CFP is now OPEN for PrestoCon 2022!  This year, live and in-person in Mountain View.  Hit me up with any questions!! <https://events.linuxfoundation.org/prestocon/program/cfp/>

what behavior are you looking for from ARRAY_ANY?

```"ID": "id1","timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"], "duration": 10} #row1
 
Select ID, max(duration), ARRAY_ANY(tags)
Form view
Group by ID
Order by max(duration)
Limit 100

Expected return

{"ID": "id1", "duration": "10", "tags": "["t1","t2","t3"]"} 

ANY_VALUE(tags) returns

{"ID": "id1", "duration": "10", "tags": "[t1, t2, t3]"}```

would it make sense if the return was:

```{"ID": "id1", "duration": "10", "tags": ["t1","t2","t3"]} ```

(wondering why it needs to be wrapped in a string at all)

assuming `sqlStringifyArrays: false` (<https://druid.apache.org/docs/latest/querying/sql-query-context.html>)

we want exact behavior as ARRAY_AGGR  but select any value

yeah what i'm saying is would it make sense if ANY_VALUE stopped converting arrays to string? and kept it as an array

yes that would be helpful. I have set
```"sqlStringifyArrays": false.```
Still arrays are converted to strings by ANY_VALUE

yeah i think this is a limitation of ANY_VALUE itself. but we could make it work better for arrays (i.e., don't stringify them)

Thanks that would help us. Currently planning to handle with custom deserializer at application code

I think we can achieve with this too ANY_VALUE( STRING_FORMAT ( '["%s"]', MV_TO_STRING(tags, '","')), 1000 )

using different mount points can help with this

This is correct. If you are deploying in a dynamic cloud like Kubernetes, then “Overlord Dynamic Config” is your friend.

1. Define the MM per datasource in your Statefulset with custom size EBS in the PVC, tailored to your input dataset.
2. Use Overlord Dynamic Config to point to the different kind of MM.
3. Profit.

It’s not easy for me to have separate statefulsets for per datasource MMs - our kafka supervisors are rather dynamic in how we add them, and I’d rather not change deployment as part of it. We also use local SSDs for indexing instead of PVCs.. although I’m starting to question validity of that
As for dynamic overlord configuration, is there somewhere I could read more about it? The documentation I found was rather simple (and I can’t seem to find it right now anyway)

I think Didip is referring to "Worker Select Strategy" <https://druid.apache.org/docs/latest/configuration/index.html#worker-select-strategy>

Yes, for example, this is what we have:
```{
  "type": "equalDistributionWithCategorySpec",
  "workerCategorySpec": {
    "categoryMap": {
      "index_kafka": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {
          "kafka-aaa": "kafka-aaa-mm",
          "kafka-bbb": "kafka-bbb-mm",
          "kafka-ccc": "kafka-ccc-mm",
          "kafka-ddd": "kafka-ddd-mm"
        }
      },
      "kill": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {}
      },
      "compact": {
        "defaultCategory": "compaction-mm",
        "categoryAffinity": {}
      },
      "index_parallel": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {}
      },
      "single_phase_sub_task": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {}
      },
      "partial_dimension_cardinality": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {}
      },
      "partial_index_generate": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {}
      },
      "partial_index_generic_merge": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {}
      },
      "partial_dimension_distribution": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {}
      },
      "partial_range_index_generate": {
        "defaultCategory": "_default_worker_category",
        "categoryAffinity": {}
      }
    },
    "strong": true
  }
}```
Each Kafka datasource points to its own MM.
Compaction also gets its own MM.

Hi Victor, seems like a known limitation at this point. See <https://github.com/apache/druid/issues/10548>

I wonder if <https://github.com/apache/druid/pull/12720> released to version 24 would solve this issue.

I found a workaround using inner joins. Ill upgrade to 0.24 our cluster to see if this gets resolved

<@U031Q3KCEHE> I'm interested! What's the workaround?

I rewrote the query using inner join
```SELECT 
APPROX_COUNT_DISTINCT_DS_THETA(THETA_SKETCH_UNION("a1", "a2"))
FROM (
  SELECT 
    DS_THETA("ad_users", 16384) AS "a1", DS_THETA("ad_free_users", 16384) AS "a2"
  FROM (
    SELECT 
      '1' as "key",
      DS_THETA("unique_users", 16384) as "ad_users"
    FROM "unified_ad_event"
    WHERE "__time" &gt;= CURRENT_TIMESTAMP - INTERVAL '1' DAY  AND "event_name" = 'impression' AND "ad_event_type" = 'sponsored'
  ) "au" INNER JOIN (
    SELECT 
      '1' as "key",
      DS_THETA("unique_users", 16384)  as "ad_free_users"
    FROM "users"
    WHERE  "ad_free" = 'true'
  ) "afu" ON "au"."key" = "afu"."key" 
)```

Thanks for sharing. Seems like this works in your case because there is only one row in each subquery, so you don't get a cartesian product from the inner join... cool!

congrats! that came so fast! This will be the first thing I check after my vacation!

congrats to all 67 contributors :slightly_smiling_face:

You should be able to use CASE for that as in:
```SELECT CASE WHEN array_length(ids) = 0 THEN null ELSE array_to_string(ids, ',') END from view```


Can you do `explain plan` for both of them and post it here?

Fast one
```{
  "queryType": "topN",
  "dataSource": {
    "type": "table",
    "name": "View"
  },
  "virtualColumns": [],
  "dimension": {
    "type": "default",
    "dimension": "traceId",
    "outputName": "d0",
    "outputType": "STRING"
  },
  "metric": {
    "type": "numeric",
    "metric": "a1"
  },
  "threshold": 100,
  "intervals": {
    "type": "intervals",
    "intervals": [
      "2022-09-16T00:28:40.000Z/2022-09-17T00:28:40.001Z"
    ]
  },
  "filter": {
    "type": "selector",
    "dimension": "tenantId",
    "value": "47a01df9-54a0-472b-96b8-7c8f64eb7cbf",
    "extractionFn": null
  },
  "granularity": {
    "type": "all"
  },
  "aggregations": [
    {
      "type": "expression",
      "name": "a0",
      "fields": [
        "originEntityIds"
      ],
      "accumulatorIdentifier": "__acc",
      "initialValue": "ARRAY&lt;STRING&gt;[]",
      "initialCombineValue": "ARRAY&lt;STRING&gt;[]",
      "isNullUnlessAggregated": true,
      "shouldAggregateNullInputs": true,
      "shouldCombineAggregateNullInputs": false,
      "fold": "array_set_add(\"__acc\", \"originEntityIds\")",
      "combine": "array_set_add_all(\"__acc\", \"a0\")",
      "compare": null,
      "finalize": null,
      "maxSizeBytes": 10000
    },
    {
      "type": "longMax",
      "name": "a1",
      "fieldName": "duration",
      "expression": null
    }
  ],
  "postAggregations": [],
  "context": {
    "populateCache": false,
    "sqlOuterLimit": 101,
    "sqlQueryId": "a9aa2d3f-a365-4efb-bac2-2355e192a6a4",
    "sqlStringifyArrays": false,
    "useCache": false,
    "useNativeQueryExplain": true
  },
  "descending": false
}```


Slow one
``` {
  "queryType": "topN",
  "dataSource": {
    "type": "table",
    "name": "View"
  },
  "virtualColumns": [],
  "dimension": {
    "type": "default",
    "dimension": "traceId",
    "outputName": "d0",
    "outputType": "STRING"
  },
  "metric": {
    "type": "inverted",
    "metric": {
      "type": "dimension",
      "previousStop": null,
      "ordering": {
        "type": "lexicographic"
      }
    }
  },
  "threshold": 100,
  "intervals": {
    "type": "intervals",
    "intervals": [
      "2022-09-16T00:28:40.000Z/2022-09-17T00:28:40.001Z"
    ]
  },
  "filter": {
    "type": "selector",
    "dimension": "tenantId",
    "value": "47a01df9-54a0-472b-96b8-7c8f64eb7cbf",
    "extractionFn": null
  },
  "granularity": {
    "type": "all"
  },
  "aggregations": [
    {
      "type": "expression",
      "name": "a0",
      "fields": [
        "originEntityIds"
      ],
      "accumulatorIdentifier": "__acc",
      "initialValue": "ARRAY&lt;STRING&gt;[]",
      "initialCombineValue": "ARRAY&lt;STRING&gt;[]",
      "isNullUnlessAggregated": true,
      "shouldAggregateNullInputs": true,
      "shouldCombineAggregateNullInputs": false,
      "fold": "array_set_add(\"__acc\", \"originEntityIds\")",
      "combine": "array_set_add_all(\"__acc\", \"a0\")",
      "compare": null,
      "finalize": null,
      "maxSizeBytes": 10000
    },
    {
      "type": "longMax",
      "name": "a1",
      "fieldName": "duration",
      "expression": null
    }
  ],
  "postAggregations": [],
  "context": {
    "populateCache": false,
    "sqlOuterLimit": 101,
    "sqlQueryId": "b30b95ce-12b7-4653-aa5d-43868e40a451",
    "sqlStringifyArrays": false,
    "useCache": false,
    "useNativeQueryExplain": true
  },
  "descending": false
}```

it that some sort of array agg on top of originEntityIds ?

You are not doing a `group by` on a regular column right?

traceId is defined as  this in our ingestion
          {
            "type": "string",
            "name": "traceId",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },

Oh wait, I think the ARRAY_AGG is a red hearing. Could you remove it from the query and perform the same experiment. Is it still slower if sorting on `traceId`?

and then as a followup, could you also remove the limit and turn off `Auto limit` in the console, and try again then. Is it slower then?

When ARRAY_AGG is replace with ANY_VALUE the performance improves around 60%

:thinking_face: could you please put the Array_agg back in and grab the flame graph (<https://support.imply.io/hc/en-us/articles/360033747953-Profiling-Druid-queries-using-flame-graphs>) for the slow query?

also I have added additional ANY_VALUE (string ) and performance degraded. Looks like difference between queries  increasing by amount of data

yes, there is like 97% chance that the historical is where the interesting stuff is happening

druid.processing.buffer.sizeBytes=1000000000
druid.processing.numMergeBuffers=11
druid.processing.numThreads=11

I don't have an answer regarding the naming, but I know that `nativeQueryId` is in the repo.

top-level union all (<https://druid.apache.org/docs/latest/querying/sql.html#top-level>) works by issuing multiple native queries and concatenating the result. this is the only case where multiple native queries are used

at some point, we'll change this to use a single native query, so it will become always 1–1

wow thanks for the detailed explanation. it really helped a lot!!

never; Druid doesn't use kafka consumer groups

Druid uses kafka's low-level consumer API &amp; tracks committed offsets in its own metadata storage

we do this so we can commit offsets along with segments for exactly-once guarantees

Thanks, in that case kafka uses its own log retention configuration to rotate data in topics I assume?

And when does druid update offsets in metadata storage? Only after segment has been offloaded to deep storage, or when data is persisted on disk locally?

&gt; if it is an actual clone or if there is shuffling going around.
It's an <https://druid.apache.org/docs/latest/data-management/update.html#reindex|overwrite of existing data>.

When you say

&gt; clone of an interval of segments in a new datasource
maybe `REPLACE` could <https://druid.apache.org/docs/latest/multi-stage-query/reference.html#replace-specific-time-ranges|work>?

```REPLACE INTO &lt;target table&gt;
OVERWRITE WHERE __time &gt;= TIMESTAMP '&lt;lower bound&gt;' AND __time &lt; TIMESTAMP '&lt;upper bound&gt;'
&lt; SELECT query &gt;
PARTITIONED BY &lt;time granularity&gt;
[ CLUSTERED BY &lt;column list&gt; ]```

That looks exactly like a reindexing job, which is what I’m doing. I mean creating an exact replica of a set of segments into a different datasource. Same segments, copy pasted into another datasource.

reindexing from Druid into a new datasource should be the exact data for that interval and do what you are looking for to look at compaction.  You could even reindex the intervals with the compaction changes in the spec to see the results I would think

what I meant to say is compaction is basically reindexing so you could see the results of changes via a druid reindex into a new datasource

<@U03HJDC4398> there isn't an API for doing what you ask, although you doing do it with metadata store surgery. as in: copy the records, edit the metadata to use the same `loadSpec` but a different datasource

you'd need to change the datasource + segment id in the metadata store record itself &amp; in the `payload`

this is fine for testing, although i wouldn't recommend it in production since it's easy to mess things up by accident :slightly_smiling_face:

not just with the initial surgery, but also later on, since the two segment records share the same file on deep storage it's — for example — easy to accidentally delete both when you meant to just delete one

Virtual Druid Summit CFP.jpeg

EXCITING announcement: The annual virtual #DruidSummit will be held December 6-7!

Interested in speaking about #ApacheDruid? Then visit the website and head to the Call for Papers form today (the deadline for submissions is Sept 30th)!

<https://druidsummit.org/druid-summit-virtual-2022/>

oh never mind, just found this <https://lists.apache.org/thread/xcfjoq696hn5w85hdgpxfb9hx4nsj4h1>

hmm we should update the release notes!

and also this page, which says "we are still in major version 0": <https://druid.apache.org/docs/latest/development/versioning.html|https://druid.apache.org/docs/latest/development/versioning.html>

i'm thinking something like this:

&gt; Druid is now at major version 24, a jump up from the prior 0.23.0 release. However, in practice, not much has changed with our versioning policy! We intend to increment the major version as needed, and call out any incompatible changes in the release notes. We also intend to minimize the number of incompatible changes, especially in widely-used areas such as the query API. This is the same policy as we have used in prior releases, with each digit promoted one place to the left.

Oh damn typo alert! are you making that change or should someone else do that PR?

shall I add this to the release notes?
&gt; Starting this release, we are dropping the leading `0` from the release version. So instead of calling the release `0.24.0`, we are calling this release `24.0.0`. In terms of backward-compatibility or breaking changes, this release is no significantly different than other previous major releases such as `0.23.0` or `0.22.0` .  Please refer to upgrading to 24.0.0 section (link to insert) for known backward-incompatible changes. 

I am thinking of sending a PSA email as well to dev community and druid forums

should have added some notes to the release announcement email.

It would make sense at the very top though

I've updated the release notes with an amalgam of the two suggestions

&gt; Starting with this release, we have dropped the leading 0 from the release version and promoted all other digits one place to the left. Druid is now at major version 24, a jump up from the prior 0.23.0 release. In terms of backward-compatibility or breaking changes, this release is not significantly different than other previous major releases such as 0.23.0 or 0.22.0. We are continuing with the same policy as we have used in prior releases: minimizing the number of changes that require special attention when upgrading, and calling out any that do exist in the release notes. For this release, please refer to the Upgrading to 24.0.0 section for a list of backward-incompatible changes in this release.

feel free to update again if desired :slightly_smiling_face:

I haven't updated `versioning.md` though — does anyone else have a chance to do that?

we have a section for "upgrading to 24.0" at the end of the release notes: <https://github.com/apache/druid/releases#24.0.0-upgrading-to-24.0>

I just upgraded 1 of my non-prod cluster. It seems fine when using the existing config from 0.23.0.

I haven't yet enabled to fancy multi stage query.

I made a video going through the new Apache Druid 24 quickstart (which has been revamped to use MSQ): <https://www.youtube.com/watch?v=PZpu8hOfRJw>

<https://unix.stackexchange.com/questions/23825/bashrc-doesnt-get-executed>

Maybe use .profile or .bash_profile or .bash_login instead?

Last call: you can still attend Project Shapeshift Milestone 2 *today at 9:30am PT*!

We know your time is precious, but we've packed everything into a 30-minute presentation. You’ll be the first to see the new Multi-stage Query Engine available for all Druid users! Register <https://imply.io/project-shapeshift/milestone-2-event?utm_source=email&amp;utm_medium=email&amp;mkt_tok=OTEwLU9UTi0yMjMAAAGG9cGmOKsqvc5sR3uPGD__dD9sY8VODotsTF9attoQzfu3sUHNT5sRil3e7OU85YWEDbLQHvKX0nq1b9yqOxVSH_57TpzVCZwSVaESheXk49M|here>.

I scrubbed through the code and don't see something that would be injecting `<http://repo.maven.apache.org|repo.maven.apache.org>` without you asking for it: it isn't even in our default list (that's just `<http://repo1.maven.org|repo1.maven.org>`)

Is it possible that the repo you have provided is sending you to <http://repo.maven.apache.org|repo.maven.apache.org> for certain artifacts?

Here you can see its first downloading from our repo then later trying from repo.maven...

Hi Druid,

We are trying to use the pull-deps tool in our
DockerFile

<https://druid.apache.org/docs/latest/operations/pull-deps.html>

 Seems like --no-default-remote-repositories
is being ignored or rather defaults are being tried "after" those
specified with -r optionStep 12/26 : RUN java -Dlog4j2.level=DEBUG
--add-opens "java.base/java.lang=ALL-UNNAMED" -classpath
"lib/*" org.apache.druid.cli.Main tools pull-deps
--no-default-remote-repositories -r <https://XXX> -h
"org.apache.hadoop:hadoop-client:3.3.3" &amp;&amp; mkdir -p
/opt/druid/extensions/druid-hdfs-storage &amp;&amp; cp
/opt/druid/hadoop-dependencies/hadoop-client/3.3.3/*
/opt/druid/extensions/druid-hdfs-storageResolved artifact org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:pom:1.1.1
from  (<https://XXX>, releases+snapshots)

Downloading artifact org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:pom:1.1.1
from central (<http://repo.maven.apache.org/maven2>, releases)

13:45:46.251 [main] DEBUG
org.eclipse.aether.internal.impl.DefaultRepositoryConnectorProvider - Using
connector AetherRepositoryConnector with priority 3.4028235E38 for <http://repo.maven.apache.org/maven2>

Downloading: <http://repo.maven.apache.org/maven2/org/apache/hadoop/thirdparty/hadoop-shaded-protobuf_3_7/1.1.1/hadoop-shaded-protobuf_3_7-1.1.1.pom>



    at
java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)

    at
java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)

    at
java.base/java.net.Socket.connect(Socket.java:633)

    at
com.squareup.okhttp.Connection.connect(Connection.java:100)

    at
com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:287)

Atleast it is clear that the calls are going to repo.maven..even after specifying the -r option.
So this looks to be issue with DefaultRepositoryConnectorProvider being using internally by pull-deps.

<https://github.com/apache/druid/issues/383>

<@U0344FW86DD>: i see you had raised this issue but its still open with tesla-aether.

Hmm.. the comment on the issue suggests that it's following repositories specified in the poms of the artifacts when looking for transitive dependencies 