Apache Druid

druid nested columns come pretty close to what you’re looking for i think <https://druid.apache.org/docs/latest/querying/nested-columns.html>

the missing piece might be an aggregate function e.g. something like `json_object_agg`, if you want ingestion to build the nested structures instead of just treat them like dimensions

i intend to add something like that function someday, but haven’t got to it yet

it could be modeled at query time with the native “expression” aggregator (which im not sure is documented, but used internally for some sql aggregations), though this hasn’t been wired up to work at ingestion time yet

the column itself is a bit bloated currently probably for your use case since it currently stores a ‘raw’ copy of the data in json form, but i have plans to allow customization of that as well to support cases where the raw data isn’t necessary, as well as control over which columns are indexed, and to allow exclusions, etc

Hi <@U030C8H59T8>,
Yes indeed very close to json type. At the beginning I thought it’s supported and tried to use json type for this map, but there are no aggregator as you mention

what is needed to implement json_object_agg ?

the cheapest way would probably be the expression aggregator, im in the process of updating some nested column stuff to have better array support and part of the refactoring should allow me to wire up the expression aggregator to work at ingestion time (i chose not to because we didn’t have support for array columns, but we soon will via the nested column stuff :sweat_smile: )

let me see if i can find something like docs for it

<https://github.com/apache/druid/pull/11104> is the original PR, but the description is a bit stale

<https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/query/aggregation/ExpressionLambdaAggregatorFactory.java#L103> for current set of parameters it accepts

for example, `ARRAY_AGG` is currently only implemented as an expression aggregator <https://github.com/apache/druid/blob/master/sql/src/main/java/org/apache/druid/sql/calcite/aggregation/builtin/ArraySqlAggregator.java#L123>

i haven’t tried the json functions in an expression aggregator yet, but i think the idea would be something like define an initial value with json_object expression, then use json_value to extract things from the object, do stuff to them, make a new object with combined values

I like your first sentence on the original PR….exactly same thoughts as I had when tried to use js aggregator

thanks a lot for the direction! I’ll try to hack something today

then all that is left is wiring up to ingest time, but some of the changes im doing for arrays might take care of that for you

heh, js aggregator and functions are security nightmare :sweat_smile:

the refactor that will make expression aggregator able to work at ingest time is hinted at in the follow-up work on this PR, <https://github.com/apache/druid/pull/13653>, so hopefully i will get to it soon

im sort of turning nested columns into a universal column, but they will keep working like they do right now too

hence the using it for schemaless ingest and adding better array support, etc

im especially excited for array typed columns, i’ve been planning and laying foundations for this for years now heh

anyway, getting off track here, feel free to ping me if you want to try to add json_object_agg or something like it as an expression aggregator that works with nested columns

with expression based aggregator version of it in place to get basic functionality added, then we can think about optimizing it with a dedicated native aggregator if performance isn’t good enough or whatever

I’m trying expression based aggregator and I came to some deadend because of json_query or json_value doesn’t permit define path by variable
in my case I need to move over keys (k) and use it to access json object with json_value(__acc, ‘$.‘||k)
but ‘$.‘||k is not literal…so I’m getting
```Error: Unknown exception
Function[json_query] second argument [('$.' || "k")] must be a literal [STRING] value
org.apache.druid.math.expr.ExpressionValidationException```
```{
      "type": "expression",
      "name": "country_id_agg",
      "fields": ["countryId"],
      "initialValue": "json_object()",
      "fold": "json_object( map( (k) -&gt; if( k==cast(countryId, string) , array(k, json_query(__acc, '$.'||k) + 1) , array(k,json_query(__acc, '$.'||k)) ), json_keys(__acc, '$') ) )",
      "combine": "json_object(map((k) -&gt; json_query(__acc, '$.'||k) + json_query(country_id_agg, '$.'||k), array_set_add_all(json_keys(__acc, '$'), json_keys(country_id_agg, '$'))))"
    }```

also tried with arrays(combine function is not yet well defined):
```{
  "type": "expression",
  "name": "country_id_agg",
  "fields": ["countryId"],
  "initialValue": "[]",
  "fold": "array_concat(fold( (kv1, kv2) -&gt; json_object('k', json_value(kv1, '$.k'), 'v', json_value(kv1, '$.v') + json_value(kv2, '$.v') ),     filter((kv) -&gt; json_value(kv, '$.k') == countryId, __acc),       json_object('k', countryId, 'v', 1)      ) ,      filter((kv) -&gt; json_value(kv, '$.k') != countryId, __acc)      )",
  "combine": "array_concat(__acc, country_id_agg)"
}```
however it fails on
```Error: Unknown exception

Cannot find strategy for type [COMPLEX&lt;json&gt;]

org.apache.druid.java.util.common.IAE```

ah yeah, i guess i restricted to only literals for the path expressions; the
```Cannot find strategy for type [COMPLEX&lt;json&gt;]```
message seems a bit strange, what druid version?

oh, i think missing an expression to set a value of a key of an existing json object to some value that will make building json_object_agg hard

i started meanwhile to implement native extension(took as example FixedBucketHistogram)

were a couple of bugs preventing array_agg working with nested types, <https://github.com/apache/druid/pull/13781>

will look into if anything besides an additional native expression would be needed for expression based json_object_agg next downtime i have

implementing native stuff would likely be faster than an expression aggregator, so is still worth exploring

yep, I have already basic code for native aggregation, now exploring how to support sql aggregator

look for `SqlAggregator` and how extensions wire them in

`FixedBucketsHistogramQuantileSqlAggregator` is the fixed bucket histogram one

<@U030C8H59T8> can you explain to me please at high level what are requirements for the aggregator to support rollup as well? (maybe it’s already) I just don’t understand how it’s connected

the aggregator does the aggregating during rollup; native ingest with rollup is like grouping on all of the ‘dimensions’; every row in the same query granularity bucket with the same tuple of dimension values will have all of the aggregators defined in the spec to accumulate stuff

its basically the same at query time with different mechanisms

aggregators don’t have to support being ingested and stored as a column and can be strictly query time

for complex types, typically you implement both an aggregate method that can process the raw inputs to build into the complex type (often explicitly defined as a “build” aggregator) and an aggregate method that can handle merging the complex values, for combining the aggregated complex values across segments (often explicitly called “merge” aggregators)

if the aggregator supports ingestion, it can strictly use the ‘merge’ style of aggregation at query time

ok
I have something we can talk about if you have time
It’s a bit different, and solves rather specific problem, and not some general capability of aggregate all kinds of JSON complex types…

<https://github.com/IgorBerman/druid-complexaggs-extension/blob/main/src/test/java/org/apache/druid/query/aggregation/complexaggs/aggregator/sql/LongFrequencySqlAggregatorTest.java|https://github.com/IgorBerman/druid-complexaggs-extension/blob/main/src/test/java/[…]/complexaggs/aggregator/sql/LongFrequencySqlAggregatorTest.java>

basic tests work, both for native aggregator and sql operator

I’d appreciate if you can clarify following :
1. how to finalize to JSON type(currently I’m using ugly json mapper in finalizer to serialize aggregated result to json-string)
2. I assume that I’m starting from regular field(currently only long) and then I count it’s occurrences. My question is - if I will enable rollup, the rollup-ed partial aggregation will be saved in intermediate format? I think I need more tests around it, I’d appreciate any pointers(I couldn’t find something that uses rollup(true) in tests)

will try to have a look soon, i need to think a bit about if you want to write out an actual nested column from an aggregator because that will be somewhat tricky, and will probably involve writing it out in two passes, the first to “serialize” it through an indexer to index the column values and then the 2nd to actually write out the nested column

<https://github.com/apache/druid/issues/12695> has details on the inner workings of nested columns

if you don’t need all of that stuff at query time (fast filtering, optimized value selection from the nested data, etc) then you have other options

lots of complex columns use a generic serializer that just stores a column of whatever variable sized blobs the object strategy spits out

if you want to do something like that, but want compress the blobs, you could re-use some of the parts of the nested column serializer (the `CompressedVariableSizedBlobColumnSerializer` and corresponding `CompressedVariableSizedBlobColumnSupplier`)

btw, if you do not use the nested column serializer directly, i don’t think you can call the type `COMPLEX&lt;json&gt;` since that would conflict with the real nested columns

anyway, i’ll try to have a look at your stuff as soon as i have some free time

oh and a bit of a heads up, the nested column stuff is under very active development, so stuff is still changing internally pretty frequently

thanks <@U030C8H59T8> for the pointers and additional info.
Will try to use compressed blobs, I think I don’t need anything besides imo
At basic level I need just ability to aggregate(both at ingestion and at query time), store partial aggregation and then access results of aggregation by subset of dimensions(when doing timeseries or group-by queries)
Yes, I have mixed things a bit, will remove `Complex&lt;Json&gt;` reference.

ok, just to update I have working version (tested on 24.0.1 druid)
meanwhile only counts long values
serialization format is array of [key,value,key2,value2]

In Druid 25 (the latest release) you can start druid vis a the `bin/start-druid` script and it will automatically determine the best configs given the available resources on the node.

Hi All, if you want to find out about new features in Apache Druid, join us today at 12PM PST. Today we'll review Apache Druid 24.0 and 25.0 and open up the floor to discuss a  feature proposal: Schema Flexibility or Really-really-schemaless Ingestion, (we haven't quite landed on a name). We want to hear what your opinions are on how it should work. Sign up here: <https://www.meetup.com/druidio/events/291307729/>

In our set up we have 8gb ram , giving mm process (128mb heap and 1gb direct memory )and spinning 5 peon tasks with ( 512mb heap and 4gb direct memory each) how does it work .
does mm not use full direct memory at a time?

I’d suggest that you review <https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html#direct-memory-sizing> . You will have Direct Memory for both Historicals and MMs

Direct memory is not pre-allocated and is used when peons are created for ingestion /compaction

also note that in Druid 25 (the latest release) you can start druid via the `bin/start-druid` script and it will automatically determine reasonable configs given the available resources on the server

I'd vote for upgrading over tuning.  Lots of better functionalities, bug fixes, etc.  Standard advice of reading release notes for 22, 23, 24, 25, to check for incompatible changes, etc.

Thanks <@U0411DE2SS0>
Can we use same ZK and megastore when upgrading the cluster from 0.21.0 version to 25.0?
I mean they are completely backward compatible versions right?

always run latest, I always say :slightly_smiling_face:

there's tons of improvements with each release

<https://www.meetup.com/druidio/events/290102358/|Join us for another Apache Druid Office Hours today at 10AM PST>.
Whether you're new to Druid, evaluating it, or already in production, come along and say hello in this virtual get-together! Whether you have questions about how to use Druid, want to discuss Druid data pipelines and use cases, or just want to hear what other people are doing in the community, come along and say hi!
These are casual, discussion-style meetups that are not recorded.

You should reach out to your Imply contacts, but IIRC they are configured and secured separately (they can both talk to the same LDAP if that is what you want). If your Druid is secured with LDAP you would still need to have a service account for imply-ui. To make sense of the system you have to remember that it was designed with totally different users in mind. imply-ui =&gt; a tool for everyone to use. Druid's APIs =&gt; for Druid admins and ui tools to access

Hi <@U030G8YG9GD>, My 2 cents.
Prometheus is common for monitoring other platforms so many Druid implementations use it because they already use it for monitoring other infrastructure.
You are correct in that you can emit metrics to a kafka topic and do real-time ingestion into a Druid cluster for real-time analytics on Druid itself.
Another thought, that may be obvious, don't put it on the same Druid cluster that you are monitoring because you don't want its workload to interfere with your regular production workloads.

Agreed, I’m fact we can emit simultaneously to both “metrics” Druid and the monitoring solution.
There are a few reasons to have a metrics druid (have a test env; use SQL to analyze; compare different clusters). 
I hope to see here some reasons for a monitoring solution in addition to being a standard.

At Imply we do both approaches: we have a Druid-monitoring-Druid setup (using Pivot, our own UI) and we also feed Druid logs and metrics into a relatively standard monitoring product

As others have said, the advantage of a standard monitoring product is you're likely to have a bunch of other stuff monitored there too.

The advantage of Druid-monitoring-Druid is that many of Druid "metrics" are actually more like structured logs; especially the `query/time` and other query metrics. We find that using the Pivot UI on a Druid backend provides a better slice and dice experience for these sorts of "loggy metrics" or "metricy logs" than standard monitoring products.

I raised this concern a while ago, hopefully if anyone wishes to actively maintain the project then he will also work on forking it to part of the Apache Druid project.

As an immediate concern, I’m sure if you have a pending PR you can just ping a few of the org committers.

Some of us here can help out; feel free to post in <#C030CMF6B70|> with things that need pushing. And I do agree with Eyal. Ideally for long term health of the operator one of these should occur

1) Someone that wants to maintain it forks it, and we update the readme on the current repo to point to the new fork

2) Someone that wants to maintain it does the legwork to get it donated to ASF, so it can become an official part of Apache Druid. Most of the legwork here is IP Clearance: <https://incubator.apache.org/ip-clearance/>

```How it works

The K8s extension takes the podSpec of your Overlord pod and creates a kubernetes job from this podSpec.```
Does it work if Im using coordinator as overlord? Or do I need to explicitly have both defined so `Overlord` pods can be ‘clone’ into ingestion tasks?

The peons will have the same Xmx/MaxDirectMemory values as Overlord? The same memory limit/request?

• It currently does not work on K8s 1.25+ but there is an active fix being written.  
• It currently does not support TLSv1.3 
• It seems to work well otherwise
• I don't know about the overlord/coordinator combined pod
• Yes it will take *some* specs from the overlord pod such as service account, resources most likely, etc. 

I've been experimenting with it, it works well but I will have to modify the resource requests / limit by forking druid itself. As it is, it uses the javaOpts array to guess at memory and hard codes cpu to 1. And I had a problem with the max task count once that I had to restart the coordinator for but I'm not sure it is related to mmless.

```   static long getContainerMemory(PeonCommandContext context)
  {
    List&lt;String&gt; javaOpts = context.getJavaOpts();
    Optional&lt;Long&gt; optionalXmx = getJavaOptValueBytes("-Xmx", javaOpts);
    long heapSize = HumanReadableBytes.parse("1g");
    if (optionalXmx.isPresent()) {
      heapSize = optionalXmx.get();
    }
    Optional&lt;Long&gt; optionalDbb = getJavaOptValueBytes("-XX:MaxDirectMemorySize=", javaOpts);
    long dbbSize = heapSize;
    if (optionalDbb.isPresent()) {
      dbbSize = optionalDbb.get();
    }
    return (long) ((dbbSize + heapSize) * 1.2);

  }

  protected Container setupMainContainer(PodSpec podSpec, PeonCommandContext context, long containerSize, String taskContents)
  {
(...)

    mainContainer.setName("main");
    ImmutableMap&lt;String, Quantity&gt; resources = ImmutableMap.of(
        "cpu",
        new Quantity("1000", "m"),
        "memory",
        new Quantity(String.valueOf(containerSize))
    );
    mainContainer.setResources(new ResourceRequirementsBuilder().withRequests(resources).withLimits(resources).build());
    return mainContainer;
  }```
Interesting so for the pod requests and limits:
• CPU: always 1
• Memory: (Xmx + MaxDirectMemorySize) + 20%

we use it everywhere, but we also wrote it :slightly_smiling_face:

so you’re using for actual production jobs?

yeah i have a few patches on top which I haven’t upstreamed yet.  I will do that this week in one big PR

<@U04L6JH4H2M> yeah we use it for 90% of our druid clusters, one really nice thing about the patch, is we are constantly rolling our druid to upgrade it and the tasks don’t die :slightly_smiling_face:

like they do with the mm-version of druid, they just resume when the overlord starts back up because they are decoupled from druid

for deployments, druid-operator is the way or is there a better way?

we use the druid-operator, its not very well maintained right now, which is a problem.

I want to discuss maybe forking it or taking over responsibility

It works sufficiently, I think the advent of mmless druid may accelerate it's use.

just exclude the mm in the spec for the operator and you are good

does it copy tolerations, affinities, topologySpreadConstraints etc from Overlord too?

yes, right now it tries to make it as easy as possible

one idea i had for a follow-up was the user could specify a template in a configmap or something, then the overlord could use that to launch peon tasks for the more power-user

but right now, the goal was to make it as easy as possible for the everyday druid user, but i do think i will need something more….

i don’t know what that is, also if you have an idea, please by all means make commits :slightly_smiling_face:

&gt; one idea i had for a follow-up was the user could specify a template in a configmap or something, then the overlord could use that to launch peon tasks for the more power-user
yeah it would be great, eventually it’d be possible to configure peons to run in spot instances with ssd disks.

so for us, our team has a way to specify spot instances based on annotations, so those can be overriden

but again, this patch was written with only my view point in mind, and I would love to hear what others want or think could be improved.

but that means your overlords are also running in spot, right?

nope, you can add additional annotations and labels for just your peon tasks

to add annotations and labels just to the peon k8s tasks

<https://druid.apache.org/docs/latest/development/extensions-contrib/k8s-jobs.html>

```druid.indexer.runner.labels
druid.indexer.runner.annotations```

those let you add additional annotations and labels to just your peon k8s jobs

but again, please let me know if you have other ideas, i just wrote it for usecases i encountered, i would love to hear what other people think / want.

also feel free to put up PRs the more people contributing will make this a better feature

dude, this is awesome, mm-less is game changer getting clusters cheaper, less things to manage

If you could specify objects in the config you could probably get pretty far by just allowing top level pod blocks in druid config, eg. affinity, env, envFrom, tolerations, limits, etc.

are the peons in mm-less capable to emit metrics?

yes, i have an internal patch for that, ill push that up this week!

before it was inheriting from the parent, i just added an internal patch to allow you to specify metrics you want to push

the 3 main things i have patched internally are,
1. metrics
2. upgrade the client to work with newer versions of k8s
3. better sidecar support

there might be more smaller things, but i will try to get a patch up this week with all my changes in a single PR so its easy for people top cherry-pick

<@U04NHCE780G> - what PRs are pending to be reviewed? You can also post the PR links on <#C030CMF6B70|> channel if they are not getting the attention from druid committers.

so we can expect these changes to a 25.0.1 or only to 26?

We haven't started the process yet. April seems like the right time

Is there any way I could convince you to release a `25.0.1`?  Things are currently broken for a standard K8s release that has existed for 6 months

i have one more patch for the k8s mm-less stuff

fyi: if you want all our internal patches here they are: <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1676401570977559>

This looks so awesome. But I have a question if you don’t mind. With this setup, how do you get the task’s logs? Are the logs still viewable from the tasks UI?

This is super! It will definitely lower the costs of running Druid for sure.

The UI just gives you k8s logs as if you were tailing the pod.

the only thing you don’t get streaming is the reports, you get those at the end if you set the `enableTaskLogPush` flag….

i didn’t want to go into the pod and start copying out files :slightly_smiling_face: also reports are not important enough during task runs to deal with that

yeah, report is the least important thing :joy:

one other thing <@U04MRT25PSP> helped find a bug with mm-less and 1.25, I will put up a patch for this but long story short

at the end of a task, the overlord launches a task, waits for it to complete and then returns the status.  Now in k8s &lt; 1.25 it would just mean once the pod is not active, go grab the status.

Now the contact has changed a bit, because in 1.25 they introduced finalizers:
<https://github.com/kubernetes/kubernetes/pull/110948>

And we noticed that when we ran tasks on jason’s druid cluster they would complete fine but marked as failure.
We outputted the job status and noticed that the job was not active, but both success and failure were not set, but there was this field that had

```uncountedTerminatedPods=UncountedTerminatedPods(failed=[], succeeded=[e916cbf9-467a-45f3-86a7-3767145d6384], additionalProperties={})```
which from the docs:
```UncountedTerminatedPods holds the UIDs of Pods that have terminated but the job controller hasn't yet accounted for in the status counters. The job controller creates pods with a finalizer. When a pod terminates (succeeded or failed), the controller does three steps to account for it in the job status: (1) Add the pod UID to the arrays in this field. (2) Remove the pod finalizer. (3) Remove the pod UID from the arrays while increasing the corresponding counter. This field is beta-level. The job controller only makes use of this field when the feature gate JobTrackingWithFinalizers is enabled (enabled by default). Old jobs might not be tracked using this field, in which case the field remains null.```
So now what happens is the job goes from a state where it is not active, to having uncountedTerminatedPods to then having a status with success or failure.  I will push up a one-line fix to make this work, but for those of you working with `1.25` version of k8s, I’m sure you will be affected as well.

Basically add another check to wait on,
Right now we wait for this:

```// block until
job.getStatus() != null &amp;&amp; job.getActive() == null
// then return 
return job.getStatus().getSucceeded() != null```
Now the change will be
```// block until
job.getStatus() != null &amp;&amp; job.getActive() == null &amp;&amp; (job.getStatus().getFailed() != null || job.getStatus().getSucceeded() !=null)
// then return 
return job.getStatus().getSucceeded() != null```
This should keep things backwards compatible and working in all versions of k8s

Can you paste your resulting ingestion spec?

You mean it shows as INT in the console or somewhere?  If so, it's probably because the schema is being infested from older existing segments.  If so, you can reindex (in the console) and change to double.  If they're no old segments left, you might need to restart coordinators (I think), so they can pick up the new schema.

Hey Ben thank you so much for your reply and really sorry for my late reply , a lot was going on recently :confused:

yes, it was showing in the console in the UI. what do you exactly mean by reindexing actually? so I just submitted another supervisor which is reading from the same kafka topic (the one I added more columns) and referring to same data source in Druid. In this case, after I submit it, what will happen to previous segments which have no new columns? Is Druid updating the data in previous segments with the new columns added after I submit it? or will I have 2 segment versions in that data source and thats why I was not able to see newly added column's type as double because the console was showing me previous segment's column type?

I would like to run my changes in Production soon, and dont know what to do with segments which we already have currently. I dont want to delete data and re-ingest again as well because it is production. Thats why I wonder how in best way I can deploy my changes again to Prod.

If I just re-ingest , those v1 old segments gonna remain there in data source? how to get rid of those?

It was showing me finally "double" btw, after I drop and kill the segments from the data source and re-submit a new ingestion. That is what I would not like to apply for Prod tbh.

The old segments to not NEED to be reingested; new columns will have a null value in the prior timeframe if you do not reingest them. But also, as Ben mentioned, there will be the data type side effect if you do not reingest while the older segments are still present.

<@U04Q25Z05U0> the way it works with mixed types is that the _latest_ (highest-timestamped) data for any given column is what determines what we consider the type to be

So if you have some doubles from Jan 1 and some longs from Jan 2 then it will be long (BIGINT). If you have longs from Jan 1 and doubles from Jan 2 it will be DOUBLE.

the important timestamp is the one in the data itself (`__time`) btw. not the time you did the ingestion

If you see something different from this, let us know so we can help you figure out what is going on

Capture d’écran 2023-02-14 à 16.09.21.png

I’ve tried something like this, but historicals within `report` tier don’t load data

This should be possible and that looks correct for the load rules.  Can you check the coordinator logs as well as the report tier logs?

Actually, I wanted to try to run historicals with `druid.segmentCache.lazyLoadOnStart` enabled

And in Coordinator logs, I can see this :
```2023-02-14T15:15:14,911 WARN [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - [report]: insufficient active servers. Cannot balance.```


I have a single historical in this tier and 1 replica in load rule

```2023-02-14T15:26:12,622 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - Balancing segments in tier [_default_tier]
2023-02-14T15:26:12,622 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - Found 6 active servers, 0 decommissioning servers
2023-02-14T15:26:12,622 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - Processing 700 segments for moving from decommissioning servers
2023-02-14T15:26:12,622 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - Processing 1000 segments for balancing between active servers
2023-02-14T15:26:14,715 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - [_default_tier]: Segments Moved: [7] Segments Let Alone: [993]
2023-02-14T15:26:14,715 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - Balancing segments in tier [report]
2023-02-14T15:26:14,715 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - Found 1 active servers, 0 decommissioning servers
2023-02-14T15:26:14,715 WARN [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - [report]: insufficient active servers. Cannot balance.```


I’ll add another Historical in that new tier

After adding another historical in `report tier` :
```2023-02-14T15:31:15,608 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - Balancing segments in tier [report]
2023-02-14T15:31:15,608 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - Found 2 active servers, 0 decommissioning servers
2023-02-14T15:31:15,608 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - No segments found. Cannot balance.```


can you see if making it the same as default_tier load rule loads the 3 months?

Capture d’écran 2023-02-14 à 17.48.41.png

When I’m using this rule, it works but I loose the capacity to load a different period per tier

that verifies everything works though.  Try putting the 1 y cold tier back to see if it now works.  It should work as others have set up a similar config.  What version of druid are you on?

Historicals are pulling and dropping segments, not sure if it goes the right way :stuck_out_tongue:

what were your load rules originally, did you have a year then decreased to 3 months?

Originally, I had `default_tier` with a P3M + future on this datasource

I’m wondering if the load rules order matters here

you may need to mark the segments as used that are past 3 months too?

It does not even load the segments in the 1st load rule

<https://druid.apache.org/docs/latest/operations/rule-configuration.html#reload-dropped-data>

Capture d’écran 2023-02-14 à 18.35.02.png

I forgot to mention that I’m trying to achieve this on a small period before trying on P1Y.
I don’t want to edit my original load rule of P3M on default tier. I want to add a new load rule on `report` tier that has a P1M retention period

I’ve changed the order of the load rules : P1M first then P3M then drop and it seems to work

hmmm ok.  I would focus on the log message that says no segments found when you have the report tier added with the longer duration

I’ve waited for Druid to finish dropping/loading segments, and it seems that I don’t get what I’ve expected.
The P1M segments are located on `report` tier only.

I’d like to also have those P1M segments loaded on `default_tier` as well

Motivation for the above is to address certain CVE's that are fixed in 1.32.0

You have tagged the wrong Abhishek :slightly_smiling_face:

We do have plans to upgrade Calcite early this year. What's unknown is whether we can do it without shading calcite dependencies.

My apologies!

This is the CVE of concern: *CVE-2022-39135*

We analyzed this CVE before and came to conclusion that druid is not affected by it.

do the results of those analysis get posted somewhere in the repo as part of an exception on CVE checks?

I see this
```owasp-dependency-check-suppressions.xml
678:    &lt;cve&gt;CVE-2022-39135&lt;/cve&gt;
702:    &lt;cve&gt;CVE-2022-39135&lt;/cve&gt;
713:    &lt;cve&gt;CVE-2022-39135&lt;/cve&gt;```

yeah Ive seen this file, ~but the notes there are not really descriptive for why its suppressed~

<https://github.com/apache/druid/pull/13119>

It does say - "We do not expose any of the SQL operators that were found vulnerable in this CVE." what additional info would be useful there?

<@U031A9D19NU> There was another one that had to with jackson-databind `CVE-2022-42003`, I see that also was in the supression list, is there documentation somewhere for this one?

This is _not_ multiple JSON, it is a single JSON Array.

You need a response with a single JSON per line and it’ll load as expected.

there are two ways of approaching this. If these jsons arrays are as small as above you could ingest as a nested column <https://druid.apache.org/docs/latest/querying/nested-columns.html>. The other option is to use nifi to split a json. I did a write up using the github api which returns json arrays <https://dev.to/vnarayaj/analysing-github-stars-extracting-and-analyzing-data-from-github-using-apache-nifir-apache-kafkar-and-apache-druidr-280>

If the dataset is not static and you’re not already using NiFi + Kafka to follow the previous suggestion, I’d add a super slim webservice only as an adapter: calling it triggers a call to the actual API (prob with some auth involved) and converts the result from a json array into a newline delimited json. Finito.

If it’s a static dataset (or you can afford to manually do it every now and then) you could just download the thing, convert into newline delimited json and load it from file.

AFAIK Druid is not a good choice for this. It’s a timeseries database, it doesn’t have a fulltext engine.

in my experience row counts matter a bit more than overall sizes, since sizes are rather dependent on schema

Thanks, keeping row count under control is much easier :slightly_smiling_face:

I think you’ll find a believer in <@U030K4UM3H7> :smile:

cc <@U030W50CNAC> who, I think, has the superpower to create a channel

That is a great idea! I created <#C04Q0047B4M|> and added everyone in this thread to it

Usually, build or use something on a separate web-facing server that's allowed to connect to the cluster.  (Like using superset iiuc, or something like Imply's Pivot, etc.)

I believe there is a `maxNumConcurrentSubTasks` parameter for compact tasks that you can use for this

I think that lets me compact in parallel using multiple compaction tasks, but it probably won’t help when the node is destroyed and recreated, deleting all local data (even if they could survive a simple restart)

There is an audit log table in the metadata store

It tracks changes to supervisor configs, retention rules, compaction configs, etc.

This is briefly talked about in <https://druid.apache.org/docs/latest/dependencies/metadata-storage.html#audit-table>

APIs to retrieve some audit entries are defined in <https://druid.apache.org/docs/latest/operations/api-reference.html#common>

you should keep an eye on `task/action/run/time` . Ideally, there shouldn't be any significant increase in that metric.

Our use is pretty simple, so generally not a lot of complexity to worry about for versioning, imo.

Coordinator can still connect to the database to poll for new segments even if there is no indexing happening.

Thanks. Assuming that no segments are loading/being created, will queries still run?
A small test we did shows that queries still run unaffected (where we upgraded the postgres version while querying)

Also, with regards to what you mentioned about the coordinator, is there a risk of the coordinator misbehaving in a way that risks the operation of the cluster while it's not able to access the metadata database (again assuming nothing is happening other than queries)?

There shouldn't be any risk to data.  Druid can handle temporary issues connecting to the metadata DB; it just can't do anything that would need to update the metadata (of course).

Yes, it’s generally not a good idea to conflate / share ZK and MDS between 2 clusters.

I have seen same zk being shared across druid clusters without issues. You will have to set some properties to make sure that they do not step on each other in zk.

For metadata store, you would have to set a different value for
```druid.metadata.storage.tables.base```
for different clusters.

`druid.zk.paths.base` for ZK. It defaults to `/druid` so just make sure the second one is set to something else like `/druid2` .

It means that druid is not able to emit its application metrics to the backend system that you have configured.

Here's the default one: <https://github.com/apache/druid/blob/master/extensions-contrib/graphite-emitter/src/main/resources/defaultWhiteListMap.json>

it's a map from metric name to the dimensions you want to include

So if I use e.g.  [“dataSource”, “type”] and [“type”] for the same metric (e.g. to get query/time splitted by datasources/type, and then just by type itself) the metrics tree in the graphite browser has different entities on the same level -- both datasources names and types intertwined. Is there any good way to walk this around?

Hmm, that I'm not sure… we've got outside what I know about graphite!

Druid version 0.21.0
Tuning config in ingestion spec:
```"tuningConfig": {
  "type": "index_parallel",
  "appendableIndexSpec": {
    "type": "onheap"
  },
  "maxRowsInMemory": 100000,
  "maxBytesInMemory": -1,
  "maxTotalRows": null,
  "numShards": null,
  "splitHintSpec": null,
  "partitionsSpec": {
    "type": "hashed",
    "numShards": null,
    "partitionDimensions": [
      "id",
      "actual_event_time"
    ],
    "partitionFunction": "murmur3_32_abs",
    "targetRowsPerSegment": 3000000
  },
  "indexSpec": {
    "bitmap": {
      "type": "roaring",
      "compressRunOnSerialization": true
    },
    "dimensionCompression": "lz4",
    "metricCompression": "lz4",
    "longEncoding": "longs",
    "segmentLoader": null
  },
  "indexSpecForIntermediatePersists": {
    "bitmap": {
      "type": "roaring",
      "compressRunOnSerialization": true
    },
    "dimensionCompression": "lz4",
    "metricCompression": "lz4",
    "longEncoding": "longs",
    "segmentLoader": null
  },
  "maxPendingPersists": 0,
  "forceGuaranteedRollup": true,
  "reportParseExceptions": false,
  "pushTimeout": 0,
  "segmentWriteOutMediumFactory": null,
  "maxNumConcurrentSubTasks": 16,
  "maxRetry": 3,
  "taskStatusCheckPeriodMs": 1000,
  "chatHandlerTimeout": "PT10S",
  "chatHandlerNumRetries": 5,
  "maxNumSegmentsToMerge": 100,
  "totalNumMergeTasks": 10,
  "logParseExceptions": false,
  "maxParseExceptions": 2147483647,
  "maxSavedParseExceptions": 0,
  "maxColumnsToMerge": -1,
  "buildV9Directly": true,
  "partitionDimensions": [
    "id",
    "actual_event_time"
  ]
}```
Tuning config from ingestion task:
```"tuningConfig" : {
  "type" : "index_parallel",
  "maxRowsPerSegment" : 3000000,
  "appendableIndexSpec" : {
    "type" : "onheap"
  },
  "maxRowsInMemory" : 100000,
  "maxBytesInMemory" : -1,
  "maxTotalRows" : null,
  "numShards" : null,
  "splitHintSpec" : null,
  "partitionsSpec" : {
    "type" : "hashed",
    "numShards" : null,
    "partitionDimensions" : [ "id", "actual_event_time" ],
    "partitionFunction" : "murmur3_32_abs",
    "maxRowsPerSegment" : 3000000
  },
  "indexSpec" : {
    "bitmap" : {
      "type" : "roaring",
      "compressRunOnSerialization" : true
    },
    "dimensionCompression" : "lz4",
    "metricCompression" : "lz4",
    "longEncoding" : "longs",
    "segmentLoader" : null
  },
  "indexSpecForIntermediatePersists" : {
    "bitmap" : {
      "type" : "roaring",
      "compressRunOnSerialization" : true
    },
    "dimensionCompression" : "lz4",
    "metricCompression" : "lz4",
    "longEncoding" : "longs",
    "segmentLoader" : null
  },
  "maxPendingPersists" : 0,
  "forceGuaranteedRollup" : true,
  "reportParseExceptions" : false,
  "pushTimeout" : 0,
  "segmentWriteOutMediumFactory" : null,
  "maxNumConcurrentSubTasks" : 16,
  "maxRetry" : 3,
  "taskStatusCheckPeriodMs" : 1000,
  "chatHandlerTimeout" : "PT10S",
  "chatHandlerNumRetries" : 5,
  "maxNumSegmentsToMerge" : 100,
  "totalNumMergeTasks" : 10,
  "logParseExceptions" : false,
  "maxParseExceptions" : 2147483647,
  "maxSavedParseExceptions" : 0,
  "maxColumnsToMerge" : -1,
  "buildV9Directly" : true,
  "partitionDimensions" : [ "id", "actual_event_time" ]
}```

Hi Srikanth,  I thought they were just different names (over time) for the same parameter ... am trying to confirm now ...

Btw, how big is the resulting segment (in rows and MB), and what shard type/spec does it say you have?