Apache Druid

Thanks Gian, It is not, same error
```SELECT ARRAY_AGG (ids, 3000)
FROM view
where __time BETWEEN '2022-08-21 22:15:16.0' AND '2022-08-21 23:20:16.0'
GROUP BY traceId
HAVING 
ARRAY_CONTAINS(ARRAY_AGG (ids, 3000), ARRAY['test']) ```


hmm, not sure if this is supposed to work or not. trying to figure out if it's a bug report or feature request :slightly_smiling_face:

what version is this?

&amp; <@U030C8H59T8> do you know if this case is meant to work?

I think its a bit of missing functionality, it looks like filtering stuff is not well wired up to arrays other than stuff that uses expression filter

since there is no column processors case for array types, so it can’t make a value matcher probably

or .. something, i can look closer at the code involved in a bit

maybe we can start with adding an `@Ignore` test for this case?

or one that expects the current failure and so breaks when we add support :shrug:

i like the `@Ignore("github issue url goes here")` but either one works

how many segments do you have in an interval?

looks like 32768
``` "rowStats": {
    "determinePartitions": {
      "processed": 86721828,
      "processedWithError": 0,
      "thrownAway": 0,
      "unparseable": 0
    },
    "buildSegments": {
      "processed": 32768,
      "processedWithError": 0,
      "thrownAway": 0,
      "unparseable": 0
    }```

we use short to represent the partition identifier in an interval so we can't exceed the 32768. what you are seeing here is short overflow

Thank you for your answer. I fixed my mistake :blush:

Window functions aren't currently supported in Druid SQL, so that's why you're seeing an error

yep that's why. the error message is misleading however. ideally, it should say something like "window functions are not supported". i wonder why it mentions MIN instead?

because `MIN` as a scalar operator indeed isn't supported. in this particular query, that's how MIN is getting read as.

we could reject a `RexCall` if it is of type `RexOver` before we attempt to convert the operator

<https://github.com/apache/druid/pull/12959/commits/b35f2cb3f73cd1b924aa0ed46fee528df5d4c26c>

Many tks all. I solve this problem by creating temp table and using join :laughing:

what emitter are you using &amp; do you notice what kinds of metrics are being emitted?

some of the emitters have configurations for controlling what gets emitted

such as the whitelist metrics config for the graphite emitter <https://druid.apache.org/docs/latest/development/extensions-contrib/graphite.html#white-list-based-converter>

Druid is emitting these metrics... can you please clarify do we need all these metrics and if not how it get rid of these metrics
```druid_emitted_metrics_histogram_sum
druid_emitted_metrics_histogram_count
druid_emitted_metrics_histogram_bucket
druid_emitted_metrics```

In the config of the service you are running check what
`druid.emitter` is set as

ah yeah. that one doesn't have an ability to filter: i believe the thought is that the http server receiving the metrics would filter if necessary

in your scenario, what's the http server? is it possible to configure filtering there?

<@U0344FW86DD> Can you please ping me any docs that how can we perform the filtering on http emitter

And can you please suggest any other emitter i can use

if you're using grafana, maybe graphite? that one supports filtering

You could also do filtering on the metric collector: whatever is listening to the HTTP emitter, you could set it up to discard certain metrics

<@U0344FW86DD> I was using the druid-exporter running as pod  which listen to one endpoint on which druid infra is emitting metrics.
From druid exporter  our prometheus is scraping metrics and we are visualising the dashboards in grafana.
We can able to see the data in prometheus for some time and then the prometheus dosent scrape the metrics

did you try <https://github.com/opstree/druid-exporter/blob/master/README.md> the `--no-histogram ` option ?

Also 5-6 Gb of metrics considering your emitter frequency is set to 3 mins means you have a really really big cluster I guess

<@U0315MQKEHK> Sorry for late response.
I had set the --no-histogram option now and the metrics emit size was in MB's now.But still it seems the value is high.
can you tell what this option is for --metrics-cleanup-ttl

I think you should look at druid compaction : <https://druid.apache.org/docs/latest/ingestion/compaction.html>

To many segments also causes more emission of metrics right <@U0315MQKEHK> ?

I am not sure about metrics emission but its generally not a good place to be

I've never used Drill (but I'm guessing someone around here has), but I just looked at the plugin. Which version of Druid are you using?

i'm not super familiar with drill &amp; this sounds like a drill error — any idea what Druid API it's calling?

maybe we can provide some advice based on that

Am not sure about the Druid API which Drill is invoking. Googled on it but couldn't find relevant information..

Out of curiosity, what are you trying to do?

if you get any additional info from logs or elsewhere, let us know

it may also be that you don't need drill + druid together? and can use druid directly? depending on what you're trying to do

Am trying to hook up Druid to Spotfire, the BI visualization tool from TIBCO. Please let me know, if you have any other route through which I can do this..

does Spotfire support JDBC? you could use JDBC, if so: <https://druid.apache.org/docs/latest/querying/sql-jdbc.html>

oh! <@U03FC3V304U> which grafana plugin do you use?

I'm currently working through an example right now using `plugins: "grafadruid-druid-datasource"`

We have this experiment right now, actually.
So theoretically you can, but in cases with "random" string values you might have huge cardinality.
Basically, you will need a bigger cluster to handle the data than in Presto.
Maybe with some data retention rules it would be mitigated.
Makes sense?

Yup! We are on the same wavelength. I want to solve data discoverability and experimentation while running only 1 type of cluster for ease of deployment.

I am thinking of limiting retention to only 2 weeks or 1 month to manage costs.

Having 1 type of cluster will reduce support tickets coming from data scientists because they mixed up 2 different SQL dialects as well.

Or, use the presto/trino druid connector?

what you mean by data stored in Presto? <@U03V6S23FHN> isn't it just a proxy/engine?
do you mean druid compared to data stored in object-storage?

So, within our group, data is first ingested into a data lake powered by Iceberg.
We then installed Trino on top of that Iceberg catalog for data discovery and experimentation.

Now, over time, we find that Trino is really quite slow and our users are getting frustrated. This is why I am wondering if Druid can simply fill the gap.

Also, Because of having both Trino and Druid, our users have to write the same thing twice.

yeah, we have pretty much the same setup.
I'm far from claiming that I understand how Druid works, but I think it boils down do costs.

There are use-cases that Druid will have trouble, weird functions that have lots of data on the broker or full-text search, and there Trino will be also slow but you expect it to be slow lol.

Maybe you can run Trino over Druid and have a "quick access" table that is druid and one slow access table that is iceberg? This way you can use one sql syntax for everything?

my 2¢: i think what you're asking is really: can druid be used as a primary, general-purpose analytical database across an organization?

i've seen some people do this with wild success!

and i've seen some people attempt this and run into issues; in this second camp, the issues people tend to run into are:

1) SQL support — druid does not support everything in SQL yet. some gaps people cite include joins that require shuffles [druid joins, today, broadcast all but the largest relation], and windowed aggregations

2) cost for infrequently-queried data — druid requires a copy of data, in druid segment format, to be stored on data servers before it can be queried. this works great for frequently-queried datasets, since it's a very high-perf approach. but for infrequently-queried datasets, it can feel wasteful of storage.

generally, that first camp of people -- the ones with wild success -- are people that don't have these two items come up in their situation

for the second camp, @ imply we're actually working actively on these items; for some details on the vision check out: <https://imply.io/blog/a-new-shape-for-apache-druid/>

the first fruit of this work that you'll see is SQL-based ingest, coming in the next Druid release. you'll see more stuff coming in future releases too, so stay tuned :slightly_smiling_face:

&gt; Or, use the presto/trino druid connector?
on this idea specifically :point_up: i don't have much experience w/ the connector, but i worry about the overhead it adds to the queries that get routed to Druid, especially in high-concurrency scenarios (Druid is designed to be able to handle hundreds-to-thousands of QPS; from what i've heard, i don't think presto/trino does well with this kind of workload).

although, if your expected druid workload is lower concurrency, it should work OK. and it's certainly part of the allure of presto/trino: one system for querying data, using one dialect, wherever it lives.

<@U0344FW86DD> - absolutely not intended as a snarky remark. I am genuinely interested to know how folks have achieved 1000s of qps in Druid. We always ran into either historicals running hot on cpus or there not being enough broker threads to keep up with the qps. Note that this was with a fairly heavily loaded cluster with no tiering and using top of the line hardware. I am assuming this involved almost all of the data in page cache which reduced segment scan time and other associated metrics significantly. 

Also, for data that effectively doesn’t rollup, what kind of strategies help when we have a few billion rows for a datasource to achieve reasonable query times? aone that comes to my mind is bucket partitioning over high cardinality column especially if majority queries filter on it. Any other strategies? 

Basically if we were to use Druid as a general purpose analytics database, even after the new sql support work, good way to handle this will be needed. In my experience, for stuff like this at scale with massive Iceberg tables, Trino/Presto shine. I suppose if we were to add support in Druid to have cold storage as a tier, it would make Druid a more attractive option as a more general purpose analytics engine. Of course, its realtime capabilities will make it stand out among other alternatives.

for the bit about 1000+ QPS: it's generally a situation where most queries hit a slice of the dataset, and things are set up such that pruning can be done at the broker level. meaning: queries naturally fall along datasource lines (and the data are partitioned by datasource; and there's lots of datasources) or they naturally fall along dimension-value lines (think: filter by userid = X and the data are partitioned using ranges of userid).

this pruning at the broker level means that most queries only hit a handful of historicals each, which enables very high concurrency as each historical is only seeing a portion of the overall queries

So the sauce is metadata caching and right kind of partitioning :grin:. Sounds familiar, but obviously needs a bit of thought when designing the schema.

for the question about getting good query times without rollup: it's kind of a common misconception that Druid is fast because of rollup. rollup is definitely helpful, but even without rollup Druid is quite fast compared to Presto/Trino due to the efficiency of the storage format and query engine. like, when measured on simple metric "how many rows can we process per CPU per second", Druid posts excellent numbers.

where it's less competitive is queries that involve multiple stages of processing, as that blog I linked discusses, &amp; which is a major focus of our effort over the next year

yep- partitioning is definitely the key to performance at higher QPS. it is a major factor in minimizing CPU use on historicals, and minimizing the impact of random fluctuations in query response time across historicals

Just read the blog. Exciting times in Druid! Happy to see all the work especially the downloading from s3 for querying. 

Following up on partitioning, we tend to use the default partitioning which is hash(all dimensions) effectively scattering the data randomly across segments for a segment interval. So does that effectively mean that broker is not able to do any pruning at all beyond time based partitioning?

yeah, that's true &amp; that's why we have been recommending `range` partitioning recently (now that we support multi-dim range partitioning)

&amp; in SQL-based ingest we will only support range partitioning (using REPLACE + CLUSTERED BY triggers range partitioning)

Druid is based on the scatter gather architecture.
We are changing a bit of it in the MSQ world but thats a separate conversation.
Now, we need a node which basically does the merge of all the historical results.

In theory we can assign any historical to become the broker and do this merge but then we will have to build a lot more things like resource isolation which is not a trivial task.

what's happening here is that SQL planner is thinking that only possible value of `tags` is tag1 and thus the query effectively becomes
```SELECT
'tags1',
 Id
FROM View
where __time BETWEEN '2022-08-21 22:15:16.0' AND '2022-08-21 23:20:16.0'
GROUP BY Id ORDER BY Id```

you should be using `MV_CONTAINS` operator instead of using `=` operator when filtering the relevant rows

also, when doing a group by a MV, it will expand each value in it's own row, so if you want to disable that, you need to use another function to keep the tags as array (I think it's MV_CONCAT or MV_TO_STRING)

or, maybe it's easier to use `groupByEnableMultiValueUnnesting: false` in the query context

btw there is some correction to be made in my query. The filter is not going to be removed. but the projection (tags --&gt; 'tags1') does get re-written.

Hi <@U031F9Q5D1P>! I haven't used this yet, but I did notice this language in the <https://github.com/apache/druid/pull/10524|PR>:

&gt; This PR provides the ability of auto scaling the number of Kafka ingest tasks based on Lag metrics when supervisors are running.
Maybe the behavior you're reporting is expected? The other thing I noticed, under the Task Autoscaler Properties, is that it's still designated as <https://druid.apache.org/docs/latest/development/extensions-core/kafka-supervisor-reference.html#task-autoscaler-properties|experimental>.

I'm hoping that other people who have used this will chime in.

Sounds like a bug to me. I think the autoscaler config for a given supervisor should be a part of its persistent definition, it should only change if you change it. You can report the bug at <https://github.com/apache/druid/issues> and perhaps link the PR referenced above.

yeah, that isn't intentional, and seems strange. in i see a unit test specifically checking that the `autoScalerConfig` gets retained

i wonder if there is something funny about your system? mixed versions, maybe?

when you go to `/druid/indexer/v1/supervisor/&lt;dataSource&gt;`, while the autoscaler is configured, what do you see?

Hello, thanks for having a look.
When I go to `/druid/indexer/v1/supervisor/&lt;dataSource&gt;` I can see my Supervisor spec with the autoScaler configuration

if that's there, then it means druid "understands" that you have an autoscaler set up — strange that it wouldn't be respected when you restart your OL

is it possible you have some mixed versions in your cluster? maybe some of your servers are older and don't have this feature?

Nope, I’m using Packer + Ansible to create custom system images, so every box runs the same version

hmm. what are you using for a metadata store? Basically the way this works is your supervisor spec is stored in the `druid_supervisors` table of your metadata store, and your OL reads from that on startup. Generally I'd expect that if the API echoes back a given config, it's also stored that way in the metadata store, which means it'll be restored that way on next boot

I’m using a MySQL.
Ok, so that’s definitely an issue since I can see autoScaler config in metadata store

Gonna do a bunch of Overlord restart and check debug logs

Hello, following the druid upgrade to 24.0.0, this problem is no more

image.png

I want to monitor my druid in single-server with druid-exporter , I config  follow <https://github.com/opstree/druid-exporter> like below
```# Enable metrics emission and tell Druid where to emit messages
druid.emitter=http
druid.emitter.http.recipientBaseUrl=<http://druid.opstreelabs.in:9091/druid>

druid.emitter.http.basicAuthentication=admin:admin```
and I run the druid-exporter with docker correctly like

I config prometheus like blow and restart

I came across this <https://groups.google.com/g/druid-user/c/KcyRTLDtMNY/m/WIhq4feqBAAJ?utm_medium=3Demail&amp;utm_source=3Dfooter|discussion> from about a year ago, but, so far, not finding much else.

unfortunately i'm not too experienced with the prometheus emitter!

i would suggest starting by looking to see if there are any emitter warnings or errors in your service logs — perhaps start with the Broker log, which should emit something for each query

Like this? <https://trino.io/docs/current/connector/druid.html>

I think he means to use Trino to read the segments that are part of the deep storage :thinking_face:

Can Trino read the deep storage data in S3 like hive ?

Sounds like you want Trino to read Druid segment files directly. I am not aware if such an integration exists but the Trino channels would be a good place to ask. It is certainly technically possible for something like this to exist... the Druid segment reader code is all open source (you know, as part of Druid), but I am not sure if anyone put it together yet. If that does exist (or you make it) please update the thread!

Is this the complete error? I'm researching but not finding a stacktrace that looks quite like this.

from some internet searching, it seems like this error can happen when the KDC referenced by your kerberos config is unavailable, or cannot be resolved via DNS

for example here is someone that ran into this with druid in the past, and fixed it by resetting their DNS: <https://groups.google.com/g/druid-user/c/g9kruOq4s0g>

so you might have an issue with unresolvable or unreachable KDC?

should i place krb5.config somewhere instead so it can pick it up?

I'm not a krb expert-- but that is what I would try next

that google groups solution says reset DNS whats that suppose to mean?

I'm not sure to be honest -- maybe he meant he restarted his DNS server? Or pointed his local config at a different resolver?

Btw, another thing to check might be if `kinit` works?

It needs to go to the same broker where the state is stored. That documentation you linked to shows how the router does this for you automatically:

&gt; If no `druid.router.avatica.balancer` property is set, the Router will also default to using the Rendezvous Hash Balancer.

If you didn't have a router you would need a solution to ensure the connections go to the same broker.

There is no way to make that driver stateless?

I guess if user uses the http endpoint, then it will be stateless.

I see this logic:
```String connectionId = getAvaticaConnectionId(requestMap);
targetServer = hostFinder.findServerAvatica(connectionId);
byte[] requestBytes = objectMapper.writeValueAsBytes(requestMap);
request.setAttribute(AVATICA_QUERY_ATTRIBUTE, requestBytes);
LOG.debug("Forwarding JDBC connection [%s] to broker [%s]", connectionId, targetServer.getHost());```


hm… looks like router does the right thing though.

```  public Collection&lt;Server&gt; getAllServers()
  {
    return hostSelector.getAllBrokers().values().stream()
                       .flatMap(Collection::stream)
                       .collect(Collectors.toList());
  }```
and then
```  public Server findServerAvatica(String connectionId)
  {
    Server chosenServer = avaticaConnectionBalancer.pickServer(getAllServers(), connectionId);
  }```

I will try to be clearer as I think maybe I added to your confusion. If you are using the router service everything will work for you automatically and there is nothing you need to do.

IF you are making connections directly to brokers via JDBC you need to make sure you hit the same broker every time. The router does this for you automatically. So if you hit the router first you don't need to worry about it.

ah, I think I saw the root cause, it’s another problem caused by non-existent IP addresses when deploying in a dynamic environment like Kubernetes.

in the old ways, our `druid.host=xxx` is ip address based.
That caused this code here:
```  public Collection&lt;Server&gt; getAllServers()
  {
    return hostSelector.getAllBrokers().values().stream()
                       .flatMap(Collection::stream)
                       .collect(Collectors.toList());
  }```
To possible contain non-existent IP addresses because sometimes pods disappeared and ZK somehow still keep track of that IP addresses.

But that’s fine, we have an answer already. Our new clusters uses DNS for ZK memberships, so this problem will go away.

I don't know if that state is actually persisted anywhere so if the Pod gets replaced even with a consistent name I don't think it would fully solve the problem of brokers being re-created mid JDBC session. Though maybe it would end up being more graceful as it would at least be forwarded to a broker that exists?

Right now, if the brokers membership changes in a cluster (new one comes or old one dies), some connections can be re-routed to a different broker. This is because there is no centralized routing of Avatica JDBC connections. Different routers use hashing to arrive at the destination broker independently. This logic works fine as long as the broker set doesn't change.

My customer said that as long as he doesn’t keep the connectionId, e.g. just close and don’t reuse connection), he no longer face issues. But that’s not ideal, preferably he wants to pool the connection still.

Yup. Not reusing the connection is as close as we can get to stateless communication.

I was not able to make this work reliability

you can try to use http discovery

<https://www.druidforum.org/t/not-able-to-replace-zookeeper-with-k8s-cluster/7671/10> &lt; further notes

I added this note to our GH issue about the extension: <https://github.com/apache/druid/issues/12904|https://github.com/apache/druid/issues/12904>

I guess we'll revert too ZK for the time being

It does look like the middle managers are dying after running their first batch of tasks

but they don't die "enough" for k8s to respawn them

Does anyone have any idea on this case? and is there anything wrong with the tests I did? I have created  issue on github , but no responds until now <https://github.com/apache/druid/issues/13006>
Best wishes.

I replied on the GH issue. Hope that is helpful.

I also replied, with some notes about secondary partitioning too — if you aren't already doing it for your query, it would be quite helpful

try
```select count(distinct item, year, location) from tableA```

You can try this
```SELECT COUNT(DISTINCT ARRAY[item, year, location]) from tableA```

it seems that we currently don't support using multiple fields in a approximate count aggregation via SQL.

oh okay so then Native query is the only option

CardinalityAggregatorFactory accepts multiple fields already. It seemed trivial to support that in SQL.

yeah. It would need changes in `BuiltinApproxCountDistinctSqlAggregator`

<https://github.com/apache/druid/pull/13014/files>

in the meantime, you can also use concat like `APPROX_COUNT_DISTINCT_DS_HLL(item || year || location)` or get fancy with separators `APPROX_COUNT_DISTINCT_DS_HLL(item || ':' || year || ':' || location)`

It's not as fast as the native option (it has to actually compute the concatted string) but it would work

Thanks <@U0344FW86DD>. the reason i am doing this is to get approx count of the records i will be reading in my application before actually firing the actual query so that i can limit the data if needed.
i am sticking to the Native query as response time is a important factor. Do you think this is the right way to go for getting the approx count of the query before actually firing the query ?

replied to similar question here: <https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1662032126151109>

there's a few different `common.runtime.properties` bundled in the distribution — try making sure you updated the right one for the start script you are using

for example if using `bin/start-micro-quickstart` you would update `conf/druid/single-server/micro-quickstart/_common/common.runtime.properties`

if that still doesn't work, make sure the function is spelled right?

the most basic datasketches function is `APPROX_COUNT_DISTINCT_DS_HLL(expr)` — you can try that one just to confirm, like:

```SELECT APPROX_COUNT_DISTINCT_DS_HLL(any_column)
FROM any_table```
this is the simplest query that should work if the datasketches extension is loaded

PasswordProvider may be good for this use case too

75 segments with total size of 40MB seems like a lot of segments! generally that amount of data would be in one segment, which would be more efficient for this high concurrency workload

issue is that when you have a lot of small segments, the query time can be dominated by per-segment overheads

can you try to compact the dataset? to do it manually as a one-time job, refer to <https://druid.apache.org/docs/latest/ingestion/compaction.html#setting-up-manual-compaction>

if that works for you you can set up auto-compaction as well. this is common for streaming ingestion

ok, thanks for the reply :slightly_smiling_face:
I have some follow up questions, sorry
• It sounds weird that 75 small segments are so slow. I read <https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html#segment-cache-size|here> that historicals would store the segments in memory. Do I need to configure something for that?
• I ran the simplest compaction <https://druid.apache.org/docs/latest/ingestion/compaction.html#example-compaction-task|example job>, and it was - waiting until I stopped the supervisor. I suppose it needs to lock the segments I want to compact. Is it possible to keep the compaction running in parallel to the supervisor? just making sure because I tried resuming the supervisor while the compaction task was running and it eventually failed.
• What is the difference between reindex job and compaction job. I assume that reindex can also create new partitions for the segments.
After running the compaction, it's 2 segments now, and testing it a bit, I see that the performance did improve when the cluster was huge, but it was still very slow. When we ran around 100 queries some queries took 15 seconds still.
I tried to shrink the cluster to 2 historicals, 2 brokers, with historical config
```  jvmMaxDirectMemorySize: 3G
  jvmMemory: '1G'
  runtime:
    bufferSizeBytes: 100MiB
    processingThreads: 4
    httpThreads: 150
    cacheSizeBytes: 256MiB```
and broker config:
```  jvmMaxDirectMemorySize: 2G
  jvmMemory: '2G'
  runtime:
    bufferSizeBytes: 100MiB
    numConnections: 60
    processingThreads: 2
    httpThreads: 70```
and I see that the scan times spike to 1.5 seconds, the scan pending is very low now, and the CPU usage is also low.
The `query/cpu/time` is quite high, 160ms. Not sure whether it's bad.
What more can I look at to understand the bottleneck?

One more thing, we are doing some sub-querying to group by some columns. This might hinder performance but I wouldn't expect it by that much... Maybe I should add more processing threads, or have larger/more buffers? Currently it's 1:2 threads to buffers

some thoughts:

&gt; It sounds weird that 75 small segments are so slow
this is mainly due to per-segment overheads: each segment has its own dictionary, indexes, etc, &amp; it's generally faster to amortize this stuff across multiple segments

&gt; I suppose it needs to lock the segments I want to compact. Is it possible to keep the compaction running in parallel to the supervisor?
compaction and active ingestion do indeed mutually lock out: so, generally, when using auto-compaction people will use the `skipOffsetFromLatest` config such that they don't run for the same time ranges

when doing a manual compaction just for testing purposes, generally I suspend the supervisor while compacting to keep things simple (&amp; then use `skipOffsetFromLatest` in production)

&gt; What more can I look at to understand the bottleneck?
to analyze bottlenecks i like to use flame graphs: <https://support.imply.io/hc/en-us/articles/360033747953-Profiling-Druid-queries-using-flame-graphs>

if you capture these on your Broker and Historical, feel free to post them here with your query and we can see if we have additional suggestions

&gt; One more thing, we are doing some sub-querying to group by some columns.
this may be relevant, if your subqueries are returning larger amounts of results, that can create a bottleneck (as all subquery results need to be written into merge buffers and then sent out to the Broker)

if you're in this situation, you could potentially benefit from more merge buffers, or from new option in 24.0 (not out yet! but soon :slightly_smiling_face:) added here: <https://github.com/apache/druid/pull/12513> (`mergeThreadLocal`). each of these things targets different bottlenecks, so the thing that would benefit you the most could be learned from the query + flame graphs

That indeed help a lot. I will test these tnigs out

There are several layers of onion to peel.
1. the latest build has Jetty metrics. See if you still have plenty of Jetty threads.
2. There are a lot of metrics that said xyz pending time. That will help you see at which stage you are stuck on.
3. If this is on AWS, see the EBS timing metrics, who knows you are stuck when loading data from segment-cache.

I'm pretty surprised that FilenameUtils.wildcardMathc allows `*` to match path separators. That's not normal for a glob. Like, in the shell, `*.txt` does not match `foo/bar.txt`

I don't think it would have anything to do with S3, since the glob library operates purely on strings

I'm not aware of a library that does it, but if I was doing it from scratch I would transform the glob to regex and use `java.util.regex.Pattern` to do the matching

like, `*` from a glob would become `[^/]*` in regex

yeah… I thought glob would provide a superior UX. But now I am regretting why not regex.

i was thinking user would provide a glob and then in the code we would transform it to a regex

this is where the nested data / JSON column stuff (merged in master now) is going next; <@U03NP2W7VF1> and <@U030C8H59T8> are the masterminds here

initial PR here <https://github.com/apache/druid/pull/12753> &amp; various followups have been done since then

next release (24.0) will include the initial version of this stuff — IIRC at first it'll support loading into a single "json" type column, not the full autodetection of all fields at root level. my understanding is that it is going in that direction for future releases

nice. how long have you been running on this server?

We had it running on a non prod server for around 3 weeks or so, almost a month. And we upgraded almost all of our prod servers last week.

But this is a small upgrade for us since we have been running 3.7.x ZK forever.