Apache Druid

Nothing really wrong with that, it's a short pause

You can make the logs less chatty by removing the GC details flags from `jvm.config`

<https://github.com/apache/druid/issues/5008> could be of help here

<@U031A9D19NU> Thanks for the help, but couldnt find a solution in that thread. Any other suggestions?

Umm no. You would need to do some tweaking to reduce the payload size of your task spec.

or you can try SQL based ingestion (MSQ) that doesn't have similar issues

historical reaching 100% capacity is fine. The disk becoming 100% full though is a problem and can cause weird problems. You should assign the capacity to historical so that disk is not full even if historicals reach 100% capacity.

Is there any discussion about this in the official docs?

it could be down to floating-point rounding

you could check this by doing, like, `m_criticality BETWEEN 0.3799 AND 0.3801` instead of `m_criticality = 0.38` and see if that is useful

Yes, it was down to rounding issue. When I give round(m_criticality, 2) &lt;- this works. So like in other programming languages, the storage representation of float must be a approximation and not an exact.
I am indexing this data to druid from parquet and that field calculation is already round( x, 2). So I did not expect this issue.

Hi <@U0344FW86DD> , I want to autoscale up middle-manager whenever any task comes in *pending* state. To achieve this, I tried the following configs that are mentioned <https://druid.apache.org/docs/latest/configuration/index.html#overlord|here> but there was not impact.

So how can I autoscale middle manager in this case?

you would also need to set a `druid.indexer.autoscale.strategy`

and some props specific to the strategy (either ec2 or gcp props)

The error appears to state that Druid doesn't know how to resolve 2 conflicting aggregators for the column `mt_l_uppts_inferece_avg`. In one segment it is calculated as a LongSum and in another it is calculated as a DoubleSum

You can resolve this by explicitly telling Druid what the metricsSpec should be in the auto-compaction spec <https://druid.apache.org/docs/latest/data-management/automatic-compaction.html#configure-automatic-compaction>

in fact, if you know the granularitySpec, dimensionsSpec and metricsSpec of the datasource - you can specify this in the auto compaction config and it should speed up your compaction jobs, as they no longer need to download segments to infer the schema.

```Compaction tasks fetch all relevant segments prior to launching any subtasks, unless the following items are all set. It is strongly recommended to set all of these items to maximize performance and minimize disk usage of the compact tasks launched by auto-compaction:```
^ Relevant part from the docs

Ok so we have to specify manually the metricSpec when configuring the AutoCompaction ? Is there any conf to indicate to use the latest segment schema as reference for instance ?

another question do you know if this conflict is only for a metric  `type` update are for any schema update (add/remove metrics/dimension) ?

&gt; Is there any conf to indicate to use the latest segment schema as reference for instance ?
I'm not aware of one, but that would be nice!

&gt; Ok so we have to specify manually the metricSpec when configuring the AutoCompaction ?
To resolve this issue via auto-compaction, yes. But you can also issue a manual compact task / a Druid re-index task to fix the time interval that auto-compaction is complaining about, and leave the auto-compaction config unchanged.

&gt; n do you know if this conflict is only for a metric  `type` update are for any schema update (add/remove metrics/dimension) ?
Adding or removing metrics should work seamlessly. This exception is only thrown when 2 aggregator types are conflicting for the same column, so Druid is unsure how to resolve the difference. In this instance I think there is the chance.

<https://github.com/apache/druid/blob/338bdb35ea19b495bc6eb587ab99978edb16ce46/processing/src/main/java/org/apache/druid/query/aggregation/AggregatorFactory.java#L186|https://github.com/apache/druid/blob/338bdb35ea19b495bc6eb587ab99978edb16ce46/proc[…]/java/org/apache/druid/query/aggregation/AggregatorFactory.java>

I'm unsure how conflicting dimension types are resolved in compaction

<https://github.com/apache/druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java#L1078> - this method seems to be the one that handles the dimensions spec from different segments.

To avoid issues like this, we usually start compaction on much older dates, e.g. 2 weeks older from today because our customers usually already update incorrect schema and performed backfills within those 2 weeks.

Ok but actually whatever the exclusion period you'll always have this conflicts if 2 segments with different schema have to be conpacted no ?

yes, correct. Not bulletproof. This will happen anytime you have 2 different schema in the same compaction time range.

try using http load queues on the coordinator and enable batching

something like this on Coordinator:

```druid.coordinator.loadqueuepeon.type=http
druid.coordinator.loadqueuepeon.http.batchSize=8```

that should allow all your loading threads to be used

Isn't it in here
<https://druid.apache.org/docs/latest/operations/metrics.html|https://druid.apache.org/docs/latest/operations/metrics.html>

Not sure <@U04KRGPGWAU> which one can be used?
The end goal is to use this port allocated to each task check the memory cosumption it is doing while it's running.
`jvm/pool/used`  
Can the above metric be used for that?

I'm using it with jvmmonitor to see gc, mem/used

<https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1683611494610449?thread_ts=1683611494.610449&amp;cid=C0309C9L90D|https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1683611494610449?thread_ts=1683611494.610449&amp;cid=C0309C9L90D>

I think you asked same question already. Have you even try for collect metrics jvm/... categories?

We are already using these metrics for monitoring

I think your point is that you want to monitoring memory usage *per task*

each task how much memory it is consuming (whether it's index, compact)

Because I want to analyse these consumption and then come up with scalable approach

I agree for that cuz tasks have bunches of step or category as you said indexing or compacting ...

<https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1684923698475239>
This question is for that only. so if the port that is allocated to that particular task I can use the port get replaced in any of these metric which gives us the memory consumption can get the utilisation

Ahh than we can troubleshooting memory consumption more directly and determine scaling

One more amazing thing we can do actually with that.
as we know for which data source this task would be running so for that particular data source we can also change ingestion Spec
when I say spec we can allocate the required memory and other required resource based on our analysis

I am also interested to implement this actually
<https://github.com/apache/druid/issues/14253>
but need some guidance actually for this. In terms of understanding required src and files.

I will look into that too. And in here, other guys can give more opinions too

Hopefully hear you soon <@U04KRGPGWAU> on this Thanks !

Each task running on the same node uses a different port. 

as always with tuning stuff "it depends" :slightly_smiling_face:

the main resource to consider is Broker memory

depending on available Broker memory, a reasonable range for total # of open statements on a Broker would be totaling 50–300 across all connections

the defaults get you a max of 100 (the default max connections is 25, and default max statements per connection is 4)

you could also check to make sure your JDBC clients are closing statements properly

if some JDBC client leaks statements then it can use an "unreasonable" amount

thanks <@U0344FW86DD>, sounds reasonable. I will check once again about statement leaks

There is a heartbeat http call that checks for historical’s liveliness.

if using ZK, the ZK client provides the liveliness check.

when it’s down temporarily, then broker will ask coordinator where is the other replica.

when it’s down for longer time, I believe there’s a flagging system that mark the node as down. Then coordinator’s background thread will pick another historical and starts a new replication process.

This is just in the nutshell. Others know much more than me.

thanx <@U03149ALHN3>!
Do you know how is this `then broker will ask coordinator where is the other replica` happening? How does the broker know to ask that?

slight correction— Broker will ask either ZK, or the historicals directly, depending on whether you use ZK or HTTP based segment announcing

in particular— Coordinator tells Historicals which segments to load, and Broker gets that info from ZK or HTTP calls to Historicals, depending on what kind of segment announcing you use. Recently the default changed to HTTP, so if you use the current version of Druid it'll be HTTP

thanx <@U0344FW86DD>!
I'm still a bit not sure about the timings.
we use ZK.
So lets say some historical goes down at 19:00, and after 10 seconds it's records disappear from ZK.
When will the broker get to know it? I guess it doesn't fetch the segments info from ZK for each query - but rather periodically. So how frequent is it? Can it be controlled?

the Broker uses ZK watches to stay in sync, so it will know about the update almost immediately after ZK does

what version are you running, I added a bunch of metrics to statsd as feature parity starting from Druid 24.

oh… hm, and you are using statsd-emitter to Datadog? That’s what I use.

if you happened to know which metrics are missing, you can send PR to this file. Very similar to what I did in the past: <https://github.com/apache/druid/pull/12762/files>

I have done compactions with mixed segment overshadowing and did not have a problem.  In your compaction spec are you only specifying the target time range to compact, not specific segments?

What process is adding the small amount of data? Can you make that process use `appendToExisting=true`?

this was my spec
```{
  "type": "compact",
  "dataSource": "dev.do.overshadow-test",
  "ioConfig": {
    "type": "compact",
    "inputSpec": {
      "type": "segments",
      "segments": [
        "dev.do.overshadow-test_2023-01-01T00:00:00.000Z_2023-02-01T00:00:00.000Z_2023-05-24T13:27:31.235Z",
        "dev.do.overshadow-test_2023-01-01T00:00:00.000Z_2023-02-01T00:00:00.000Z_2023-05-24T13:15:52.256Z"
      ]
    }
  }
}```
since the older `*T13:15` segment was unused i got an error
tried it now using a time interval
```{
  "type": "compact",
  "dataSource": "dev.do.overshadow-test",
  "ioConfig": {
    "type": "compact",
    "inputSpec": {
      "type": "interval",
      "interval": "2023-01-01/2023-01-02"
    }
  }
}```
that produced a third segment but it is identical to the second one (the one with less data)

So your original segments had MONTH granularity, and you overshadowed it with an update with (DAY?) granularity?

Or are you just replacing one MONTH segment with another MONTH segment?

If MONTH -&gt; MONTH, then compaction should merge the two segments.

To Kyle's comment -- if you ingested the second segment with append=false, then the contents of the newer segment should replace the contents of the older segment ... and if you ingested the second segment with append=true, then the contents should be unioned together in the resulting segment.

For your second compaction spec, you specified a DAY range, not a MONTH range ... was that a typo?

<@U030MBK46BD> Thats another thing we are trying to figure out. At first I assumed it was a message that was bouncing around in the web of kafka streams for multiple hours, until the currently segment was published, but there are also some really obscure cases that need further investigation
Indeed, the affected datasource did not use `appendToExisting` because it didnt use the `dynamic` partitioning type
<@U04C4593VDY> the actual data has segmentGranularity=day, this is a smaller example with segmentGranularity=month

`appendToExisting` is only relevant to the ingestion task, not the existing datasource.  So it doesn't matter how the original segments got there, this parameter only affects what happens when you ingest the new data.

I guess a small caveat to this ... if the original segments were already a result of multiple ingestions into the same time period and you haven't compacted that time period yet, then yes the existing segments may be marked as having been ingested in overwrite or append mode, and that will affect what queries see, and what compaction will do.

Back to your original question:

&gt; I have a datasource where several days=segments with millions of rows are overshadowed by segments that were added later with only a tiny fraction of rows
"overshadowed" implies that the newer segments were ingested with appendToExisting=false ... and if all segments involved were at the DAY level, then compaction (using your second spec) should merge them.

ingestion-overshadow.json

yes, its still important to make sure we dont overshadow our data again,after restoring it, but doesnt solve the immediate problem ob being able to access both the original data and the new data which overshadowed it.
Are you sure about compaction merging inactive segments by default? Assuming the granularity doesnt change, It is supposed to pack multiple partitions for one interval into fewer new segments, I dont remember anything in the docs about handling multiple versions. And thats what my tests shown.
Here is the simple inline ingestion i used:
after its finished, i ran it again with different `"data": "time,id,value\n2023-01-01,1.01,1.01\n2023-02-01,2.01,2.01\n2023-03-01,3.01,3.01"`
to overshadow the previous segment  and then ran the compactions on it

On my system the DRUID_SEGMENTS table (in the metadata DB) has only has a "USED" field in it.   The other attributes such as is_published, is_available, is_overshadowed, are therefore derived by the Coordinator when the SYS virtual tables are populated.  Which also means to me that you may not automatically update the SYS table content by manually updating the USED flag in  DRUID_SEGMENTS ... or it might take a while for the Coordinator to refresh it's view of the segment states.

===

Just to confirm ingestion/compaction  behavior, on my local quickstart install I just tested this on a wikipedia datasource ingested with HOUR granularity:
• Ingested a second segment of HOUR granularity with REPLACE (append=false) and it completely overshadowed the original existing segment, moments later the original segment was automatically deleted.  No need for compaction in the case of full overshadowing.
• Ingested a second segment of HOUR granularity using INSERT (append=true) into the table, resulting in two active segments for that hour.  After compaction these consolidated to one segment that has the union of data from both segments.
===

This last spec you posted has segment granularity at MONTH.  If you ran two ingestions with different data sets over the same three month period, then you should end up with 6 segments in total, and 6+3 = 9 records of data in it.  None of these should be overshadowed ... they should all be additive.

And if you run compaction at MONTH granularity (or just set up default auto-compaction with P0D offset and kick it off a few times) you should end up with 3 segments in total with those 9 records still in them.  Is this not what you are seeing?

The java stack btw:
```Caused by: java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]
	at java.nio.DirectByteBuffer.&lt;init&gt;(DirectByteBuffer.java:118) ~[?:?]
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
	at org.apache.druid.segment.CompressedPools$4.get(CompressedPools.java:102) ~[druid-processing-25.0.0.jar:25.0.0]
	at org.apache.druid.segment.CompressedPools$4.get(CompressedPools.java:95) ~[druid-processing-25.0.0.jar:25.0.0]
	at org.apache.druid.collections.StupidPool.makeObjectWithHandler(StupidPool.java:171) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.collections.StupidPool.take(StupidPool.java:154) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.segment.CompressedPools.getByteBuf(CompressedPools.java:110) ~[druid-processing-25.0.0.jar:25.0.0]
	at org.apache.druid.segment.data.DecompressingByteBufferObjectStrategy.fromByteBuffer(DecompressingByteBufferObjectStrategy.java:49) ~[druid-processing-25.0.0.jar:25.0.0]
	at org.apache.druid.segment.data.DecompressingByteBufferObjectStrategy.fromByteBuffer(DecompressingByteBufferObjectStrategy.java:28) ~[druid-processing-25.0.0.jar:25.0.0]```

Seems to me like data sketches uses a lot more direct memory than expected!

IIRC, the sketch’s internal hash table can be up to 2x nominalEntries, so 256kiB since each entry is 64 bits, plus something like 32 extra bytes for extra internal use (storing configured nominal entries, current size, etc)

A direct sketch should be identical to the serialized image of a sketch

Thank you for your reply. Our cluster has been running for a very long time, and yesterday, after we added several broker pods, this issue suddenly occurred. So it may not be a configuration issue

can you log into the pod and see what does `curl` command outputs? that might tell you where to look next. you can also check if for some reason, broker is taking too long to startup

image.png

when I log into pod and curl output as follow.

But after 10 minutes the output is true. So now the pod be ready need maybe 10 minutes, before is 2 minutes .
I don’t why it cost so long time.

I can help you troubleshoot though it would need a bit of work on your end as well :slightly_smiling_face:

When we see this, it's usually the result of broker taking time to initialize its segment metadata view. So if there is a way for you to look at the metrics for segment metadata queries for the broker that you just started, you can see how much time those queries are taking.

One other thing that you can do is take flame graphs right after the broker starts
<https://support.imply.io/hc/en-us/articles/360033747953-Profiling-Druid-queries-using-flame-graphs>

you should take 5 flame graphs collected over 2 minutes.

If this is a bit too much for you, you can just increase the health check time for brokers and move on. But otherwise, these are some of the things you can do to learn more and help us learn more.

Thanks very much for all your help!:raised_hands:
In fact, I have indeed seen many metadata queries, and I will calculate the time consumption of these queries.
Later I will open the flame diagram to find the root cuase. I’m not familiar with flame diagrams, I’ll take some time to study them. Thanks~

thanks. flame graphs seem daunting but are a very useful tool to do perf troubleshooting.

I observed the logs and found that the broker has always had a query for segmentMetadata. Do I need to check the query for segmentMetadata that was druid broker just started?

out.svg

We have 4 historical pods, the first one took 10 minutes to start, and the second one was not ready for even longer. I generated a flame chart when these two pods started.

But in some cases, each pod takes about 10 minutes to start successfully.
I saw some segment metadata refresh failed in the logs, as well as connection history timeout. May I ask if these will cause the block broker process to start?

broker should retry the failed queries. can you attach the broker logs? Also, is it possible for you to generate the flame graph as html as outlined in the link I shared. Lastly, we wanted the broker flame graph and not the historical flame graph.

I see the broker send the segment metadata queries， and usually the queries timed out. The warn log says connect historical-0 timed out. 

I see the historical-0 log and cpu metrics，it is normal. 

the metrics worth looking at will be busy threads on historical-0.

how many segments do you have in your cluster btw?

the historical overview metrics is normal,such as pod cpu/memory/iops . What metrics or log I should looking at?

The segments may 1.18million. we have 50 historical pods. Each pod has 2T volume, disk usage 80%

There is a metric to see how many jetty threads are busy. Whats the druid.server.http.numThreads set on your historical servers?

Sorry, I made a mistake. The druid.server.http.numThreads in historical we set is 115, and during the period when the timed out,  the value of Druid_Historical_ Jetty_numOpenConnection is always much larger than the set 115
After 05-24 16:00, the timed out occur very frequently.
So we should increase the config druid.server.http.numThreads?
And increase broker number maybe leads to this timed out?

yeah. I will suggest increasing this config. There is some advice here on how to tune the number of threads on historical. <https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html#sizing-the-connection-pool-for-queries>

Thank you so much for your guidance！ I will try to decrease broker number and increase historical threads number.      I want ask why the metadata query always timeout, but groupby queries seem success, there has priority in historical?

Once the system is stable and all brokers are up, do you still metadata queries taking longer?

Yes, there also have metadata refresh failed after broker startup. Also with historical timed out.
log error:
```<http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>: Query[c05a44d0-71d4-4fc1-a47a-640e90650fec] url[<http://rt-druid-historical-26.rt-druid-historical.rt-druid.svc.cluster.local:8083/druid/v2/>] timed out.
	at org.apache.druid.client.DirectDruidClient$1.checkQueryTimeout(DirectDruidClient.java:415) ~[druid-server-0.17.1.jar:0.17.1]
	at org.apache.druid.client.DirectDruidClient$1.handleChunk(DirectDruidClient.java:307) ~[druid-server-0.17.1.jar:0.17.1]
	at org.apache.druid.java.util.http.client.NettyHttpClient$1.messageReceived(NettyHttpClient.java:249) [druid-core-0.17.1.jar:0.17.1]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:119) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:485) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.6.Final.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]```


I know of use cases using thousands of values for IN clause. Not sure if it has a limit. Have you tested it?

I did test  till 5K worked fine to me, but was trying to understand the impact of it if the list is huge. I am assuming the slowness of bitmap will have some impact ? Not sure though.

Sounds like a decoding problem in the Parquet reader. I'll preface this with I don't know the code well, so I could be completely off the mark. But, I looked at the parquet extension code and found what may be the issue, precision and scale are not used when dealing with precisions &lt; 19 which in turn seems to just represent it as the internal integer type: <https://github.com/apache/druid/blob/2086ff88bcdbfdba507d52d78bd3c7605beab280/extensions-core/parquet-extensions/src/main/java/org/apache/druid/data/input/parquet/simple/ParquetGroupConverter.java#L370|https://github.com/apache/druid/blob/2086ff88bcdbfdba507d52d78bd3c7605beab280/exte[…]ache/druid/data/input/parquet/simple/ParquetGroupConverter.java>

Someone with more code experience should take a look.

I dont understand this code to be frank.. will try share around to make sense. Hope there was a work around. For the time being, I reverted the datatype to float in our iceberg/parquet and it works. But it has an issue that if I store round(value, 2) as float -&gt; then 0.13 is actually stored as 0.129999999 .. I am assuming this is because float is an approximation type and hence cannot store exact values like decimal.

I'm also facing this issue. Is there any solution for this ?

Hi all, What happens when I use servers under minimum requirements?

What happens when servers are under minimum is probably low performance as long as the configs are adjusted to not use more resources then available. What's your deployment? single server or cluster? If you are seeing problems with segment availability you should look at the coordinator log to determine why historicals are not able to load those segments.

Thank you, it may have been caused by our single server meeting minimum reqs. Next step, I'm migrating from a single server to a cluster.

hi <@U059CQX8MUM>. The error message in this case may be slightly misleading because you’re specifying a post aggregator `quantilesDoublesSketchToQuantile` and I do see `druid-datasketches` extension in the load list which houses all the sketch-related aggregators and post-aggregators. The error says that you can only have aggregators of those specific types (and no post-aggregators).

Going back to your question, are you specifying the post aggregator during ingestion time or query time? I don’t think you can specify them at ingestion time. Also, are `name` and `fieldName`  correct? (they seem to have spaces in them). Please see <https://druid.apache.org/docs/latest/querying/post-aggregations.html> on how to use post aggregators. Hope that helps

Yes. There is no aggregator such as `quantilesDoublesSketchToQuantile`. What made you try that out? is that something we say in the docs?

yes, we have `quantilesDoublesSketchToQuantile` in the post-aggregators section here: <https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles.html#quantile>

that's a post-aggregator though, not an aggregator.

You could try changing the replica count for just that one time chunk to see if it cleans it up ...

It does seem strange that if the historical server encounters an issue when trying to publish the segment then the coordinator is not informed of this…  perhaps the coordinator should not mark the segment as ‘available’ until the historical confirms a successful load of the segment?

Litttle work to find out where the datasource is distributed , but if u can remove it from segment-cache then it will be forced to load form deep storage

This happened when they try to use `array_agg()` function.
```array_agg(coalesce(the_column, ''))```

I think I came across something similar a while ago

Trying to dig out a bit more info about it. What happened was that there was a null character in my string. Languages like C++ use null characters to terminate strings whereas Java uses string length. So, this has the potential of having vulnerabilities. So, Druid SQL doesn't allow this. The solution was to use a regex to remove the null character

I haven't been able to find the relevant info yet but something like `
```REPLACE(myString, Char(0x00), '')```
` should work

You are welcome. You need to test out the regex a bit. I am not quite sure whether this is the right one or not.

<@U03149ALHN3> once you test it out please let me know whether it worked or not. I am curious to know :slightly_smiling_face:.

<@U0344FW86DD> would you have any suggestions/advice here?

<@U032X7PGEN4>, a couple of thoughts. Is the supervisor stuck reading non-existent offsets on select partitions?

Yeah, currently the supervisor reset API clears everything from the metadata table pertaining to a supervisor. IMO, it’d be nice to add support to reset metadata for a subset of partitions (would be less risky as well).

In the mean time, I wonder if you can edit the metadata table for the supervisor in question? Definitely exercise caution while doing this especially if this is a production system :slightly_smiling_face:

Thanks <@U04CLDXP7HC> Yes, the consumers stuck with negative offset lag were due to the non-existent offsets in the partitions.

We looked into updating the offsets in the metadata table. However, it is stored there in encoded form (presumably, of a json). So it's bit involved and unsafe and we decided to not go that route.

Thanks though.

<@U032X7PGEN4> got it, please feel free to open a GitHub issue/proposal to extend the supervisor reset API to reset specific partitions. Seems like a common and useful functionality to have in Druid :+1:

I see a config, maybe an answer.
`druid.broker.segment.awaitInitializationOnStart：Whether the Broker will wait for its view of segments to fully initialize before starting up. If set to 'true', the Broker's HTTP server will not start up, and the Broker will not announce itself as available, until the server view is initialized. See also druid.sql.planner.awaitInitializationOnStart, a related setting.`	
default is true

That is my understanding: the broker will not be open for business until it has the segment map when this is `true`.

I suggest you check out the new multi-stage query engine:  <https://druid.apache.org/docs/latest/multi-stage-query/index.html>

It is known to ingest much faster than the native engine, and it supports all of the same batch datasources through the EXTERN() spec: <https://druid.apache.org/docs/latest/multi-stage-query/reference.html#extern-function>

Hi All
We upgraded druid from 24.0.2 to 25.0.0. We have 2 historicals but after upgrade one couldn't come up and has gone in Crashloopbackoff.
In the logs error says = "Unrecognized VM option 'PrintGCDateStamps'Error: Could not create the Java Virtual Machine.Error: A fatal exception has occurred. Program will exit."
what caused this to happen? because when we upgrade from 0.22.0 to 24.0.2 no issue happend as such.

I believe this above issue is happening with JAVA version being used as 11 probably it's not recognising few of the old flags in old version of java.

The error means that the historical timed out when responding. What's the query? What's the data? The default query timeout is `druid.server.http.defaultQueryTimeout`

druid.server.http.defaultQueryTimeout=60s, the query is segmentMetadata type, query is too long.
I want to know the timeout maybe in historical, or in broker?

The error seems from the historical. What are your configs for broker and historicals?

broker config:
```# HTTP server settings
    ##druid.server.http.numThreads=60
    druid.server.http.defaultQueryTimeout=60000
    druid.server.http.maxQueryTimeout=60000

    # HTTP client settings
    druid.broker.http.numConnections=35
    druid.broker.http.maxQueuedBytes=50000000

    # Processing threads and buffers
    druid.processing.buffer.sizeBytes=536870912
    druid.processing.numMergeBuffers=4
    druid.processing.numThreads=13

    druid.query.groupBy.maxOnDiskStorage=104857600

    # Query cache disabled -- push down caching and merging instead
    druid.broker.cache.useCache=false
    druid.broker.cache.populateCache=false

    druid.sql.enable=true
    # We need to disable after broker started through kubectl edit for fast failed recovery
    # We need to set it true when rolling upgrade
    # Default true
    druid.sql.planner.awaitInitializationOnStart=true```

historical config:
```    # HTTP server threads
    druid.server.http.numThreads=115
    druid.server.http.defaultQueryTimeout=60000
    druid.server.http.maxQueryTimeout=60000

    # Processing threads and buffers
    druid.processing.buffer.sizeBytes=1073741824
    druid.processing.numThreads=30
    druid.processing.numMergeBuffers=50

    druid.query.groupBy.maxOnDiskStorage=104857600
    druid.segmentCache.numLoadingThreads=40

    # Segment storage
    druid.segmentCache.locations=[{"path":"/var/druid/segment-cache","maxSize":1900000000000}]
    druid.server.maxSize=1900000000000
    druid.segmentCache.lazyLoadOnStart=true

    # Query cache
    druid.historical.cache.useCache=true
    druid.historical.cache.populateCache=true
    druid.cache.type=caffeine
    druid.cache.sizeInBytes=10737418240
    druid.historical.cache.unCacheable=[]```

There has 40 historical and 8 broker. I know that historical threads not enough, it should increase.
But when the error happen, only one broker is startup, others are down. And because brokers are down, historical not too busy.
After some times, and brokers retry and retry, the segment metadata refresh success. I this the metadata in historical memory, it should be return quickly. The problem is most likely to occur on the broker？

i see you've commented out the http threads for the broker... the recommendation is:
```druid.server.http.numThreads on the Broker should be set to a value slightly higher than druid.broker.http.numConnections on the same Broker.```
8 brokers seems like a lot for 40 historicals, the ratio is usually 1:10, so 4 should be a good starting point. Another issue is that the http.numThreads for the historicals, the recommendation is:
```For Historicals, druid.server.http.numThreads should be set to a value slightly higher than the sum of druid.broker.http.numConnections across all the Brokers in the cluster.```
So given 8 brokers at 35 numConnections each. Historicals and Middle Manager peons should be configured to 8*35+10 = 290. with 4 brokers =150. Depending on what was going on in the historicals, it could be that they ran out of http threads and could not respond to the broker's requests.

If you haven't seen it already... the basic cluster tuning guide has a lot of great info on tuning a configuration: <https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html>

Thank you so much!  I will adjust the parameters of the broker and historical nodes as you instructed. And I’m still curious to know the root cause of being blocked when the broker starts, what causes metadata refresh failed?

:+1: Thanks for the update, let us know how it goes

hi, was it part of a rolling restart upgrade? Do you see any errors and warnings in the task logs?

Hi Kai,  you don't want to necessarily reduce these:

• When the data is first ingested it is placed on-heap in a "row buffer", which stores the data in simple row format.  This data is unindexed, and any queries against real-time data will scan these row buffers in their entirety.  This is known to not be the most performant scan.
• The intermediate perist takes the contents of the row buffer, converts it to a column-based indexed and encoded "mini-segment" and stores it on local disk.  This persisted file is much quicker for queries to access than the row store because of the format and indexing.
• Before a segment is built there can be many (dozens or even over a hundred) of these intermediate persist files.  When a query runs against these again the performance is fairly good.  The properties `maxBytesInMemory` and `intermediatePersistPeriod` control when persists occur. 
• Eventually the size or time threshold is reached and a segment is built and published.  At that time these persist files are all collected into one regular sized segment and sent to deep storage.
I would suggest reading through this section of the ingestion doc for an explanation of the various parameters involved: <https://druid.apache.org/docs/latest/development/extensions-core/kafka-supervisor-reference.html#kafkasupervisortuningconfig>

Because the persisted files serve queries faster than the row buffer, we generally see people lean towards more persisting and less time in the row buffer, balanced with appropriate timing for building segments.

There are a lot of moving parts in this area of the product, so feel free to post more detailed questions.