Apache Druid

I can find this file - <https://github.com/algolia/docsearch-configs/commits/master/configs/apache_druid.json> - but its readonly now

performance on this is looking pretty nice now :rocket: <https://github.com/apache/druid/pull/12277>

```
          if (httpResponseStatus.getCode() == 400) {
            // don't bother retrying if it's a bad request
            throw new IAE(exceptionMessage.toString());
          } else {
            throw new IOE(exceptionMessage.toString());
          }```


When using HTTP discovery the status code is 400 when the request produces a "no route to host" resulting from the removal of a pod.

Since this situation can always occur if a pod is spontaneously lost, this becomes a normal condition.  I'm considering handling this as an exception I can propagate that retains the information about the failure.  2 questions:
1. Is this the correct way to handle this? 
2. Would it make sense to have another exception in addition to `NoTaskLocationException` and `TaskNotRunnableException` that represents this situation?  Or does `TaskNotRunnableException` already cover this?  I'm not clear on the exact intent of `TaskNotRunnableException`

Oh, it looks like NoRouteToHost bubbles up as an IOException that I can trap in my task client.  Perhaps this will work as-is.

Confirmed.  I can handle the NoRouteToHost in my supervisor.

makes sense that it can be detected this way: it's not _really_ an HTTP protocol-level error — it's something lower level

curious - which component here is producing 400 for "No route to host" since that isn't technically a 400.

Here is the full exception from the logs
```{"level":"ERROR","@timestamp":"2022-10-04T16:18:56.598Z","thread":"HttpServerInventoryView-3","message":"failed to get sync response from [<http://10.4.154.126:8091/_1664898749884>]. Return code [0], Reason: [null]","exception":{"exception_class":"io.netty.channel.ChannelException","exception_message":"Faulty channel in resource pool","stacktrace":"io.netty.channel.ChannelException: Faulty channel in resource pool\n\tat org.apache.druid.java.util.http.client.NettyHttpClient.go(NettyHttpClient.java:125)\n\tat org.apache.druid.server.coordination.ChangeRequestHttpSyncer.sync(ChangeRequestHttpSyncer.java:218)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: Host is unreachable: /10.4.154.126:8091\nCaused by: java.net.NoRouteToHostException: Host is unreachable\n\tat java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\tat java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)\n\tat io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)\n\tat io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n"},"hostName":"storage--druid-coordinator-589db54d45-x4ndp"}```

`ChangeRequestHttpSyncer` is the Druid class making the call

ready for review - <https://github.com/apache/druid/pull/13170/files>

My current path of investigation is to track down where the segments unannounce.  I am pretty sure this is happening in the indexers, when a task completes and the segment is sent to deep storage.

In situations like this is can be tough to identify the "real" bug. The idea here is that `ChangeRequestHttpSyncer` will keep trying forever, because something _else_ tells it when the server went away. Peeling apart how it's supposed to work:

The `ChangeRequestHttpSyncer` is held by a `DruidServerHolder`. This is created by `HttpServerInventoryView#serverAdded` in two cases:

1a) `DruidNodeDiscovery` fires `nodesAdded` to its listeners. One of these listeners is in `HttpServerInventoryView` and creates the `DruidServerHolder`.
2a) `HttpServerInventoryView#scheduleSyncMonitoring` notices a server holder is not OK and re-creates it.

Then, `ChangeRequestHttpSyncer#stop` is called by `DruidServerHolder#stop` in two cases:

1b) `DruidNodeDiscovery` fires `nodesRemoved` to its listeners. One of these listeners is in `HttpServerInventoryView` and stops the `DruidServerHolder`.
2b) `HttpServerInventoryView#scheduleSyncMonitoring` notices a server holder is not OK and stops it prior to re-creating it.

So, what's supposed to happen when the pod goes away is (1b) above. The `DruidNodeDiscovery` created by `K8sDruidNodeDiscoveryProvider` is supposed to call `nodesRemoved` for that pod on its listeners, which in turn causes the `ChangeRequestHttpSyncer` to stop trying.

Two things that could be going wrong here:

• Perhaps the k8s-based discovery provider doesn't properly call `nodesRemoved` on its listeners.
• Perhaps it _does_ properly call `nodesRemoved`, but the `ChangeRequestHttpSyncer#stop` method doesn't "work": it keeps trying even after `stop` is called.
My guess is it's the first one: something wrong with the k8s discovery provider. That's because the `ChangeRequestHttpSyncer` is used anytime http segment announcing is in play, which is relatively common, and we haven't seen such problems with it there

Looking at the code for `K8sDruidNodeDiscoveryProvider` I see the listener fires in one place:

```                  case WatchResult.DELETED:
                    baseNodeRoleWatcher.childRemoved(item.object.getNode());
                    break;```
Is it possible something's wrong with the way we do these watches?

This part seems suspicious:

```              } else {
                // Try again by starting the watch from the beginning. This can happen if the
                // watch goes bad.
                LOGGER.debug("Received NULL item while watching node type [%s]. Restarting watch.", this.nodeRole);
                return;
              }```
I'm not sure what "watch goes bad" means. But the logic does `return` from a method that is processing an iterator. Is it possible there is more stuff in the iterator that we're now ignoring, like perhaps another `WatchResult.DELETED` that we'll miss?

btw, I just wrote this comment on your other PR: <https://github.com/apache/druid/pull/13175#pullrequestreview-1138158359>. It should be re-branched onto `master` &amp; something seems wrong with the build

That really clarifies the process for me, thank you!  I am also suspecting the k8s deletion is misfunctioning.  I can debug that directly now that I know where to look.

FYI: I found some hidden issues in the move to k8s api v16, so I am working with 11.0.3 (latest v11 with security patches).  I will fix the bugs with deleted nodes and anything else I uncovered against 11.0.3, then do the upgrade separately afterwards to decouple the changees.

ah, makes sense. a comment on GitHub about that would be great, in case someone only follows that conversation

thank you for digging into this!

No worries, glad to help, and I'm enjoying the deep dive.

Thanks for the reminder, <@U03149ALHN3>. Will try to take a look this week.

<@U03149ALHN3> the PR looks good to me for the most part though I don't follow the examples in the docs. I have posted the comments in the PR itself

my comments are also 100% doc related at this point -- the rest lgtm

<https://github.com/apache/druid/pull/13027#pullrequestreview-1162701634>

<@U0344FW86DD> Addressed! I got an unrelated Overlord timeout issue.

<@U03149ALHN3> - some examples use `**` in `objectGlob` and some use `*` . e.g. there is `**.parquet` and there is also `*.json` .  I might be missing something.

Checking… I thought I searched and replaced everything 

I too see a lot of `"objectGlob": "*.json"` remaining in the docs, these should all be `"objectGlob": "**.json"`

Thanks, Didip. While you are at it, can you also check if the description of `objectGlob` is same across all examples?

<@U031A9D19NU> I believe I took cared of everything now

<@U031A9D19NU> <@U0344FW86DD> I think I fixed all the docs now.

Screen Shot 2022-11-03 at 8.07.56 AM.png

just merged this, but belatedly realized that Travis did not run for some reason: <https://github.com/apache/druid/pull/13302/checks>

let's all keep on the lookout for ZK/Curator issues in other PRs, since that may potentially indicate some problem with the Curator upgrade

noticed this issue: <https://github.com/apache/druid/pull/13306>

<https://github.com/apache/druid/pull/13315> fixes the flaky UT: testResumeAllActivelyReadingTasks

Ugh, nevermind. I should have looked closer.
`VectorObjectSelector` is what I was looking for.

yep was just about to say that. it is supported through VectorObjectSelector

Yeah, makes sense. Only core extensions should require being marked as "experimental".
contrib extensions are kind of experimental by default. So maybe we should just add a single line that clarifies as much, and then we can skip listing the individual contrib extensions.

On that note, is there some other place which lists the contrib extensions? I couldn't find it in the docs.

this is the list <https://druid.apache.org/docs/latest/development/extensions.html>

yeah, my feeling is contrib extensions should be assumed to be not fully supported, inherently

so no reason to call them out specifically

in fact, calling some out might be confusing.

because it might suggest _some_ contrib extensions _are_ fully supported

which is not true :slightly_smiling_face:

I think some folks are working on it -- I've seen newer PRs do GitHub Actions for some of the test groups (although Travis is still there too)

Currently few static-checks are already migrated from Travis to GitHub Actions. There is a PR ready that migrates rest of the static-checks and removes them from Travis. After that python scripts conversion that limit checks for only certain code changes and migrating unit tests. If I'm not wrong Elliott is working on migrating ITs.

Awesome! Thank you for working on this. It will help everyone’s productivity for sure.

build failed w/

```An exception has occurred in the compiler (1.8.0_352). Please file a bug against the Java compiler via the Java bug reporting page (<http://bugreport.java.com>) after checking the Bug Database (<http://bugs.java.com>) for duplicates. Include your program and the following diagnostic in your report. Thank you.
[INFO] -------------------------------------------------------------
Error:  COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
Error:  An unknown compilation problem occurred```

```java.lang.NullPointerException
	at com.sun.tools.javac.jvm.Code.emitop1w(Code.java:962)
	at com.sun.tools.javac.jvm.Items$LocalItem.load(Items.java:399)
	at com.sun.tools.javac.jvm.Gen.genArgs(Gen.java:966)
	at com.sun.tools.javac.jvm.Gen.visitNewClass(Gen.java:1935)
	at com.sun.tools.javac.tree.JCTree$JCNewClass.accept(JCTree.java:1516)
	at com.sun.tools.javac.jvm.Gen.genExpr(Gen.java:947)
	at com.sun.tools.javac.jvm.Gen.genArgs(Gen.java:966)
	at com.sun.tools.javac.jvm.Gen.visitNewClass(Gen.java:1935)
	at com.sun.tools.javac.tree.JCTree$JCNewClass.accept(JCTree.java:1516)```

not every day you get a java compiler bug

it works on java 17, though :disguised_face:

next PR for removing java 8 and java 11 support? :stuck_out_tongue:

ah, figured out that it didn't like this inner class: <https://github.com/apache/druid/pull/13506/commits/7b6c47a02493ec9552b990788bc062321036f059>

<@U03M03LA544> has been doing a bunch of looking into this

you can read about the adventure on <https://github.com/apache/druid/pull/12032>

i think where things are at now is there are some funny test issues related to memory pooling

<https://github.com/apache/druid/pull/12032#issuecomment-1251607235>

There’s a bunch of new MSQ code that needs to be ported to Netty 4 too :)

Mostly the new mock clients for testing. 

&gt; MSQ code that needs to be ported to Netty 4
if you point me in the right direction I can get started

MSQTestBase is the class where all the stuff happens.

```/**
 * Base test runner for running MSQ unit tests. It sets up multi stage query execution environment
 * and populates data for the datasources. The runner does not go via the HTTP layer for communication between the
 * various MSQ processes.
 *
 * Controller -&gt; Coordinator (Coordinator is mocked)
 *
 * In the Ut's we go from:
 * {@link MSQTaskQueryMaker} -&gt; {@link MSQTestOverlordServiceClient} -&gt; {@link Controller}
 *
 *
 * Controller -&gt; Worker communication happens in {@link MSQTestControllerContext}
 *
 * Worker -&gt; Controller communication happens in {@link MSQTestControllerClient}
 *
 * Controller -&gt; Overlord communication happens in {@link MSQTestTaskActionClient}```


I updated <https://github.com/apache/druid/pull/12032> so all the code compiles and unit tests pass.

still seeing various new failures, and some OOM issues in some integration tests

<@U0315MQKEHK> are you still looking into helping out with this?

Just saw this thread. <@U03M03LA544> What kind of help do you need ?

raised a PR here so it can run through the tests: <https://github.com/apache/druid/pull/13544>

Sounds like a good improvement request would be to considers all  source columns for the parquet reader including any used in `transforms`.  Would you like to write one up here: <https://github.com/apache/druid/issues>

Just want to confirm if my understanding is correct and/or if anyone ran into similar issue with hadoop ingestion on parquet file.

I haven’t use hadoop / parser in a long time haha

That's more than I've used it :wink: , but you seem to have uncovered a problem with it, so it would be good to document.

Why use hadoop when we have MSQE right? :joy:

Exactly! :smile: I wonder what the behavior is with MSQE + Parquet field selection... sounds like an interesting test.

i vaguely remember this code, iirc part of the problem is that transforms happen on top of stuff so the part that handles the flattenSpec doesn’t have access to which columns are needed, only the final set of output dims

native inputFormats sort of have this problem too in that there is no way to know the full set of columns required to be read from the input source, at least from within the format

we might want to pass the transformSpec in .. somewhere

for parquet it is more of a problem than others because it tries to trim the set of columns it reads because its format makes stuff a bit expensive to read stuff you don’t need, so its trying to avoid waste

I took a look at the parquet inputFormat and it doesn’t have this problem because it just read all the columns of the parquet file. It doesn’t even consider any of the Druid’s spec (dimensionSpec, metricSpec, etc.)

I am looking at ParquetReader#intermediateRowIterator
```final org.apache.parquet.hadoop.ParquetReader&lt;Group&gt; reader;

reader = closer.register(org.apache.parquet.hadoop.ParquetReader.builder(new GroupReadSupport(), path).withConf(conf).build());```

The reader (that is reading in the parquet file) is built without any of Druid’s spec (dimensionSpec, metricSpec, etc.)

The conf in the withConf is hadoop-common Configuration class (which has nothing to do with Druid’s spec)

I think one way we can fix the Parquet Parser is to just return fullSchema

similar to what we already do if parseSpec instanceof ParquetParseSpec &amp;&amp; flattenSpec != null

Although the `HadoopDruidIndexerConfig` (which we already gotten from
`HadoopDruidIndexerConfig.fromConfiguration(context.getConfiguration());`),
does have `DataSchema` spec. So we can get `transformSpec`. Although not sure how easy it is to get the source field from the `TransformSpec`. Took a quick look and maybe the `RowFunction` inside the `Transform` would need to expose a new method. In this new method, we would call `analyzeInputs` of the `expr`  getting back `BindingAnalysis`. Then we can call `getRequiredBindings` . Maybe?

&gt; I took a look at the parquet inputFormat and it doesn’t have this problem because it just read all the columns of the parquet file.
oh, that’s kind of sad

it does seem useful for the hadoop reader and inputFormat both to be able to know the full set of required columns that are needed

need to look at some stuff to see if there is an obvious way to do this

&gt; it does seem useful for the hadoop reader and inputFormat both to be able to know the full set of required columns that are needed
I agree. However, i think the most important thing is to fix hadoop ingestion of parquet file for the case described above. If it is not too much work, we can do the right fix (correctly getting required columns) straight away.

lmk if you have a chance to look at RowFunction/BindingAnalysis in the Transform

Also I think the flatten spec is applied after reading from parquet

so the columns used in flattenSpec (if not part of the dimensionSpec) must also be added to the set of required columns

oh actually we currently return fullSchema if there is a flattenSpec.

we can think if we want to optimize this too

```// this is kind of lame, maybe we can still trim what we read if we
// parse the flatten spec and determine it isn't auto discovering props?
if (parseSpec instanceof ParquetParseSpec) {
  if (((ParquetParseSpec) parseSpec).getFlattenSpec() != null) {
    return fullSchema;
  }
}```

I put up a fix for this at <https://github.com/apache/druid/pull/13612>, in case anyone is interested

Screen Shot 2022-12-15 at 10.03.29 PM.png

Screen Shot 2022-12-15 at 7.12.53 PM.png

these clusters don’t have identical query patterns so don’t have an exact comparison on query times, but nothing sticks out as bad on the cluster with the `frontCoded` data

Hello Devs,

I have a unique requirement where we need to query data for last n days in druid, but the rows have a column which defines the timezone that  row belongs to . So querying on one timezone is ok as druid can do the timeshift for us, but how to achieve this when  data needs to be aggregated per timezone, in query.  Looking forward to any suggestions.

was there ever a repo? It's created locally while tests are being run.

I got this error:
```Pulling druid-it-hadoop (druid-it/hadoop:9.9.9)...
pull access denied for druid-it/hadoop, repository does not exist or may require 'docker login': denied: requested access to the resource is denied```
When trying to run hadoop IT

ah yep you are right. forgot to add -Ddocker.build.hadoop=true

The NPE that is produced
```java.lang.NullPointerException: null
        at org.apache.druid.client.HttpServerInventoryView$2.toDruidServer(HttpServerInventoryView.java:181) 
        at org.apache.druid.client.HttpServerInventoryView$2.lambda$nodesRemoved$1(HttpServerInventoryView.java:163)```
comes from
```              private DruidServer toDruidServer(DiscoveryDruidNode node)
              {

                return new DruidServer(
                    node.getDruidNode().getHostAndPortToUse(),
                    node.getDruidNode().getHostAndPort(),
                    node.getDruidNode().getHostAndTlsPort(),
                    ((DataNodeService) node.getServices().get(DataNodeService.DISCOVERY_SERVICE_KEY)).getMaxSize(),
                    ((DataNodeService) node.getServices().get(DataNodeService.DISCOVERY_SERVICE_KEY)).getServerType(),
                    ((DataNodeService) node.getServices().get(DataNodeService.DISCOVERY_SERVICE_KEY)).getTier(),
                    ((DataNodeService) node.getServices().get(DataNodeService.DISCOVERY_SERVICE_KEY)).getPriority()
                );
              }
            }```
Specifically, `((DataNodeService) node.getServices().get(DataNodeService.DISCOVERY_SERVICE_KEY))` returns null for the `PEON` as its `services` map is empty.

I am  now stuck trying to determine why the `PEON` event is causing the problem.

I have determined that the underlying issue with the k8s extension is that  the `PEON` and `INDEXER` services that are announced by an indexer pod when it starts up are both on the same host/port.  This results in a collision in the `DruidNodeDiscoveryProvider` `serviceDiscoveryMap` which uses the host and port string as a key.  I patched the k8s extension to prevent the notification of the downstream listeners if a pod deletion arrives for a druid node with no services.  That prevents the issue, but makes me wonder if this `PEON` should even be getting announced.  It has no services declared, so it can't be assigned any work.  If this `PEON` announcement is the issue then the real fix is prevent it.
Is it expected that the indexer announces a peon with no services at startup?   If yes, what is the purpose of this peon.  If no then I can determine why the k8s extension is making this announcement.

At indexer starup the coordinator sees the following
```│ 2023-01-11T22:02:02.433753502Z [K8sDruidNodeDiscoveryProvider-ListenerExecutor] Node[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/indexer', host='10.0.183.191', bindOnHost=false, port=-1, plaintextPort=8091, enablePlai │
│ ntextPort=true, tlsPort=-1, enableTlsPort=false}, nodeRole='PEON', services={}}] discovered but doesn't have service[lookupNodeService]. Ignored.                                                                                       │
│ 2023-01-11T22:02:02.434313196Z [K8sDruidNodeDiscoveryProvider-ListenerExecutor] Node[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/indexer', host='10.4.151.172', bindOnHost=false, port=-1, plaintextPort=8091, enablePlai │
│ ntextPort=true, tlsPort=-1, enableTlsPort=false}, nodeRole='PEON', services={}}] discovered but doesn't have service[lookupNodeService]. Ignored.```
Indicating that the peon that has been announced is essentially invalid.  This leads me to believe that either the peon announcement should not occur or that it is missing information.

```
[main] Announcing DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/indexer', host='10.0.96.19', bindOnHost=false, port=-1, plaintextPort=8091, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeRole='INDEXER', services={dataNodeService=DataNodeService{tier='_default_tier', maxSize=0, serverType=indexer-executor, priority=0}, workerNodeService=WorkerNodeService{ip='storage--druid-indexer-577957494-zhrhs', capacity=128, version='0', category='_default_worker_category'}, lookupNodeService=LookupNodeService{lookupTier='__default'}}}]
[main] Announced DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/indexer', host='10.0.96.19', bindOnHost=false, port=-1, plaintextPort=8091, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeRole='INDEXER', services={dataNodeService=DataNodeService{tier='_default_tier', maxSize=0, serverType=indexer-executor, priority=0}, workerNodeService=WorkerNodeService{ip='storage--druid-indexer-577957494-zhrhs', capacity=128, version='0', category='_default_worker_category'}, lookupNodeService=LookupNodeService{lookupTier='__default'}}}]
[main] Announcing DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/indexer', host='10.0.96.19', bindOnHost=false, port=-1, plaintextPort=8091, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeRole='PEON', services={}}]
[main] Announced DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/indexer', host='10.0.96.19', bindOnHost=false, port=-1, plaintextPort=8091, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeRole='PEON', services={}}]```
Above is the announcements inside the indexer.  Note that the `PEON` announcements have `services={}`

Definitely good to fix this, my understanding is limited, but aren't Peons supposed to communicate on different ports configured with `druid.indexer.runner.startPort` and `druid.indexer.runner.endPort`. I'm not sure what the Peon announcement is for.
On a side note, have you looked at the new <https://druid.apache.org/docs/latest/development/extensions-contrib/k8s-jobs.html|MM-less K8s extension >`druid-kubernetes-overlord-extensions` to avoid the need for MMs completely? It's new and experimental, but sounds like it would fit your deployment well.

I tried setting the port range for the runners explicitly to `8100-8103` but the announced peon still uses `8091`, which is why Im really suspicious of it.

I am super interested in this extension, I'm definitely going to look into running it

I have created a PR to upstream master to patch the bug in the kubernetes extensions and would greatly appreciate a review <https://github.com/apache/druid/pull/13667>

&gt;  I have determined that the underlying issue with the k8s extension is that  the `PEON` and `INDEXER` services that are announced by an indexer pod when it starts up are both on the same host/port.
why does this happen, though? two services can't start on the same host and port

This is the announcement sequence inside the indexer
```[main] Announcing DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/indexer', host='10.0.206.35', bindOnHost=false, port=-1, plaintextPort=8091, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeRole='INDEXER', services={dataNodeService=DataNodeService{tier='_default_tier', maxSize=0, serverType=indexer-executor, priority=0}, workerNodeService=WorkerNodeService{ip='storage--druid-indexer-76c796f796-q4mlf', capacity=128, version='0', category='_default_worker_category'}, lookupNodeService=LookupNodeService{lookupTier='__default'}}}]

[main] Announced DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/indexer', host='10.0.206.35', bindOnHost=false, port=-1, plaintextPort=8091, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeRole='INDEXER', services={dataNodeService=DataNodeService{tier='_default_tier', maxSize=0, serverType=indexer-executor, priority=0}, workerNodeService=WorkerNodeService{ip='storage--druid-indexer-76c796f796-q4mlf', capacity=128, version='0', category='_default_worker_category'}, lookupNodeService=LookupNodeService{lookupTier='__default'}}}]

[main] Announcing DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/indexer', host='10.0.206.35', bindOnHost=false, port=-1, plaintextPort=8091, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeRole='PEON', services={}}]

[main] Announced DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/indexer', host='10.0.206.35', bindOnHost=false, port=-1, plaintextPort=8091, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeRole='PEON', services={}}]```
Note that I have the peon port explicitly set to `8100`
```  druid_indexer_runner_startPort: "8100"
  druid_indexer_runner_endPort: "8100"```

I;m trying to determine where in the source the indexer process is getting started so I can debug why the port is incorrect for the peon

I tried this locally and I don't see an announcement for `peon`. I tried with curator node discovery but that shouldn't matter. what kind of indexing task are you running?

There are no tasks running, this occurs at startup.

Above is the full logs including the announcement of the peon with no services.

<@U030MNRGBNG> - FYI I recently learned that the problem you encountered is fixed by <https://github.com/apache/druid/pull/12640> (24.0 release)

<@U030C8H59T8> also added a safety check anyway here in this PR - <https://github.com/apache/druid/pull/13930>

how are you diagnosing it as a performance issue? It could be a functional issue as well. are some hand offs succeeding?

Yes when the volume is less , hand off gets succeeded
so thought that its not able to handle volume during peak hours

I was wondering if someone with merge privileges could merge in <https://github.com/apache/druid/pull/12032>

I figured it out.  sigar and zstd-jni need to installed into `/Library/Java/Extensions` and in my case I needed `zstd-jni-1.5.2-5-darwin_aarch64.jar` to match the M2 architecture

Still didn't get it working.  I ended up skipping the specific tests that use this library.  To do that I added
```-Dtest='!org.apache.druid.java.util.common.CompressionUtilsTest, !org.apache.druid.java.util.metrics.*Test'```
to the maven command.

is this specific to M2? does it work on M1?

I don't know, I only have an M2 and no way to test on and M1.

This issue seems to be occurring due to libsigar not providing aarch64 based binaries. When I faced this issue in the past what worked is changing jdk from aarch64 based to x86 one (maybe Rosetta - the translation environment of mac m1 is allowing x86 arch based jdk to run x86_64 based libsigar-universal64-macosx.dylib file).

This issue seems to occur for quite some time on both master and PR builds. Upon initial scanning, it seems like the system can't connect to ubuntu servers (maybe due to network issues?). Details here - <https://app.travis-ci.com/github/apache/druid/jobs/593243289>

I only run the unit test for the classes I touched locally.

Running all unit tests can take a long time, so it's best to do a small validation first, and then run all unit tests for the extension or integration tests that use the extension

specifically -- i am going to reintroduce a <https://github.com/apache/druid/pull/12109|PR 12109> that will optimize this path and depending on what other changes are happening, they might overlap or be more optimizations

i opened this PR
<https://github.com/apache/druid/pull/13710>

cc <@U035FNDU6HH> <@U030GL518NR>

it looks like we could maybe eliminate the lock entirely but we'd need to know what the plan is for that part of the codebase to do it safely

Thanks <@U030C8H59T8>  that explains the continued use of locks;

I will pull forward from 12109 the stamped lock implementation too as that was significant improvement

Any idea when would `druid.indexer.tasklock.forceTimeChunkLock` come out of its experimental status? I wonder if there is an effort towards that.

Thanks for the update, looking forward for that. The minor compaction usecase is why I was curious about this config. It would be useful for compacting real time segments when there is late arriving data