Apache Druid

Yes, compaction does download all the segments for an interval to get the metadata before it can proceed. Thus, it faces the risk of running out of disk space. I think there is a GitHub issue for an improvement to avoid downloading the segments.

Auto-compaction also launches separate tasks for each interval grain, which is certainly not needed while doing a re-index.

I wonder if auto-compaction could just submit a Druid re-index task instead of using a specialized CompactionTask.

I think that would be way better, and the code will be more general as well.
Perhaps, in the future CompactionTask can be deprecated and removed, just tell all customers to migrate to Druid ingestion.

Compaction task has the advantage of user not needing to specify all the specs (dimensions, metrics, etc) and is the reason it needs to download all the segments for an interval

So, after the segments are downloaded and the specs are determined, it effectively becomes a regular re-indexing task, right? 
If that's the case, the right direction for now would be to figure out a way to have enough metadata that helps us eliminate or reduce the segment downloads needed to figure out the specs.

yes, definitely; for this reason it's related to the catalog effort

catalog, if used, would provide us with enough info to do a compaction without inspecting all segments first

ah, i see. If I re-ingest from Druid datasource then I need to describe the dimensions and metrics explicitly?
I was about to ask if there’s a property that I can pass to Druid ingestion spec to just use all the dimensions and metrics as-is.

`So, after the segments are downloaded and the specs are determined, it effectively becomes a regular re-indexing task, right?`-&gt; yes
`If that's the case, the right direction for now would be to figure out a way to have enough metadata that helps us eliminate or reduce the segment downloads needed to figure out the specs.`-&gt; yes, one way is to store the ingestionSpec used for the creation of the segment in the metadata store. This is similar to storing lastCompactionState that we store for all segment created by a compaction job. Another way is the catalog effort

`ah, i see. If I re-ingest from Druid datasource then I need to describe the dimensions and metrics explicitly?` -&gt; yes, if you forget to explicitly specify any metric or dimension then you can lose data!
`I was about to ask if there's a property that I can pass to Druid ingestion spec to just use all the dimensions and metrics as-is.`-&gt; that’s what the (Auto)Compaction is there for.

Sorry for this dumb question, how come the Console UI is able to know all of the columns when the “wizard” is helping the user craft the ingestion JSON?

<@U030W50CNAC> knows something compaction job don’t :smile:

The console uses the `sampler` API. It samples the data for any inputSource :stuck_out_tongue:

so, it could miss some columns, potentially, if your segments have different schema from segment to segment

ooo gotcha, glad that I didn’t YOLO this Druid re-ingestion to prod.

Hm… actually, our group always versioned the ingestion spec in Git. This may still work.

Have you tried `EnvironmentVariableDynamicConfigProvider`?
<https://druid.apache.org/docs/latest/operations/dynamic-config-provider.html#environment-variable-dynamic-config-provider>

Do I define this `druid.dynamic.config.provider` on common config?

ooo it’s in the `consumerProperties` . Should have read further.

Another question, can I pass the dynamic variable to other params in the form of $var? example:

```"consumerProperties": {
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "SCRAM-SHA-512",
    "sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"$username\" password=\"$password\";"            
    "bootstrap.servers": "<http://broker-bootstrap.example.com:443|broker-bootstrap.example.com:443>",
    "druid.dynamic.config.provider": {
        "type": "environment",
        "variables": {
            "username": "USERNAME"
            "password": "PASSWORD"
        }
    }                
},```


ah… I have to put the entire `sasl.jaas.config` in the dynamic environment.

what is Bitnami zookeeper? I prefer zookeeper clients connect to the servers directly instead of through an LB. With the latter, the client sees just one server. If the request from LB to the backend fails, then there isn't a fallback for the client.  This is based on my experience running a zookeeper cluster some years ago. Things might have changed since then.

<https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ>
<https://dev.zookeeper.apache.narkive.com/KPQWwpjQ/zookeeper-behind-load-balancer>

these two links throw more light. Based on these, I can say how quickly and reliably can LB take out a bad zk node.

Bitnami (a company) does a lot of Docker image packaging, they created a docker image for Zookeeper, and it is very popular. Most people who deploy Zookeeper on Kubernetes tend to use Bitnami’s Docker image.

wow, these docs are very useful. Thank you!

I also remembered that ZK clients have a lot of logic on their own to load balance amongst themselves.
Thanks for helping me double check.

&gt;  so I'm assuming some setting or default may have changed between 0.23.0 and 24.0.0
That's possible. A flavor of 24.0.0 is running on many of our prod clusters and we haven't seen such a drop in throughput.

You could take stack traces of a slow task to see if it's doing something expensive.

Can you walk me through doing that (or point me in the right direction at least)? We're running druid containerised within kubernetes

one way is to issue `-3` signal via `kill -3` command to the task. so if task pid is 2, you can do `kill -3 2` in the container. This will result in thread dump getting logged in the task logs. Then you can look at the stack trace in the task logs.

this technique is also useful: <https://support.imply.io/hc/en-us/articles/360033747953-Profiling-Druid-queries-using-flame-graphs>

you would want to capture them from the task JVMs (`internal peon`)

if you post the `.std` or `.html` files we can take a look

Apologies for digging up an old thread but I've just got into a position to look at this again.

Within the peon logs, we are seeing (debug level) fetch timeouts to kafka that appear to coincide with the drop in ingest (feels kinda obvious). I'm keen to work out what's causing these as the kafka cluster itself is running fine with no errors in the logs. I'm assuming the peon task might be blocking on something which is causing the fetch request to timeout somehow.

I'm unable (easily) to export the flame graph from site.  I have a 5 minute sample of an indexing task misbehaving so I'm keen to learn how to analyze and understand the process.

For kafka ingest, what do I need to look for in the flame graph (specific threads and states etc)?

there will be a `task-runner-*` thread that's particularly important.

in case of `LOST_CONTACT_WITH_STREAM`, you can see the actual exception by looking at the supervisor report: <https://druid.apache.org/docs/latest/development/extensions-core/kafka-supervisor-operations.html#getting-supervisor-status-report>

what do you mean by "segment merge wait time"?

we have data in S3. When we query in druid, segments gets loaded from there to disk and from disk to memory right, if i am not wrong?  We want to reduce the time to fetch while querying the data in terms of that. we are using i3.4x nodes(30). wanted to check if there are few druid config properties that can be tweaked to achieve better performance.

<@U043WQK8XDJ> Segments are fetched from s3 when the historicals are coming up not at query time

So are you trying to optimize on query time or starting the historical faster

Please share the query.
The segment shard spec
and the number of segments

What do you think is getting bottlenecked

image.png

<@U0315MQKEHK> sorry for late response, as i was out for few days. single segment for an hour, 3 replicas, total segment for a datasource - ~10k .  these are select queries with zeroFilledTopN or groupby

Can you please share the metrics underneath the graph ?
<https://druid.apache.org/docs/latest/operations/metrics.html#historical>
<https://druid.apache.org/docs/latest/operations/metrics.html#historical-1>
It would be present from the above list

there isn't an API to change it, but you can edit it directly in pgsql if you are willing to go :face_with_cowboy_hat: mode

just make sure to suspend the supervisor first

also, wondering why an outage would cause you to need to ingest at an earlier offset? the ingest is transactional (offsets are written along with segment metadata), so if you have a druid outage then we will resume reading from where we left off

I am just drawing up worst case scenario :smile:

I think this stems from a known issue in Druid 24.0.0

The fix for the same has been done here, <https://github.com/apache/druid/pull/13138> and should be present in 24.0.1 release of Druid.

we're kicking off the 24.0.1 release process soon; you can follow it in email on <mailto:dev@druid.apache.org|dev@druid.apache.org> or at <https://lists.apache.org/thread/17jh8yf0tch9m4l1o6mdkjf144bff015>. although if you use web interface note that the release process does involve creating new threads from time to time, so you should check the index <https://lists.apache.org/list.html?dev@druid.apache.org> too

i fixed this by building from source druid-indexer-service and replacing the jar on the lib

Capture d’écran 2022-10-18 à 18.11.44.png

And here is the Coordinator dynamic config :
```{
  "millisToWaitBeforeDeleting": 900000,
  "mergeBytesLimit": 524288000,
  "mergeSegmentsLimit": 100,
  "maxSegmentsToMove": 1000,
  "percentOfSegmentsToConsiderPerMove": 100,
  "useBatchedSegmentSampler": true,
  "replicantLifetime": 15,
  "replicationThrottleLimit": 1000,
  "balancerComputeThreads": 200,
  "emitBalancingStats": false,
  "killDataSourceWhitelist": [],
  "killPendingSegmentsSkipList": [],
  "maxSegmentsInNodeLoadingQueue": 1000,
  "decommissioningNodes": [],
  "decommissioningMaxPercentOfMaxSegmentsToMove": 70,
  "pauseCoordination": false,
  "replicateAfterLoadTimeout": false,
  "maxNonPrimaryReplicantsToLoad": 2147483647
}```

It is okay if there are always items in the load queue. This is most likely because of balancing. The coordinator constantly tries to find the optimal home for segments for improved query performance. So you will always see some segments moving around.

You can verify this by looking at the metric `segment/moved/count` . It denotes the number of segments chosen for balancing in a coordinator cycle. The other metric to look at would be `segment/assigned/count`. This is the number of segments that were missing from historicals and were assigned for loading. The assigned count is high when there are new published segments in the system that must be made available on historicals or when segments are being moved across tiers (if you have multiple tiers).

Hello, thanks for answer. But I’m often seeing unavailable segments which is annoying for some internal teams.

Did you start seeing this issue after upgrading to 24 or after using `cachingCost`?

Currently, i have an avg of 45 segments unavailable per day. Before `cachingCost` it was 6

And the segments I’m talking about are not fresh ones

Hmm, that is problematic. Have you tried increasing the load queue size? `maxSegmentsInNodeLoadingQueue`

I noticed that historicals and coordinator was not using a lof of resources when balancing was happening

Do you have an idea of the metric `coordinator/time` before and after the change?
With the filter `duty = org.apache.druid.server.coordinator.duty.RunRules`

I want to see if the coordinator runs are taking more time after you started using cachingCost strategy.

It decreased after changing to `cachingCost` and got more “stable”

That's good. `cachingCost` is supposed to be faster.

How does `segment/assigned/count` look? Are we assigning enough segments in each run for loading?

Hmm, then I am inclined to believe that the issue is probably not with `cachingCost`. Because the number of segment assignments is really the only thing that can be affected by the strategy. Since the runs are also faster and the number of assignments is okay, the issue is something else.

Is the `segment/assigned/count` much smaller than the load queue size? If not, you could consider increasing it further if you have more segments being published.

Thanks a lot for taking the time to troubleshoot this :slightly_smiling_face:
I’m gonna check this tomorrow and post results in the thread

Hello :slightly_smiling_face:
The `segment/assigned/count` is smaller than the `segment.loadQueue`

Hi :slightly_smiling_face: . Hmm, if the `segment/assigned/count` is always smaller than `maxSegmentsInNodeLoadingQueue`, then we don't need to increase it further.

Do you see anything in the logs for the segments that are unavailable?

```coordinator.20221012.log:2022-10-12T14:16:51,586 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - [_default_tier]: Balancer move segments queue has a segment stuck: {class=org.apache.druid.server.coordinator.duty.BalanceSegments, segment=xxxx_2021-11-19T00:00:00.000Z_2021-11-20T00:00:00.000Z_2021-11-30T17:20:23.435Z_6, server=DruidServerMetadata{name='druid-historical-xxx:8083', hostAndPort='druid-historical-xxx:8083', hostAndTlsPort='null', maxSize=6000000000000, tier='_default_tier', type=historical, priority=0}}```


And btw, yesterday I’ve switched back to `cost` strategy and I’m not seeing those unavailable segments anymore

This seems to be a stuck balancing move, this should not cause segment unavailability.

&gt; And btw, yesterday I’ve switched back to `cost` strategy and I’m not seeing those unavailable segments anymore
Interesting. `cachingCost` does have some cost computation-based issues but it shouldn't really cause unavailability of any segment.

The load rules would also be printing logs like `Current replication: [...]` . Do you see logs like this for the segment in question?

INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Dropping segment

It looks like a same segment was present on multiple historicals (no replication set for this datasource)

The system `server_segments` table contains that information:
```SELECT *
FROM sys.server_segments ss 
WHERE segment_id IN (SELECT segment_id FROM sys.segments WHERE datasource='&lt;datasource name&gt;')```


That query will give you all the segment_ids and their current historical(s) for a given data source. You can change the condition for a specific segment_id.

sorry, I post to the wrong thread. This is meant to the next thread.

Are all segment intervals in that table on 2021-06-29?

One segment in 2021-06-29 should have 500M data at least

In theory, all rows within that time interval should be returned. Have you compared rows from a `SELECT count(*)` for the same time interval and the number of rows returned to your result file? You might need to remove the "pretty" to get one row per line in your result.
Is this a clustered deployment, is that URL the broker? I'm just checking that you are not issuing the request to a historical in a cluster of multiple historicals.

Screen Shot 2022-10-18 at 4.37.27 PM.png

This showed that on 2021-06-29, there are multiple segments with 500M data and 1M rows.

I can confirm this request is to the router. In fact, this cluster has only one broker and one historical.

if there is some error (or timeout) during query processing, then the response will be cut off early

Just making sure,... the "size" column in the result you are showing is the size in bytes of the segment partition, does the num_rows column in that result have the numbers you expect?

The response may be cut out. There is only one line here for this 70M file. Let me retry use "list" instead of "compact list"

Actually, without "?pretty", it is one line. Let me use "?pretty" to confirm if the response is "cut off"

Yes, the response is cut off. Bellow is the last 100 lines of the response file.
```}, {
    "__time" : 1624925857991,
    "dc" : "pzh",
    "device" : "<http://stmfa1la2-lapp1-3-prd.eng.sfdc.net|stmfa1la2-lapp1-3-prd.eng.sfdc.net>",
    "env" : "perf",
    "eventType" : "transaction",
    "name" : "xosvvrnw",
    "orgId" : "org-hxpijeqprl",
    "pod" : "pod-baz",
    "superPod" : "superpod-pt",
    "userId" : "04158427",
    "sampled" : "false",
    "serviceName" : "scrt",
    "traceId" : "0000000000000000000000c08ca0c8ec",
    "spanId" : "00004ada0c3f40e5",
    "error" : "false",
    "errorMessage" : "errMsg:-tlksffquufvfwzpftgytwswfyomcea",
    "errorType" : "rmava",
    "httpStatusCode" : "404",
    "uri" : "/zyppowbiultrgdv",
    "transactionType" : "hybernate",
    "closed" : "true",
    "closingTimestamp" : "1674363913877",
    "dbCallCount" : 41962,
    "duration" : 335132,
    "dbCallDuration" : 8891576,
    "externalCallCount" : 62862,
    "externalCallDuration" : 4186772
  }, {
    "__time" : 1624925858001,
    "dc" : "fqo",
    "device" : "<http://stmfa1la2-lapp1-1-prd.eng.sfdc.net|stmfa1la2-lapp1-1-prd.eng.sfdc.net>",
    "env" : "perf",
    "eventType" : "transaction",
    "name" : "upmaaiuc",
    "orgId" : "org-xibawqscka",
    "pod" : "pod-umq",
    "superPod" : "superpod-vy",
    "userId" : "% ```

&gt; Just making sure,... the "size" column in the result you are showing is the size in bytes of the segment partition, does the num_rows column in that result have the numbers you expect?
No, it does not have the number of lines I expect. Way lower

Wondering is this due to the druid query context time out? or curl time out? I think curl timeout is unlikely. This always around 22 secs.

Do you see any errors in the historical or broker logs?

Let me check the logs. Here, I did see `druid.server.http.defaultQueryTimeout` is set as `druid.server.http.defaultQueryTimeout=600000`

This is 10 minutes query timeout. It does not seem to apply here.

Here is the error logs from Druid brokers:
```2022-10-19T00:05:29,867 ERROR [qtp1727196188-246] org.apache.druid.server.QueryResource - Unable to send query response. (com.fasterxml.jackson.databind.JsonMappingException: Query[0b6845d6-714b-46ff-a495-bb4bef152d74] url[<http://10.64.210.125:8088/druid/v2/>] failed with exception msg [Channel disconnected])
2022-10-19T00:05:29,868 WARN [qtp1727196188-246] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [0b6845d6-714b-46ff-a495-bb4bef152d74] (com.fasterxml.jackson.databind.JsonMappingException: Query[0b6845d6-714b-46ff-a495-bb4bef152d74] url[<http://10.64.210.125:8088/druid/v2/>] failed with exception msg [Channel disconnected])
2022-10-19T00:05:29,869 WARN [qtp1727196188-246] org.eclipse.jetty.server.HttpChannel - handleException /druid/v2/ com.fasterxml.jackson.databind.JsonMappingException: Query[0b6845d6-714b-46ff-a495-bb4bef152d74] url[<http://10.64.210.125:8088/druid/v2/>] failed with exception msg [Channel disconnected]```

Here is the error logs from Druid historical:
```2022-10-19T00:05:28,864 ERROR [qtp2085313771-134[scan_[perf_data_1T_1]_0b6845d6-714b-46ff-a495-bb4bef152d74]] org.apache.druid.server.QueryResource - Unable to send query response. (org.eclipse.jetty.io.EofException)
2022-10-19T00:05:28,864 WARN [qtp2085313771-134] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [0b6845d6-714b-46ff-a495-bb4bef152d74] (org.eclipse.jetty.io.EofException)```

I'm<https://druid.apache.org/docs/latest/querying/troubleshooting.html#query-fails-due-to-internal-communication-timeout| reading> <https://druid.apache.org/docs/latest/querying/troubleshooting.html#query-fails-due-to-internal-communication-timeout|this> which seems relevant.

Let me get the router error messages to you too.

Do you see a TimeoutException in the Historical log?

```2022-10-19T00:05:30,959 ERROR [qtp774840504-131] org.apache.druid.server.AsyncQueryForwardingServlet - Exception handling request: {class=org.apache.druid.server.AsyncQueryForwardingServlet, exceptionType=class java.io.EOFException, exceptionMessage=HttpConnectionOverHTTP@698f503a::SocketChannelEndPoint@1c4942b6{l=/10.64.196.185:49932,r=/10.64.196.217:8088,ISHUT,fill=-,flush=-,to=0/900000}{io=0/0,kio=0,kro=1}-&gt;HttpConnectionOverHTTP@698f503a(l:/10.64.196.185:49932 &lt;-&gt; r:/10.64.196.217:8088,closed=false)=&gt;HttpChannelOverHTTP@11826201(exchange=HttpExchange@4d81d165{req=HttpRequest[POST /druid/v2/ HTTP/1.1]@5e9c566f[TERMINATED/null] res=HttpResponse[HTTP/1.1 200 OK]@896f3f8[PENDING/null]})[send=HttpSenderOverHTTP@a860a81(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@49cfe84d{s=START}],recv=HttpReceiverOverHTTP@755cf035(rsp=CONTENT,failure=null)[HttpParser{s=CLOSED,78527284 of -1}]], exception=java.io.EOFException: HttpConnectionOverHTTP@698f503a::SocketChannelEndPoint@1c4942b6{l=/10.64.196.185:49932,r=/10.64.196.217:8088,ISHUT,fill=-,flush=-,to=0/900000}{io=0/0,kio=0,kro=1}-&gt;HttpConnectionOverHTTP@698f503a(l:/10.64.196.185:49932 &lt;-&gt; r:/10.64.196.217:8088,closed=false)=&gt;HttpChannelOverHTTP@11826201(exchange=HttpExchange@4d81d165{req=HttpRequest[POST /druid/v2/ HTTP/1.1]@5e9c566f[TERMINATED/null] res=HttpResponse[HTTP/1.1 200 OK]@896f3f8[PENDING/null]})[send=HttpSenderOverHTTP@a860a81(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@49cfe84d{s=START}],recv=HttpReceiverOverHTTP@755cf035(rsp=CONTENT,failure=null)[HttpParser{s=CLOSED,78527284 of -1}]], query=ScanQuery{dataSource='perf_data_1T_1', querySegmentSpec=LegacySegmentSpec{intervals=[2021-06-29T00:00:00.000Z/2021-06-30T00:00:00.000Z]}, virtualColumns=[], resultFormat='list', batchSize=20480, offset=0, limit=9223372036854775807, dimFilter=null, columns=[], context={queryId=0b6845d6-714b-46ff-a495-bb4bef152d74}}, peer=127.0.0.1}
java.io.EOFException: HttpConnectionOverHTTP@698f503a::SocketChannelEndPoint@1c4942b6{l=/10.64.196.185:49932,r=/10.64.196.217:8088,ISHUT,fill=-,flush=-,to=0/900000}{io=0/0,kio=0,kro=1}-&gt;HttpConnectionOverHTTP@698f503a(l:/10.64.196.185:49932 &lt;-&gt; r:/10.64.196.217:8088,closed=false)=&gt;HttpChannelOverHTTP@11826201(exchange=HttpExchange@4d81d165{req=HttpRequest[POST /druid/v2/ HTTP/1.1]@5e9c566f[TERMINATED/null] res=HttpResponse[HTTP/1.1 200 OK]@896f3f8[PENDING/null]})[send=HttpSenderOverHTTP@a860a81(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@49cfe84d{s=START}],recv=HttpReceiverOverHTTP@755cf035(rsp=CONTENT,failure=null)[HttpParser{s=CLOSED,78527284 of -1}]]
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:385) ~[jetty-client-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1620) ~[jetty-http-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:269) ~[jetty-client-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:185) ~[jetty-client-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:80) ~[jetty-client-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.client.HttpReceiver.demand(HttpReceiver.java:118) ~[jetty-client-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.client.HttpReceiver$ContentListeners.demand(HttpReceiver.java:701) ~[jetty-client-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.client.ResponseNotifier.lambda$notifyContent$1(ResponseNotifier.java:139) ~[jetty-client-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.client.api.Response$AsyncContentListener.lambda$onContent$0(Response.java:192) ~[jetty-client-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.util.Callback$3.succeeded(Callback.java:142) [jetty-util-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.util.Callback$Nested.succeeded(Callback.java:285) [jetty-util-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.proxy.AsyncProxyServlet$StreamWriter.complete(AsyncProxyServlet.java:284) [jetty-proxy-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.proxy.AsyncProxyServlet$StreamWriter.onWritePossible(AsyncProxyServlet.java:265) [jetty-proxy-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.server.HttpOutput.run(HttpOutput.java:1541) [jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1525) [jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:600) [jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:439) [jetty-server-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.47.v20220610.jar:9.4.47.v20220610]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.47.v20220610.jar:9.4.47.v20220610]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]```

&gt;  Do you see a TimeoutException in the Historical log?
Do you have some string to search? In fact, I don't think so. Because I tried it one more time and there is the single historical. I did wait for the error message with a `tail -f` and the above are the only ones.

And the default for `druid.server.http.maxIdleTime` is 5 minutes, so that's not it either.

Just thinking out loud...Really seems like a network problem. Seems like the historical was attempting to return more results when the connection to the broker was lost.

Screen Shot 2022-10-18 at 5.23.11 PM.png

one thing that seems odd is that all the ports are set to 8088, not necessarily a problem because they are on different hosts, but this is unusual.

right. That is due to some historical reasons. As you said, this was not an problem that we are aware of.

Were you able to do this scan query before?

<@U030K4UM3H7>, is it that we are say the broker and historical connections somehow breaks? From this error message in broker:
```2022-10-19T00:05:29,867 ERROR [qtp1727196188-246] org.apache.druid.server.QueryResource - Unable to send query response. (com.fasterxml.jackson.databind.JsonMappingException: Query[0b6845d6-714b-46ff-a495-bb4bef152d74] url[<http://10.64.210.125:8088/druid/v2/>] failed with exception msg [Channel disconnected])```
Basically Broker complains it can't send query response to router due to `<http://10.64.210.125:8088/druid/v2/>` end point failed? And subsequent router exception is due to broker close the connection/ sending error to the router?

&gt; Were you able to do this scan query before?
This is for perf test. We did do quite some query before, but not this large scale scans.

At this point in the query, the historical seems to be returning results and half way through it encounters the
```Unable to send query response. (org.eclipse.jetty.io.EofException) ```
The broker then errors out right after that with a
```Query[0b6845d6-714b-46ff-a495-bb4bef152d74] url[<http://10.64.210.125:8088/druid/v2/>] failed with exception msg [Channel disconnected]```
I think the router error is just a side effect. You can prove this by submitting the request directly to the broker.
But we still don't know why the connection is being lost between historical and broker.

I actually did a little bit more testings.
• I used k8 port-forwarding to run this scan against the historical
    ◦ This turned out to last for over 3 minutes and streamed 1G data.
    ◦ The response json is also correctly formed at the last line
• I used k8 port-forwarding to run this scan agains the broker
    ◦ I got the same error in around 22 secs

This seems to be related to some broker/historical settings? The setting somehow breaks the historical and broker connection (maybe by closing it?) after 20 secs.

Both historical and broker can have `druid.server.http.defaultQueryTimeout` . Is it maybe lower on the brokers?

Just checked, both broker and historical, the setting is `druid.server.http.defaultQueryTimeout=600000`

Also, if this timeout is triggered, based on <https://druid.apache.org/docs/latest/querying/troubleshooting.html#query-fails-due-to-internal-communication-timeout|this>, it should have `java.util.concurrent.TimeoutException: Idle timeout expired:` in the historical log. And in my case, there is no such log.

In fact, what I have in historical is this `EofException`
```2022-10-19T00:05:28,864 ERROR [qtp2085313771-134[scan_[perf_data_1T_1]_0b6845d6-714b-46ff-a495-bb4bef152d74]] org.apache.druid.server.QueryResource - Unable to send query response. (org.eclipse.jetty.io.EofException)```
EofException means end of file, right? It is likely that the broker side closed the socket somehow?

Yeah the historical side error looks like it thinks the Broker closed the connection 

no real reason it'd do that after only 22 seconds -- default timeouts are all way longer than that

i wonder if you have a proxy or load balancer between you and Druid that is closing these connections early?

If you do have one -- try contacting the Broker directly 

&gt; If you do have one -- try contacting the Broker directly
<@U0344FW86DD>, Can you illustrated a little bit what you want to try by contacting broker directly? Here, I deployed all the servers in k8. I did try using port forwarding to issue the query against broker and historical respectively. Trying historical individually has not such EOF exception. While trying broker with this query, it still terminate at 22 seconds.

I mean as directly as you can, with as little stuff in the middle as possible (whatever that means for your environment)

I was guessing, based on your URL, that <https://monc-ra-common-lab1.sfproxy.monitoring.dev1-uswest2.aws.sfdc.cl/> is some kind of proxy or load balancer and not the Broker itself

you could try getting a shell on the Broker pod and contacting `localhost`

that's pretty direct :slightly_smiling_face:

That is a good point. The URL is from mesh network settings. Let me try and report the result here.

Here is the test result from broker shell directly:
```bash-4.2$ curl -X POST '<http://127.0.0.1:8088/druid/v2/?pretty>' -H 'Content-Type:application/json'  -d @tst.json &gt; his.out
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  491M    0  491M  100   157  30.7M      9  0:00:17  0:00:16  0:00:01 34.4M```
Basically, at 17 secs, the network connection would terminate. And the result is partial result.
```bash-4.2$ head his.out 
[ {
  "segmentId" : "perf_data_1T_1_2021-06-28T00:00:00.000Z_2021-06-29T00:00:00.000Z_2022-10-14T21:45:47.249Z",
  "columns" : [ "__time", "dc", "device", "env", "eventType", "name", "orgId", "pod", "superPod", "userId", "sampled", "serviceName", "traceId", "spanId", "error", "errorMessage", "errorType", "httpStatusCode", "uri", "transactionType", "closed", "closingTimestamp", "dbCallCount", "duration", "dbCallDuration", "externalCallCount", "externalCallDuration" ],
  "events" : [ {
    "__time" : 1624838400001,
    "dc" : "ifc",
    "device" : "<http://istla4la2-lapp1-4-prd.eng.sfdc.net|istla4la2-lapp1-4-prd.eng.sfdc.net>",
    "env" : "test",
    "eventType" : "transaction",
    "name" : "rqrxxpqj",
bash-4.2$ vim his.out 
bash: vim: command not found
bash-4.2$ vi his.out 

[1]+  Stopped                 vi his.out
bash-4.2$ fg
vi his.out
bash-4.2$ tail his.out 
    "transactionType" : "hybernate",
    "closed" : "false",
    "closingTimestamp" : "1722314312545",
    "dbCallCount" : 23803,
    "duration" : 532518,
    "dbCallDuration" : 8444575,
    "externalCallCount" : 59096,
    "externalCallDuration" : 4503417
  } ]
}```

That looks like a complete set for at least one segment. If you grep for “segmentId” on the file, do you see more segments?

This is at the end of a complete segment, but still not well-formed json. It should end with `]`. Also, let me post the error logs, this time, a little bit more from historical side.

Error logs from Druid historical
```2022-10-19T18:51:41,474 WARN [qtp1727196188-205] org.apache.druid.client.JsonParserIterator - Query [b6759bd2-1936-4efa-acef-ee26c651eeaf] to host [10.64.210.125:8088] interrupted
java.io.IOException: Query[b6759bd2-1936-4efa-acef-ee26c651eeaf] url[<http://10.64.210.125:8088/druid/v2/>] failed with exception msg [Channel disconnected]
        at org.apache.druid.client.DirectDruidClient$1$3.read(DirectDruidClient.java:408) ~[druid-server-0.23.0-sfmc.jar:0.23.0-sfmc]
        at java.io.InputStream.read(InputStream.java:170) ~[?:1.8.0_162]
        at java.io.SequenceInputStream.read(SequenceInputStream.java:207) ~[?:1.8.0_162]
        at com.fasterxml.jackson.dataformat.smile.SmileParser._loadToHaveAtLeast(SmileParser.java:289) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
        at com.fasterxml.jackson.dataformat.smile.SmileParser._decodeShortAsciiValue(SmileParser.java:2223) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
        at com.fasterxml.jackson.dataformat.smile.SmileParser.getText(SmileParser.java:996) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
        at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:672) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
        at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapObject(UntypedObjectDeserializer.java:895) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
        at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:654) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
        at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapArray(UntypedObjectDeserializer.java:831) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
        at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:668) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:530) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:528) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:417) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1292) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
        at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4189) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2525) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
        at org.apache.druid.client.JsonParserIterator.next(JsonParserIterator.java:112) ~[druid-server-0.23.0-sfmc.jar:0.23.0-sfmc]
        at org.apache.druid.java.util.common.guava.BaseSequence.makeYielder(BaseSequence.java:90) ~[druid-core-0.23.0-sfmc.jar:0.23.0-sfmc]
        at org.apache.druid.java.util.common.guava.BaseSequence.access$000(BaseSequence.java:27) ~[druid-core-0.23.0-sfmc.jar:0.23.0-sfmc]
        at org.apache.druid.java.util.common.guava.BaseSequence$1.next(BaseSequence.java:114) ~[druid-core-0.23.0-sfmc.jar:0.23.0-sfmc]
        at org.apache.druid.java.util.common.guava.MergeSequence.makeYielder(MergeSequence.java:132) ~[druid-core-0.23.0-sfmc.jar:0.23.0-sfmc]
...
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [jetty-util-9.4.47.v20220610.jar:9.4.47.v20220610]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.47.v20220610.jar:9.4.47.v20220610]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.47.v20220610.jar:9.4.47.v20220610]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: org.jboss.netty.channel.ChannelException: Channel disconnected
        at org.apache.druid.java.util.http.client.NettyHttpClient$1.channelDisconnected(NettyHttpClient.java:345) ~[druid-core-0.23.0-sfmc.jar:0.23.0-sfmc]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.cleanup(ReplayingDecoder.java:570) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.10.6.Final.jar:?]
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[netty-3.10.6.Final.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_162]
        ... 1 more
2022-10-19T18:51:41,476 ERROR [qtp1727196188-205] org.apache.druid.server.QueryResource - Unable to send query response. (com.fasterxml.jackson.databind.JsonMappingException: Query[b6759bd2-1936-4efa-acef-ee26c651eeaf] url[<http://10.64.210.125:8088/druid/v2/>] failed with exception msg [Channel disconnected])
2022-10-19T18:51:41,477 WARN [qtp1727196188-205] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [b6759bd2-1936-4efa-acef-ee26c651eeaf] (com.fasterxml.jackson.databind.JsonMappingException: Query[b6759bd2-1936-4efa-acef-ee26c651eeaf] url[<http://10.64.210.125:8088/druid/v2/>] failed with exception msg [Channel disconnected])
2022-10-19T18:51:41,477 WARN [qtp1727196188-205] org.eclipse.jetty.server.HttpChannel - handleException /druid/v2/ com.fasterxml.jackson.databind.JsonMappingException: Query[b6759bd2-1936-4efa-acef-ee26c651eeaf] url[<http://10.64.210.125:8088/druid/v2/>] failed with exception msg [Channel disconnected]```

And historical side, the error message is still
```2022-10-19T18:51:40,471 ERROR [qtp2085313771-179[scan_[perf_data_1T_1]_b6759bd2-1936-4efa-acef-ee26c651eeaf]] org.apache.druid.server.QueryResource - Unable to send query response. (org.eclipse.jetty.io.EofException)
2022-10-19T18:51:40,472 WARN [qtp2085313771-179] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [b6759bd2-1936-4efa-acef-ee26c651eeaf] (org.eclipse.jetty.io.EofException)```

<@U0344FW86DD>, <@U030K4UM3H7>, if I try from the shell of the historical server to run this scan, after 2 minutes, 36sec or so, I did have *timeout* exception. Do we have any idea of what is this *timeout*?
Here is the log:
```2022-10-19T19:19:36,275 ERROR [qtp2085313771-120[scan_[perf_data_1T_1]_40b50275-9167-475a-82a9-e185f9f261d2]] org.apache.druid.server.QueryResource - Unable to send query response. (com.fasterxml.jackson.databind.JsonM
appingException: Query [40b50275-9167-475a-82a9-e185f9f261d2] timed out)
2022-10-19T19:19:36,275 WARN [qtp2085313771-120] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [40b50275-9167-475a-82a9-e185f9f261d2] (com.fasterxml.jackson.databind.JsonMappingException: 
Query [40b50275-9167-475a-82a9-e185f9f261d2] timed out)
2022-10-19T19:19:36,275 WARN [qtp2085313771-120] org.eclipse.jetty.server.HttpChannel - handleException /druid/v2/ com.fasterxml.jackson.databind.JsonMappingException: Query [40b50275-9167-475a-82a9-e185f9f261d2] timed
 out```
And the query is:
```bash-4.2$ date; curl -X POST '<http://127.0.0.1:8088/druid/v2/?pretty>' -H 'Content-Type:application/json' -d @scan.json &gt; his.out
Wed Oct 19 19:16:59 GMT 2022
 % Total  % Received % Xferd Average Speed  Time  Time   Time Current
                 Dload Upload  Total  Spent  Left Speed
100 11.3G  0 11.3G 100  157 74.2M   1 0:02:37 0:02:36 0:00:01 78.0M```

I couldn't find any timeout setting on the historical that would do that in 2:37 timeframe. The defaults are set to 5 minutes. This still seems like a networking issue. What version of k8s are you on?
I found a related article which is kind of old that refers to unexpected connection loss with large workloads (like large file downloads) here: <https://kubernetes.io/blog/2019/03/29/kube-proxy-subtleties-debugging-an-intermittent-connection-reset/>
It says this is resolved in kubernetes version 1.15+.

I will check with the K8 teams, but I think or k8 is above 1.15 which is pretty old.

That said, this error with `timed out` is different. We can search the code to have some clue.
```2022-10-19T19:19:36,275 ERROR [qtp2085313771-120[scan_[perf_data_1T_1]_40b50275-9167-475a-82a9-e185f9f261d2]] org.apache.druid.server.QueryResource - Unable to send query response. (com.fasterxml.jackson.databind.JsonM
appingException: Query [40b50275-9167-475a-82a9-e185f9f261d2] timed out)```

In fact, this line is from QueryResource.java:
```log.noStackTrace().error(ex, "Unable to send query response.");```
```Response.ResponseBuilder responseBuilder = Response
    .ok(
        new StreamingOutput()
        {
          @Override
          public void write(OutputStream outputStream) throws WebApplicationException
          {
            Exception e = null;

            CountingOutputStream os = new CountingOutputStream(outputStream);
            try {
              // json serializer will always close the yielder
              jsonWriter.writeValue(os, yielder);

              os.flush(); // Some types of OutputStream suppress flush errors in the .close() method.
              os.close();
            }
            catch (Exception ex) {
              e = ex;
              log.noStackTrace().error(ex, "Unable to send query response.");
              throw new RuntimeException(ex);
            }
            finally {
              Thread.currentThread().setName(currThreadName);

              queryLifecycle.emitLogsAndMetrics(e, req.getRemoteAddr(), os.getCount());```


I think that means that the output stream failed to write, flush or close. The stack trace above also indicates a network communication problem, this is where the failure occurred:
<https://github.com/netty/netty/blob/netty-3.10.6.Final/src/main/java/org/jboss/netty/channel/socket/nio/NioWorker.java#L93>
which seems to indicate that the read failed here <https://github.com/netty/netty/blob/5f56a03bcb0aa23bdc9f68686e5c2abcf963a9cd/src/main/java/org/jboss/netty/channel/socket/nio/NioWorker.java#L64>
(could've been a ClosedChannel exception which is caught in line 72 and intentionally ignored which would lead to either ret=0 or failure=true and therefore end up firing the ChannelDisconnectedException here:
``` at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) ```
<https://github.com/netty/netty/blob/5f56a03bcb0aa23bdc9f68686e5c2abcf963a9cd/src/main/java/org/jboss/netty/channel/socket/nio/AbstractNioWorker.java#L360>
so, I'm still thinking network problem.

<@U030K4UM3H7>, with the code pointers, are you talking about the last historical *timeout* issue?

I was following the bottom of the stack trace that you posted last.
I've been trying to find similar conditions. Some users report this with large queries as well, here: <https://github.com/apache/druid/issues/5340>
and they are describing solutions/workarounds by adjusting thread count and/or heap and maxmemory on the broker and historical (<https://github.com/apache/druid/issues/5340#issuecomment-657761335|see comment here>)
perhaps it is worth a try.

My understanding if the code is kind of limited as I haven't really dig deep in this codebase. That said, here is my digging. Let me know if I am wrong somewhere.
1/ Setup of Json serialization mapper is in DefaultObjectMapper.java. With this setup, this line in response `jsonWriter.writeValue(os, yielder);` knows how to serialize a `Sequence` Yielder.
```public DefaultObjectMapper(JsonFactory factory)
{
  super(factory);
  registerModule(new DruidDefaultSerializersModule());
  registerModule(new GuavaModule());
  registerModule(new GranularityModule());
  registerModule(new AggregatorsModule());```
Note, here it is registering Jackson module, not Guice module.

2/ DruidDefaultSerializersModule we have how to serialize a Yielder (sequence's yielder), which basically yield one object at a time, and call `jgen.writeObject(o)`
```addSerializer(
    Yielder.class,
    new JsonSerializer&lt;Yielder&gt;()
    {
      @Override
      public void serialize(Yielder yielder, final JsonGenerator jgen, SerializerProvider provider)
          throws IOException
      {
        try {
          jgen.writeStartArray();
          while (!yielder.isDone()) {
            final Object o = yielder.get();
            jgen.writeObject(o);
            yielder = yielder.next(null);
          }
          jgen.writeEndArray();
        }
        finally {
          yielder.close();
        }
      }
    }
);```
3/ In the QueryResource code to build response
```try {
  // json serializer will always close the yielder
  jsonWriter.writeValue(os, yielder);

  os.flush(); // Some types of OutputStream suppress flush errors in the .close() method.
  os.close();
}
catch (Exception ex) {
  e = ex;
  log.noStackTrace().error(ex, "Unable to send query response.");
  throw new RuntimeException(ex);
}```
Here, `jsonWriter` actually wraps a DefaultObjectMapper above to write yielders. Note, this is a streaming operation. One the main reason to use Yielder/Sequence (not common pattern) is to stream the data while doing some business logic processing. And here, the `ex`  has the following message `
```(com.fasterxml.jackson.databind.JsonM
appingException: Query [40b50275-9167-475a-82a9-e185f9f261d2] timed out)```
This `ex` is thrown out from the above serialization code
```          jgen.writeStartArray();
          while (!yielder.isDone()) {
            final Object o = yielder.get();
            jgen.writeObject(o);
            yielder = yielder.next(null);
          }
          jgen.writeEndArray();```
Note, *this san test is from historical. So this yielder is not really get data from another server via network socket.* Instead, it is getting the data from the local disk. So I think the timeout it complains is with some disk/cache/mmaped segment file etc.

<@U030K4UM3H7>, I see. <https://github.com/apache/druid/issues/5340> is also an interesting read. In fact, now I have seen two problems here.
1/ From broker, run scan, I see exception similar to <https://github.com/apache/druid/issues/5340>
2/ Taking advice from <@U0344FW86DD>, run scan on historical, I see the timeout in around 100sec (2 mins, 37sec).
The above code going through is for the 2/ timeout issue.

You probably understand it better than I do. Given the comments on that issue. I'm wondering if you are having a memory issue on the historical that is causing this when dealing with large results. What are your JVM settings for the historical?

Following <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1666216174905729?thread_ts=1666135297.463899&amp;cid=C0309C9L90D|this post>, I did a little bit more digging. The basic idea is that this line `log.noStackTrace().error(ex, "Unable to send query response.");` which give `timeout` is from the `ex` exception, whose stack trace was not logged. However, we can still find out where in the code it may be logged out. The current intellij has an older copy of code that I did not find the exact pattern of `Query [40b50275-9167-475a-82a9-e185f9f261d2] timed out`. However, from latest code repo <https://github.com/apache/druid/search?p=5&amp;q=%22timed+out%22|here>, I did find out several having this pattern. And this one, <https://github.com/apache/druid/blob/c83115e4e19a5825413c09566d165aa142a712a7/processing/src/main/java/org/apache/druid/query/scan/ScanQueryEngine.java#L180|https://github.com/apache/druid/blob/c83115e4e19a5825413c09566d165aa142a712a7/proc[…]/src/main/java/org/apache/druid/query/scan/ScanQueryEngine.java> seem to be the line emitting the  `Query xxxxx timed out` exception when serialization of json response.

And the timeout seems from query context. as this line showed here <https://github.com/apache/druid/blob/c83115e4e19a5825413c09566d165aa142a712a7/processing/src/main/java/org/apache/druid/query/scan/ScanQueryEngine.java#L80-L81|https://github.com/apache/druid/blob/c83115e4e19a5825413c09566d165aa142a712a7/proc[…]/src/main/java/org/apache/druid/query/scan/ScanQueryEngine.java>
```    final boolean hasTimeout = query.context().hasTimeout();
    final Long timeoutAt = responseContext.getTimeoutTime();```

Wondering if we have a doc about where to set/check query context timeout from the native query, or sql?

this is where the timeout is set <https://github.com/apache/druid/blob/c83115e4e19a5825413c09566d165aa142a712a7/processing/src/main/java/org/apache/druid/query/scan/ScanQueryRunnerFactory.java#L96|https://github.com/apache/druid/blob/c83115e4e19a5825413c09566d165aa142a712a7/proc[…]in/java/org/apache/druid/query/scan/ScanQueryRunnerFactory.java>

I followed that back to the query context attribute called "timeout" <https://github.com/apache/druid/blob/c83115e4e19a5825413c09566d165aa142a712a7/processing/src/main/java/org/apache/druid/query/QueryContexts.java#L45|here>.

Actually,  see this doc <https://druid.apache.org/docs/latest/querying/query-context.html>, query context may be tuned from query input.

however, it does not have the 100sec that I experienced.

just found that too. should be defaulting to 5 minutes. but maybe we can disprove the timeout as the root cause by setting the query context with timeout really high.

Let me try. I will post the result here later.

Some updates:
1/ Validated that there are some platform issues. Disable some feature in k8/mesh, I do see 10minutes (configured and expected) time out streaming data from router using native scan query. Thus, the EOF exception historical see is due to this platform config.
2/ However, issuing the same scan query to the historical local shell, we still have the 2 mins 37 sec (100sec or so) query context timeout. I don't know where this time out is from. This is a minor issue though.

<@U030KVC05FH> could you share what the k8/mesh feature/setting was that was causing this? It'll be useful to know if anyone else encounters this symptom.

Istio mesh sidecar has a default timeout. Tuning this one would help.

I think that happens when you use multiple subqueries.

<@U030K4UM3H7> <@U0344FW86DD> Can you help answer the query here - <https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1666142039444669?thread_ts=1665514948.145529&amp;cid=C0303FDCZEZ>

a union SQL query can have multiple native queries with each one having its own native query id.

subqueries don't have a different id but a different subquery id.

<@U031A9D19NU> I guess the relevant question is whether subquery id's get their own metrics emitted separate from the main query id?

they do get their own metrics emitted. though the "id" dimension should be same for these metrics.

What are the results that you are getting b/w 1 and 2. Sketches are supposed to be approximations so minor variations are expected.

As per my understanding the subquery should have finalized the aggregations. Therefore for the v1, the aggregation should be applied on the  finalized version of sub_query (which would already have some approximated and aggregated data) whereas for v2 it should be applied on the whole table (which would be all the raw data). Therefore some variance is expected. This is my understanding of the sketches.

My bad: I did not do any post agg (p95 on the ) in  sub-query , so i was expecting the aggregation that i will get is raw sketch from sub-query.

`sub_query = quantile_sketch(event.duration) by service, api`

results from the queries

```quantile_p95_v1 = 3767
quantile_p95_v2 = 5000```


are you using native queries or sql? in sql i believe if you use DS_QUANTILES_SKETCH in the sub query, then the result will remain a sketch and can be further aggregated in an out query 

in native, subquery aggregations are by default not finalized. (finalization refers to converting sketch to estimate.) so if you refer to the aggregation from the subquery directly then it should work as you expect

something like this

```{
  "queryType": "groupBy",
  "dataSource": {
    "type": "query",
    "query": {
      "dataSource": "events",
      "queryType": "groupBy",
      "filter": {
        "type": "and",
        "fields": [
          {
            "type": "bound",
            "dimension": "create_time",
            "lower": 1666071966,
            "lowerStrict": "false",
            "upper": 1666158366,
            "upperStrict": "false",
            "ordering": "numeric"
          }
        ]
      },
      "intervals": [
        "2022-10-18T05:30:00Z/2022-10-19T06:00:00Z"
      ],
      "granularity": "all",
      "virtualColumns": [],
      "aggregations": [
        {
          "type": "quantilesDoublesSketch",
          "name": "duration_quantile",
          "fieldName": "duration",
          "k": 32768
        }
      ],
      "postAggregations": [
        {
          "type": "expression",
          "name": "duration_quantile_post_agg",
          "expression": "duration_quantile"
        }
      ],
      "dimensions": [
        {
          "type": "extraction",
          "dimension": "create_time",
          "outputName": "__ts",
          "outputType": "LONG",
          "extractionFn": {
            "type": "bucket",
            "size": 60000000,
            "offset": 1666071966
          }
        }
      ]
    }
  },
  "intervals": [
    "2022-10-18T05:30:00Z/2022-10-19T06:00:00Z"
  ],
  "granularity": "all",
  "context": {
    "queryId": "f93ff520-d4b2-4493-8c4c-6ede6d9393f8_1666071966741_1d",
    "timeout": 60000,
    "finalize": true
  },
  "aggregations": [
    {
      "name": "duration_quantile_post_agg_agg",
      "fieldName": "duration_quantile_post_agg",
      "type": "quantilesDoublesSketch"
    }
  ],
  "dimensions": [],
  "postAggregations": [
    {
      "type": "quantilesDoublesSketchToQuantile",
      "name": "quantile_over_time_0.95",
      "field": {
        "type": "fieldAccess",
        "fieldName": "duration_quantile_post_agg_agg",
        "name": "duration_quantile_post_agg_agg_p95"
      },
      "fraction": 0.95
    }
  ],
  "subtotalsSpec": [
    []
  ]
}```


try reading duration_quantile from the subquery directly

when you read it through the expression postaggregator (duration_quantile_post_agg) it gets finalized 

```druid-broker-27 druid-broker 2022-10-19T05:17:55,959 ERROR [DruidSchema-Cache-0] org.apache.druid.query.ResultLevelCachingQueryRunner - Error while preparing for result level caching for query e5b21375-030e-48ec-9d2a-0ca41f75215d with error Faulty channel in resource pool 
druid-broker-27 druid-broker org.apache.druid.query.QueryInterruptedException: Faulty channel in resource pool
druid-broker-27 druid-broker      at org.apache.druid.client.JsonParserIterator.convertException(JsonParserIterator.java:273) ~[druid-server-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:191) ~[druid-server-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.client.JsonParserIterator.hasNext(JsonParserIterator.java:93) ~[druid-server-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.BaseSequence.makeYielder(BaseSequence.java:89) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:69) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$ResultBatch.fromSequence(ParallelMergeCombiningSequence.java:879) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$SequenceBatcher.block(ParallelMergeCombiningSequence.java:929) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at java.util.concurrent.ForkJoinPool.compensatedBlock(ForkJoinPool.java:3450) ~[?:?]
druid-broker-27 druid-broker      at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3433) ~[?:?]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$SequenceBatcher.getBatchYielder(ParallelMergeCombiningSequence.java:918) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$YielderBatchedResultsCursor.initialize(ParallelMergeCombiningSequence.java:1025) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$PrepareMergeCombineInputsAction.compute(ParallelMergeCombiningSequence.java:732) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194) ~[?:?]
druid-broker-27 druid-broker      at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) ~[?:?]
druid-broker-27 druid-broker      at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) ~[?:?]
druid-broker-27 druid-broker      at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) ~[?:?]
druid-broker-27 druid-broker      at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) ~[?:?]
druid-broker-27 druid-broker      at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) ~[?:?]
druid-broker-27 druid-broker Caused by: org.jboss.netty.channel.ChannelException: Faulty channel in resource pool
druid-broker-27 druid-broker      at org.apache.druid.java.util.http.client.NettyHttpClient.go(NettyHttpClient.java:132) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.http.client.CredentialedHttpClient.go(CredentialedHttpClient.java:48) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.client.DirectDruidClient.run(DirectDruidClient.java:453) ~[druid-server-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.getSimpleServerResults(CachingClusteredClient.java:706) ~[druid-server-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.lambda$addSequencesFromServer$8(CachingClusteredClient.java:668) ~[druid-server-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at java.util.TreeMap.forEach(TreeMap.java:1282) ~[?:?]```

```druid-broker-27 druid-broker Caused by: java.nio.channels.UnresolvedAddressException
druid-broker-27 druid-broker      at sun.nio.ch.Net.checkAddress(Net.java:149) ~[?:?]
druid-broker-27 druid-broker      at sun.nio.ch.Net.checkAddress(Net.java:157) ~[?:?]
druid-broker-27 druid-broker      at sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:816) ~[?:?]
druid-broker-27 druid-broker      at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:839) ~[?:?]```

Yeah, I think Kube’s CoreDNS was caching old ip addresses.

After waiting a bit and performing rolling restart on broker and router, the problem went away.

gave me a tiny heart attack, haha.
“We performed so many upgrades and restarts already, how come there’s still an unaccounted fail-case?”

We hit exact same issue on multiple clusters after upgrading

so i have listed down the current issues in the following manner
query1 -  without LIMIT - this FAILS with timeout
query 2 - query1 with LIMIT - this PASSES

query 3 - which we found yesterday, this FAILS with timeout both with and without LIMIT. i am assuming this is an empty result set case

for all these cases i have found the equivalent queries in wikipedia datasource and tested and got similiar results and have raised in github. please refer the github issue for more details

Did some more research
seeing similar issue in 0.22.1, 0.23.0 and 24.0.0
not seeing the issue in 0.21.1
Any help would be appreciated

is this one the correct? `ZOO_LOG4J_PROP: INFO,ROLLINGFILE` ?

One unrelated question regarding the zookeeper logs - from the ls output it appears that the zk log files are not getting purged even though you've set
```ZK_AUTOPURGE_SNAPRETAINCOUNT: '3',
ZK_AUTOPURGE_PURGEINTERVAL: '1'```
which means that only last 3 snapshots and logs should be present (ref : <https://zookeeper.apache.org/doc/r3.5.9/zookeeperAdmin.html#sc_strengthsAndLimitations>). I wonder if these are the correct configs to use for enabling auto purge?

I am also setting `ZK_ROLLING_LOG_FILE_MAX_COUNT: '10'`

yeah I also thought above settings would purge logs but not sure if thats the correct setting or not

I'm not a k8s expert but if you're using zk docker image in k8s, then maybe you can check <https://hub.docker.com/_/zookeeper>

they seem to be mentioning `ZOO_AUTOPURGE_PURGEINTERVAL` and `ZOO_AUTOPURGE_SNAPRETAINCOUNT`

I am setting those env vars by using helm values yaml available in Druid GitHub repo

Oh, I see. do you have github link to those vars? Sorry, I wasn't able to find them

Well you won't find the exact env vars in the file, but you know they exist

<https://github.com/apache/druid/blob/master/helm/druid/values.yaml#L381|https://github.com/apache/druid/blob/master/helm/druid/values.yaml#L381>

Zookeeper has a setting that auto prune consensus logs every hour, just turn that on.

what setting is that? are not they the one mentioned above?

In my Bitnami Zookeeper Helm chart, there’s this setting:
```autopurge:
  ## @param autopurge.snapRetainCount Retains the snapRetainCount most recent snapshots and the corresponding transaction logs and deletes the rest
  ##
  snapRetainCount: 3
  ## @param autopurge.purgeInterval The time interval in hours for which the purge task has to be triggered
  ## Set to a positive integer (1 and above) to enable the auto purging
  ##
  purgeInterval: 1```
Your helm chart should have something similar.

Yeah exactly those env vars are the one mentioned above

And I have those in the statefulset for ZK

you could probably also checkout <https://github.com/kubernetes-retired/contrib/blob/89f6948e24578fed2a90a87871b2263729f90ac3/statefulsets/zookeeper/README.md>
it mentions the properties as `ZK_SNAP_RETAIN_COUNT` and `ZK_PURGE_INTERVAL`
I found this while searching for zk image using the properties you've mentioned

I am not sure what the actual env var is, if `ZK_AUTOPURGE_SNAPRETAINCOUNT` or `ZK_SNAP_RETAIN_COUNT`

maybe you could add the other two env vars in the config too if possible. if they are right ones the logs should clear up. I think having extra env vars shouldn't cause any problems

this is what my ZK config look like now:
```zookeeper: {
                    enabled: true,
                    env: {
                        ZK_HEAP_SIZE: '512m',
                        ZK_INIT_LIMIT: '5',
                        ZK_SYNC_LIMIT: '2',
                        ZK_TICK_TIME: '2000',
                        ZK_MAX_CLIENT_CNXNS: '0',
                        ZK_AUTOPURGE_SNAPRETAINCOUNT: '3',
                        ZK_SNAP_RETAIN_COUNT: '3',
                        ZK_AUTOPURGE_PURGEINTERVAL: '1',
                        ZK_PURGE_INTERVAL: '1',
                        ZOO_LOG4J_PROP: 'INFO,ROLLINGFILE',
                        ZK_ROLLING_LOG_FILE_MAX_COUNT: '10'
                    },
                    persistence: {
                        size: '15Gi'
                    }
                }```

btw, if I run an export in the pod I see the following
```zookeeper@druid-test-zookeeper-0:/apache-zookeeper-3.5.5-bin/bin$ export
declare -x AWS_DEFAULT_REGION="us-east-1"
declare -x AWS_REGION="us-east-1"
declare -x AWS_ROLE_ARN="arn:aws:iam::204717343847:role/druid-test-5533abc"
declare -x AWS_WEB_IDENTITY_TOKEN_FILE="/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
declare -x DRUID_CLUSTER_US2_BROKER_PORT="<tcp://172.20.107.228:8082>"
declare -x DRUID_CLUSTER_US2_BROKER_PORT_8082_TCP="<tcp://172.20.107.228:8082>"
declare -x DRUID_CLUSTER_US2_BROKER_PORT_8082_TCP_ADDR="172.20.107.228"
declare -x DRUID_CLUSTER_US2_BROKER_PORT_8082_TCP_PORT="8082"
declare -x DRUID_CLUSTER_US2_BROKER_PORT_8082_TCP_PROTO="tcp"
declare -x DRUID_CLUSTER_US2_BROKER_SERVICE_HOST="172.20.107.228"
declare -x DRUID_CLUSTER_US2_BROKER_SERVICE_PORT="8082"
declare -x DRUID_CLUSTER_US2_BROKER_SERVICE_PORT_HTTP="8082"
declare -x DRUID_CLUSTER_US2_COORDINATOR_PORT="<tcp://172.20.186.60:8081>"
declare -x DRUID_CLUSTER_US2_COORDINATOR_PORT_8081_TCP="<tcp://172.20.186.60:8081>"
declare -x DRUID_CLUSTER_US2_COORDINATOR_PORT_8081_TCP_ADDR="172.20.186.60"
declare -x DRUID_CLUSTER_US2_COORDINATOR_PORT_8081_TCP_PORT="8081"
declare -x DRUID_CLUSTER_US2_COORDINATOR_PORT_8081_TCP_PROTO="tcp"
declare -x DRUID_CLUSTER_US2_COORDINATOR_SERVICE_HOST="172.20.186.60"
declare -x DRUID_CLUSTER_US2_COORDINATOR_SERVICE_PORT="8081"
declare -x DRUID_CLUSTER_US2_COORDINATOR_SERVICE_PORT_HTTP="8081"
declare -x DRUID_CLUSTER_US2_HISTORICAL_PORT="<tcp://172.20.242.232:8083>"
declare -x DRUID_CLUSTER_US2_HISTORICAL_PORT_8083_TCP="<tcp://172.20.242.232:8083>"
declare -x DRUID_CLUSTER_US2_HISTORICAL_PORT_8083_TCP_ADDR="172.20.242.232"
declare -x DRUID_CLUSTER_US2_HISTORICAL_PORT_8083_TCP_PORT="8083"
declare -x DRUID_CLUSTER_US2_HISTORICAL_PORT_8083_TCP_PROTO="tcp"
declare -x DRUID_CLUSTER_US2_HISTORICAL_SERVICE_HOST="172.20.242.232"
declare -x DRUID_CLUSTER_US2_HISTORICAL_SERVICE_PORT="8083"
declare -x DRUID_CLUSTER_US2_HISTORICAL_SERVICE_PORT_HTTP="8083"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_PORT="<tcp://172.20.38.117:8091>"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_PORT_8091_TCP="<tcp://172.20.38.117:8091>"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_PORT_8091_TCP_ADDR="172.20.38.117"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_PORT_8091_TCP_PORT="8091"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_PORT_8091_TCP_PROTO="tcp"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_SERVICE_HOST="172.20.38.117"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_SERVICE_PORT="8091"
declare -x DRUID_CLUSTER_US2_MIDDLE_MANAGER_SERVICE_PORT_HTTP="8091"
declare -x DRUID_CLUSTER_US2_ROUTER_PORT="<tcp://172.20.233.94:8888>"
declare -x DRUID_CLUSTER_US2_ROUTER_PORT_8888_TCP="<tcp://172.20.233.94:8888>"
declare -x DRUID_CLUSTER_US2_ROUTER_PORT_8888_TCP_ADDR="172.20.233.94"
declare -x DRUID_CLUSTER_US2_ROUTER_PORT_8888_TCP_PORT="8888"
declare -x DRUID_CLUSTER_US2_ROUTER_PORT_8888_TCP_PROTO="tcp"
declare -x DRUID_CLUSTER_US2_ROUTER_SERVICE_HOST="172.20.233.94"
declare -x DRUID_CLUSTER_US2_ROUTER_SERVICE_PORT="8888"
declare -x DRUID_CLUSTER_US2_ROUTER_SERVICE_PORT_HTTP="8888"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_PORT="<tcp://172.20.192.66:2181>"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_PORT_2181_TCP="<tcp://172.20.192.66:2181>"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_PORT_2181_TCP_ADDR="172.20.192.66"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_PORT_2181_TCP_PORT="2181"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_PORT_2181_TCP_PROTO="tcp"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_SERVICE_HOST="172.20.192.66"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_SERVICE_PORT="2181"
declare -x DRUID_CLUSTER_US2_ZOOKEEPER_SERVICE_PORT_CLIENT="2181"
declare -x HOME="/home/zookeeper"
declare -x HOSTNAME="druid-test-zookeeper-0"
declare -x JAVA_BASE_URL="<https://github.com/AdoptOpenJDK/openjdk8-upstream-binaries/releases/download/jdk8u232-b09/OpenJDK8U-jre_>"
declare -x JAVA_HOME="/usr/local/openjdk-8"
declare -x JAVA_URL_VERSION="8u232b09"
declare -x JAVA_VERSION="8u232"
declare -x JMXAUTH="false"
declare -x JMXDISABLE="false"
declare -x JMXPORT="1099"
declare -x JMXSSL="false"
declare -x KUBERNETES_PORT="<tcp://172.20.0.1:443>"
declare -x KUBERNETES_PORT_443_TCP="<tcp://172.20.0.1:443>"
declare -x KUBERNETES_PORT_443_TCP_ADDR="172.20.0.1"
declare -x KUBERNETES_PORT_443_TCP_PORT="443"
declare -x KUBERNETES_PORT_443_TCP_PROTO="tcp"
declare -x KUBERNETES_SERVICE_HOST="172.20.0.1"
declare -x KUBERNETES_SERVICE_PORT="443"
declare -x KUBERNETES_SERVICE_PORT_HTTPS="443"
declare -x LANG="C.UTF-8"
declare -x OLDPWD="/apache-zookeeper-3.5.5-bin"
declare -x PATH="/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/apache-zookeeper-3.5.5-bin/bin"
declare -x PWD="/apache-zookeeper-3.5.5-bin/bin"
declare -x SHLVL="1"
declare -x TERM="xterm"
declare -x ZK_AUTOPURGE_PURGEINTERVAL="1"
declare -x ZK_AUTOPURGE_SNAPRETAINCOUNT="3"
declare -x ZK_HEAP_SIZE="512m"
declare -x ZK_INIT_LIMIT="5"
declare -x ZK_MAX_CLIENT_CNXNS="0"
declare -x ZK_REPLICAS="3"
declare -x ZK_ROLLING_LOG_FILE_MAX_COUNT="10"
declare -x ZK_SYNC_LIMIT="2"
declare -x ZK_TICK_TIME="2000"
declare -x ZOOCFGDIR="/conf"
declare -x ZOO_ADMINSERVER_ENABLED="true"
declare -x ZOO_AUTOPURGE_PURGEINTERVAL="0"
declare -x ZOO_AUTOPURGE_SNAPRETAINCOUNT="3"
declare -x ZOO_CONF_DIR="/conf"
declare -x ZOO_DATA_DIR="/data"
declare -x ZOO_DATA_LOG_DIR="/datalog"
declare -x ZOO_INIT_LIMIT="5"
declare -x ZOO_LOG4J_PROP="INFO,ROLLINGFILE"
declare -x ZOO_LOG_DIR="/logs"
declare -x ZOO_MAX_CLIENT_CNXNS="60"
declare -x ZOO_PORT="2181"
declare -x ZOO_STANDALONE_ENABLED="false"
declare -x ZOO_SYNC_LIMIT="2"
declare -x ZOO_TICK_TIME="2000"```
looks like the vars are ?
```declare -x ZOO_AUTOPURGE_PURGEINTERVAL="0"
declare -x ZOO_AUTOPURGE_SNAPRETAINCOUNT="3"```

and in the zoo.cfg file
```zookeeper@druid-test-zookeeper-0:/conf$ cat zoo.cfg
clientPort=2181
dataDir=/data
dataLogDir=/data/log
tickTime=2000
initLimit=5
syncLimit=2
maxClientCnxns=0
minSessionTimeout= 4000
maxSessionTimeout= 40000
autopurge.snapRetainCount=3
autopurge.purgeInterval=0
4lw.commands.whitelist=*
server.1=druid-test-zookeeper-0.druid-test-zookeeper-headless.druid-cluster.svc.cluster.local:2888:3888
server.2=druid-test-zookeeper-1.druid-test-zookeeper-headless.druid-cluster.svc.cluster.local:2888:3888
server.3=druid-test-zookeeper-2.druid-test-zookeeper-headless.druid-cluster.svc.cluster.local:2888:3888```