Apache Druid

I think "node 2" is probably referring to a Kafka broker

I wonder if this is due to a Kafka client update?

Or a Kafka broker issue that happened to start around the same time?

0.23.0 is kafka 3.1.0 and 24.0.0 is kafka 3.2.0

These disconnections are definitely referring to Kafka brokers. But the brokers themselves show no errors and we’ve deployed an entire new instance of Kafka and the same errors are seen (leading us back to assuming Druid being the root cause somehow). 

hmm. i can't think of a reason why druid code would be getting in the way of the kafka client -&gt; broker comms; so, my first thought is let's look into whether there is some problem with the new kafka client

would you be able to downgrade the kafka client to 3.1.0 (the version used in 0.23.0) and see if that changes anything?

either by building a patched distribution or by replacing the 3.2.0 jar with a 3.1.0 jar

This is a Kafka client-side (Kafka consumer in the Druid) timeout for Fetch requests (not networking socket timeout). The client will wait for the response to a request for 30 seconds in the current scenario. If the answer is not received before the timeout elapses, the client will resend the request (retry logic).
Internally the Kafka client iterates over all the inflight requests and expires any requests that have exceeded the configured request timeout (Kafka consumer logic). The connection to the node (broker) associated with the request will be terminated and will be treated as a disconnection.

 The issue can be the networking layer, broker utilization, and potentially GC on the Peon JVM.
Do you have the client connectivity metrics (like new connections rate) and max request processing time from the broker? Thanks

we did some changes semi-recently, around the 24 timeframe, to greatly improve java 17 support

including building, ability for unit tests to pass, etc

ok ok, I will stick with Java 11 for 0.23.0

if `col_a` is a `long` (aka `BIGINT`) column then i believe these are equivalent.

if `col_a` is string then they are not, since (2) will insert a cast to number, so for example, the string `100.0` will still match. this means that for string `col_a`, (2) will match more values, and will also be slower (due to the cast)

you can confirm if they're equivalent or different by doing `EXPLAIN PLAN FOR` in front of your query

to see the native query that will execute

<https://apachedruidworkspace.slack.com/archives/C030K0Z7S1Z/p1668629819059589?thread_ts=1668623186.619739&amp;cid=C030K0Z7S1Z>

thanks! my point was that it seemed that mm tasks upload and announce segments even _before_ task duration is exhausted (for example when segment granularity or rows per segment is met). my question was, in that case, what is really the significance of task duration? or do I misunderstand something?

I think it's mainly related to the segments merging, but I need to check that with empirical testing (because I can't read the java code to follow the 'path'), but I think if you set task duration 1hr, but limit each segment to 1 row (that would be very inefficient) and during the task duration, you would have multiples segments, but after the task finishes¹ it should merge then all back into N optimized segments, even if you don't have any auto-compaction enabled

¹ not really finished, but after the new offsets are passed into another MM task be the owner

image.png

hmm
so, according to the docs, this merge does not happen automatically, so task duration is either to you control planned downtime, but that could also be archived by intermediateHandoffPeriod, so now I'm confused :laughing:

after reading <https://druid.apache.org/docs/latest/design/architecture.html> I'm certain that after the handoff, the task offsets are updated as well, so the MM task may only be useful for metrics accumulation?!

I guess I was confused about `Intermediate persist period` vs `Intermediate handoff period` <https://stackoverflow.com/questions/57923060/what-is-intermediate-persist-in-apache-druid>

kafka tasks run for a `taskDuration` amount of time (unless they exit early due to e.g. supervisor suspension or supervisor config change)

during that time, they may do intermediate handoffs, where they publish some data due to reaching a trigger like a certain number of rows

or, they may not, if the data volume isn't super high

either way, when they reach the `taskDuration`, they will do one final publish/handoff of anything that hasn't been published yet, and then exit

by this time, they will have been replaced with a new 'wave' of tasks

the purpose for the waves (rather than using the same tasks forever) is mainly so task logs don't get too long, and so you can do rolling updates of MMs more easily (by disabling them and then waiting for their tasks to exit)

hope this clears things up; feel free to ask something else if not :slightly_smiling_face:

I would suggest using a newer version 24.0 being the latest version to see if the issue still happens

Druid .019 is old, really old :slightly_smiling_face:

Screen Shot 2022-11-27 at 9.37.23 PM.png

Hi, `earlyMessageRejectionPeriod`  should have helped.
Do you not see events being thrown away at all?

Screen Shot 2022-11-27 at 9.58.44 PM.png

<@U0319HD8HEC> I do see some events being thrown away

`select distinct "start" from sys.segments where "is_realtime" = true`  should help identify the intervals after setting the message rejection periods. Could you please try this out?

Screen Shot 2022-11-27 at 10.24.57 PM.png

The message rejection seems to work.
You could try marking segments that have already been ingested to unwanted intervals as unused

Sounds good. Thanks! Im not sure why I thought the new future segments were still getting created

Screen Shot 2022-11-27 at 10.05.17 PM.png

This data source is being pushed into “cold” storage, while tranquility is being pushed into “hot”, could this impact the numbers?

Could you please share how large the records are?
You could increase recordBufferSize and maxRecordsPerPoll as well if the records aren't too large

How would I get the record size?
I am seeing for segments:
269.44 MB and
2,894,139 rows but this seems a little low at less then a KB per record

It might be worth trying an increase in recordBufferSize and maxRecordsPerPoll.

Also, if the records are this small, could you also please set maxRowsInMemory back to the default value of 1,000,000?
75k seems low given that the records are small and could be leading to unnecessary persists to disk

These have not worked yet, but still working to adjust. I notice that “maxBytesInMemory” is set to 0, although I did not set this explicitly. Could this be causing issues?

<https://druid.apache.org/docs/latest/development/extensions-core/kinesis-ingestion.html#kinesissupervisortuningconfig>
The default value is 1/6th of the JVM memory. I don't think maxBytesInMemory is the problem

whenever performance is a question, i like to start with profiling. a flame graph of one of your kinesis tasks would illuminate where it is spending its time:

<https://support.imply.io/hc/en-us/articles/360033747953-Profiling-Druid-queries-using-flame-graphs>

in this case you'd be looking at the `internal peon` processes that have `index_kinesis` in the name

if you post one we may be able to offer some more specific advice

(best to do a task that is actively reading data, btw, as opposed to one that is doing the final publish-and-exiting.)

Yes, MMs use local storage for intermediate files used while ingesting and to respond to queries. 

What are the other components, which require disk space. I have added disk space to historicals, middle managers, coordinator and brokers. Does overload and router also require disk space

overlord and router require only minimal disk space; a few small tmp files and whatever you want to allocate to log files

main disk space users are Historicals (need to store all your queryable data, and some temporary data during queries if groupBy disk spilling is enabled); MMs (need to store temporary data used during ingestion); and Brokers (temporary data during queries if groupBy disk spilling is enabled)

Segment replicas on other Historicals will help.

A query retry would be the only way I guess

if you can kill the pod cleanly (i.e. sigterm the proc inside it, and wait for it to exit) then you will see fewer failures

as during a clean exit, the historical waits for currently-running queries to finish before shutting down

It does sound like maxTotalrows limit is being hit. Did you restart the supervisor after the change?

Yeah, the supervisor has been restarted (I even performed a complete restart of all daemons today for something unrelated). 

Same behavior. :man-shrugging:

<@U0344FW86DD> Any insight into this (you've been ridiculously helpful with my understanding of druid ingest to date)?

I just looked into this a bit, and I think the doc for kafka `maxTotalRows` (<https://druid.apache.org/docs/latest/development/extensions-core/kafka-supervisor-reference.html>) is incorrect: default appears to actually be 20,000,000, not unlimited

but, as far as i can tell, overriding it should work

i only read the code, i stopped short of actually trying it, so could be wrong :slightly_smiling_face:

it looks like it's all wired up though for a default of 20M and for being overridable through `tuningConfig`

so before fixing the docs i'd like to figure out what's going on with your situation

these are suppose to be the triggers for publishing:

1) any task has read more than `maxRowsPerSegment` (default: 5M) into a single segment id. [note this is doing a sum across all intermediate persists with the same segment id, not just looking at the in-memory data]

2) any task has read more than `maxTotalRows` (default: 20M, apparently) across all segment ids. [note this is per-task]

3) `intermediateHandoffPeriod` time has passed

4) `taskDuration` time has passed

what makes it look like (1) is not happening?

i would guess that by the time you read 20M rows in a single task, you probably also read at least one segment with 5M rows. only way this _wouldn't_ be true is if you're generating &gt;4 segments, which isn't common (you'd need late data spread over a variety of time chunks in such a way that none exceeded 5M).

I think on reflection I am hitting 5M rows across a single segment. Once I enabled debug logging on the IndexMergerV9 that became apparent.

Thanks for the explanation, it really is useful context and the clarification that I'm not going insane on the 20,000,000 maxTotalRows default existing

```druid-overlord-1 druid-overlord 2022-11-29T03:00:12,941 WARN [qtp475249993-453] org.eclipse.jetty.server.HttpChannel - /druid/indexer/v1/leader
druid-overlord-1 druid-overlord org.apache.druid.java.util.common.IAE: No userMap is available for authenticator with prefix: [basicAuthAuthenticator]
druid-overlord-1 druid-overlord 	at org.apache.druid.security.basic.authentication.validator.MetadataStoreCredentialsValidator.validateCredentials(MetadataStoreCredentialsValidator.java:64) ~[?:?]
druid-overlord-1 druid-overlord 	at org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:200) ~[?:?]
druid-overlord-1 druid-overlord 	at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59) ~[druid-server-0.23.0.jar:0.23.0]
druid-overlord-1 druid-overlord 	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-overlord-1 druid-overlord 	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-overlord-1 druid-overlord 	at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77) ~[druid-server-0.23.0.jar:0.23.0]
druid-overlord-1 druid-overlord 	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-overlord-1 druid-overlord 	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-overlord-1 druid-overlord 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-overlord-1 druid-overlord 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-overlord-1 druid-overlord 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-overlord-1 druid-overlord 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-overlord-1 druid-overlord 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-overlord-1 druid-overlord 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-overlord-1 druid-overlord 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-overlord-1 druid-overlord 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
druid-overlord-1 druid-overlord 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]```
and
```druid-overlord-1 druid-overlord 2022-11-29T03:00:15,123 WARN [CoordinatorPollingBasicAuthorizerCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (7 of 9) in 53,797ms.
druid-overlord-1 druid-overlord com.fasterxml.jackson.core.JsonParseException: Input does not start with Smile format header (first byte = 0x3c) and parser has REQUIRE_HEADER enabled: can not parse
druid-overlord-1 druid-overlord  at [Source: (byte[])"&lt;html&gt;
druid-overlord-1 druid-overlord &lt;head&gt;
druid-overlord-1 druid-overlord &lt;meta http-equiv="Content-Type" content="text/html;charset=utf-8"/&gt;
druid-overlord-1 druid-overlord &lt;title&gt;Error 401 User authentication failed.&lt;/title&gt;
druid-overlord-1 druid-overlord &lt;/head&gt;
druid-overlord-1 druid-overlord &lt;body&gt;&lt;h2&gt;HTTP ERROR 401 User authentication failed.&lt;/h2&gt;
druid-overlord-1 druid-overlord &lt;table&gt;
druid-overlord-1 druid-overlord &lt;tr&gt;&lt;th&gt;URI:&lt;/th&gt;&lt;td&gt;/druid-ext/basic-security/authorization/db/basicAuthAuthorizer/cachedSerializedGroupMappingMap&lt;/td&gt;&lt;/tr&gt;
druid-overlord-1 druid-overlord &lt;tr&gt;&lt;th&gt;STATUS:&lt;/th&gt;&lt;td&gt;401&lt;/td&gt;&lt;/tr&gt;
druid-overlord-1 druid-overlord &lt;tr&gt;&lt;th&gt;MESSAGE:&lt;/th&gt;&lt;td&gt;User authentication failed.&lt;/td&gt;&lt;/tr&gt;
druid-overlord-1 druid-overlord &lt;tr&gt;&lt;th&gt;SERVLET:&lt;/th&gt;&lt;td&gt;default&lt;/td&gt;&lt;/tr&gt;
druid-overlord-1 druid-overlord &lt;/table&gt;
druid-overlord-1 druid-overlord 
druid-overlord-1 druid-overlord &lt;/body&gt;```


is the cached user-role mapping contains server/hostname information?

```Input does not start with Smile format header (first byte = 0x3c) and parser has REQUIRE_HEADER enabled: can not parse
at [Source: (byte[])```
seems to indicate that the cached object is blank. I don’t understand why that will break Overlord.

I don't think it contains hostname-specific info

maybe the table that has the user maps is missing? or maybe some entries from that table are missing?

sadly i don't remember the name of the table

option a: use a approximation query if possible
option b: set a value to
```druid.query.groupBy.maxMergingDictionarySize=100000000
        druid.query.groupBy.maxOnDiskStorage=2147483648```
on historicals, brokers, and MM

eg: instead of count (distinct column)
use APPROX_COUNT_DISTINCT_DS_THETA(column)

but if you are using array_agg  or distinct array_agg maybe you should remodel your data, if the number of distinct is high you gonna need more memory

Hello Santos!! 
I did the same as you suggested.. but still I'm not getting the result.. instead I'm getting 504 error

504 is a timeout, this query might just be taking a really long time

i would first consider rethinking your approach as Renato has mentioned — maybe there is a more effective way to do the thing you are trying to do?

or if you want to hit it with the brute force hammer, if the result set is large or if the query processing is intense, you can raise various timeouts as described on <https://support.imply.io/hc/en-us/articles/360034310953-Tuning-Druid-for-Large-Result-Sets>. you would not hit these unless your query is taking ~minutes

Also it seems that the `PasswordProvider` approach doesn't work either:
```    druid.escalator.internalClientPassword={"type":"environment","variable":"LDAP_INTERNAL_PASSWORD"}
    druid.escalator.internalClientUsername={"type":"environment","variable":"LDAP_INTERNAL_USER"}
    druid.auth.authenticator.ldap.initialAdminPassword={"type":"environment","variable":"LDAP_ADMIN_PASSWORD"}
    druid.auth.authenticator.ldap.initialInternalClientPassword={"type":"environment","variable":"LDAP_INTERNAL_PASSWORD"}
    druid.auth.authenticator.ldap.credentialsValidator.bindPassword={"type":"environment","variable":"LDAP_ADMIN_PASSWORD"}
    druid.auth.authenticator.ldap.credentialsValidator.bindUser={"type":"environment","variable":"LDAP_ADMIN_USER"}```
Here we get:
```used by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token                                                                                                                                                                                                 │
│  at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: org.apache.druid.security.basic.authentication.BasicHTTPEscalator["internalClientUsername"])                                                                                                                                                                                  │
│     at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)                                                                                                                                                                                                                                           │
│     at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1445)                                                                                                                                                                                                                                  │...```

It's supposed to work for the password ones, but not for `bindUser`

`bindUser` does need to be specified inline

Thank you, we removed the user names from the dynamic config and left only password properties, but that unfortunately didn't change anything.

do you get a different error once you do that? the ones you posted initially mentioned username-related parameters

Druid SQL has a placeholder syntax `?` specifically developed (<https://druid.apache.org/docs/latest/querying/sql.html#dynamic-parameters>) to pass large single values into the query as a parameter. This was originally developed for Bloom filters (<https://druid.apache.org/docs/latest/development/extensions-core/bloom-filter.html>) that work with huge literal strings.

I don't think that will help in your case though

because you are passing whole filter expressions

I don't know how generic your usecase is but if you are just looking to do a comparison of count per separate value of a given dimension like your limited example seems to indicate you could have a lookup `dim1Value =&gt; threshold` then apply that lookup in your query and have it as another column and then write the having filter as `HAVING COUNT(*) &gt; threshold`

but then the lookup itself stores the complexity of your rules

I took count just as an example, but similar thing i would need for other aggregation/quantile (post_Agg) too. But all in all it is about comparing with some value for given dimensions (grouping).

I have not used lookup so far, but  seems like if go by lookup route, i would need look up per query and assuming we have 1000 of such queries running, will there be issues with respect to scale ?

can you elaborate on your usecase? It does not fit any pattern I recognize from what you said so far

one such use case is find out if count by service, api is below a threshold.  Every combination of service,api will have a it's own threshold.

usually this data is either only queired by service,api (so just a few filters per query)

or you can pre-store the values when you are saving the data (so you where is `service IN (..) AND over_threshold='yes' ` but if you change the threshold value you need to either reprocess everything or accept that only new data will be using the new threshold

Using lookup on filters does work, but I don't recommend using it for that, but it's possible you add a lookup with "${service_name}:${api}": $threshold
then use:

WHERE always_same_dim &gt;= CAST (lookup(concat(dim_service, ':', dim_api)) AS Int)

A few questions which may help us track it down...

Can you confirm where you took the dataset from - maybe share the link.

Also when you say this is a small instance of druid, I take it you mean you did the small quickstart? What version of Druid are you using? Did you accept default values for the quickstart &amp; batch ingest or tune anything like in the ingestion spec or JVM?

Seems like you've identified the 56th file as being the spot where there was a problem. Is it possible to ingest that file on it's own to prove it is valid?

You could also review the Overlord log, task log and middle manager logs to see if there is more info.

Attached is an image showing which data I was fetching. I added it using the "connect external data" button on the "query" page. I'm pretty much running defaults for everything. I'm using `apache-druid-24.0.1/bin/start-single-server-small` . I'll go search the other logs to see if there's anything notable and once I've copied those files then I'll try to run the 56th &amp; 57th file to see if the issue reoccurs.

On the overseer logs I'm seeing a lot of
`2022-11-30T22:43:50,235 WARN [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - No available [_default_tier] servers or node capacity to assign segment [trips_xaa_2014-04-26T00:00:00.000Z_2014-04-27T00:00:00Z_2022-11-29T11:30:27.958Z]! Current replication: [[_default_tier:1/2]]`

Getting more than 20 of those every millisecond. Makes it hard to find actual information

The files being loaded are of this format:
<https://static.imply.io/example-data/trips/trips_xcc.csv.gz>
<https://static.imply.io/example-data/trips/trips_xcd.csv.gz>
etc.
You can get the full list in the payload of the original job.

So you should be able to redo the ingest but for a single file (query, connect external data) but now select HTTP for the location and paste one of these links in.   Be interesting to also test loading this file using "batch classic" as well as MSQ.

For the overseer logs, maybe you can make it more manageable by doing something like ...
grep -v WARN overseerlogfile &gt;&gt; overseer.new.log.

That should filter out the swarm of WARN messages. Then look for FAIL/ERROR/FATAL messages in this overseer.new.log.

guessing by the error you posted -- i think the connection to the http server that hosts the files got lost partway through a download

you could download them first and switch druid ingestion to use a `local` input source

i'll try it too and see if it works for me

and report back in some number of hours :slightly_smiling_face:

I tried loading these individually (the one that errored out and a couple each side) and they completed without error.  So a timeout is most likely (or a resource issue)

CleanShot 2022-12-02 at 10.02.15@2x.png

FWIW, I tried the same configuration ( `bin/start-single-server-small`) for the NYC cab data for MSQE and the data set is ingested fully with no errors.
So I concur with what Adam and Gian has stated about `http timeout`

Yes when I tried those files individually then I found that there wasn't any issue with the files themselves.

A timeout error matches up with an "unexpected end of stream" error, too. I'll make sure to anticipate such errors when designing the backend that's going to integrate with druid. Thank you all for the valuable insights!

Hmm.. tried twice to load the nyc cabs data from a local disc to a single server - first for the whole dataset and then only for the first 26 input chunks.

Both attempts failed with the following OutOfMemory error after about 10 hours. I'm going to send off another batch now with just 13 input chunks but I feel there must be something I'm missing.

I'm currently using the `batch` `index_parallel` import from local files, with the filter `trips_xa*.csv.gz` . Every other setting is left at its default.
```2022-12-04T21:19:48,994 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory
	at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:1035) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:526) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.run(AbstractBatchIndexTask.java:187) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runSequential(ParallelIndexSupervisorTask.java:1199) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runTask(ParallelIndexSupervisorTask.java:532) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.run(AbstractBatchIndexTask.java:187) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
	at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAndClear(BatchAppenderatorDriver.java:149) ~[druid-server-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAllAndClear(BatchAppenderatorDriver.java:133) ~[druid-server-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.common.task.InputSourceProcessor.process(InputSourceProcessor.java:122) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:922) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	... 11 more
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]
	at java.nio.DirectByteBuffer.&lt;init&gt;(DirectByteBuffer.java:118) ~[?:?]
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
	at org.apache.druid.segment.CompressedPools$4.get(CompressedPools.java:102) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.CompressedPools$4.get(CompressedPools.java:95) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.collections.StupidPool.makeObjectWithHandler(StupidPool.java:170) ~[druid-core-24.0.1.jar:24.0.1]
	at org.apache.druid.collections.StupidPool.take(StupidPool.java:153) ~[druid-core-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.CompressedPools.getByteBuf(CompressedPools.java:110) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.data.DecompressingByteBufferObjectStrategy.fromByteBuffer(DecompressingByteBufferObjectStrategy.java:49) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.data.DecompressingByteBufferObjectStrategy.fromByteBuffer(DecompressingByteBufferObjectStrategy.java:28) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.data.GenericIndexed$BufferIndexed.get(GenericIndexed.java:483) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.data.BlockLayoutColumnarDoublesSupplier$BlockLayoutColumnarDoubles.loadBuffer(BlockLayoutColumnarDoublesSupplier.java:173) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.data.BlockLayoutColumnarDoublesSupplier$1.get(BlockLayoutColumnarDoublesSupplier.java:68) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.data.ColumnarDoubles$1HistoricalDoubleColumnSelector.getDouble(ColumnarDoubles.java:79) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.selector.settable.SettableDoubleColumnValueSelector.setValueFrom(SettableDoubleColumnValueSelector.java:36) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.QueryableIndexIndexableAdapter$RowIteratorImpl.setRowPointerValues(QueryableIndexIndexableAdapter.java:322) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.QueryableIndexIndexableAdapter$RowIteratorImpl.moveToNext(QueryableIndexIndexableAdapter.java:301) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.ForwardingRowIterator.moveToNext(ForwardingRowIterator.java:62) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.MergingRowIterator.lambda$new$0(MergingRowIterator.java:84) ~[druid-processing-24.0.1.jar:24.0.1]
	at java.util.stream.IntPipeline$9$1.accept(IntPipeline.java:368) ~[?:?]
	at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:104) ~[?:?]
	at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:699) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:550) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
	at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:517) ~[?:?]
	at org.apache.druid.segment.MergingRowIterator.&lt;init&gt;(MergingRowIterator.java:92) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.IndexMergerV9.makeMergedTimeAndDimsIterator(IndexMergerV9.java:1360) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.IndexMergerV9.makeIndexFiles(IndexMergerV9.java:293) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.IndexMergerV9.merge(IndexMergerV9.java:1303) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.IndexMergerV9.multiphaseMerge(IndexMergerV9.java:1120) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:1062) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:923) ~[druid-server-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:784) ~[druid-server-24.0.1.jar:24.0.1]
	at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[guava-16.0.1.jar:?]
	... 3 more```

(like `bin/start-single-server-large` or something like that)

&amp; did you change the # of tasks or are you using the default (2)?

btw, on our end we just tried this dataset with 20 tasks (below), it took a little less than 2.5 hrs (reading from http). we also tried a smaller server with 2 tasks and ended up running out of disk space before hitting any of the errors you hit. we're trying again with a bigger disk, but in the meantime, i wonder if the problem you are having is linked to the particular combination of startup script and task count — that's why i asked.

i'd like to get to the bottom of this, since we want the out of box setup to always yield good results :face_with_monocle:

My disc is 500GB exactly. Druid gets pretty much all of that, besides the Debian install. The machine is a `start-single-server-small` and fits the definition - 8vCPU, 64GB RAM

I tried using more than 2 tasks before but the database would hang and not do any work. From reading the documentation I took that to mean that you can only have one worker per physical (virtual) machine.

Since that OutOfMemory exception I posted, I've been gradually increasing load to see where the point of error is. Loading 8 chunks at once seems to be ok - I'm now trying 10. The point where it failed before was loading 13 chunks at once.

(Chunks meaning local `.<http://csv.tz|csv.tz>` files)

hmm… you can definitely do more than one worker per physical machine. although the out of box config may be limited to 2 or 3, it's adjustable

let me try the exact config you're using… `bin/start-single-server-small`, with 2 tasks, on a machine with 64 GB RAM

when you see errors at various numbers of local files, is it always that `OutOfMemoryError: Direct buffer memory` or do you see other ones too?

what SQL are you running? want to make sure i'm doing the same thing

Considering this is just a benchmarking vm, I should probably be able to generate an image for you if you'd be interested - I'd have to let this job finish first which might take 6 hours or might fail depending on our luck haha

I'm not sure what you're referring to regarding SQL - I don't remember installing any variant of SQL when setting up Druid and I've been using the batch ingestion since I started loading local files rather than remote ones.

I've also kicked this off. I took an r5.2xlarge EC2 instance (64g RAM) and started the quickstart with *start-single-server-small*.
Just so we are on the same page, I followed these steps:   Query -&gt; Connect External Data -&gt; Example Data -&gt; NYC Cabs (All files) -&gt; Load with defaults: Max Tasks 2.
Current estimate this will take another 8 or 9 hours.

When you say you were using docker,  are you building a custom image rather than using the druid/docker quickstart we provide (see <https://docs.imply.io/druid/docs/tutorials/docker|here>)?

I'm not actually using docker directly - I've been benchmarking / playing with Druid on GCP and so have options to export images and am using vCPUs etc

I used the conventional druid quickstart and initially used the `Query -&gt; Connect External Data` route, although since hitting that timeout error at the top of the thread, I've since been using the `index_parallel` batch loading method instead. I have the ingestion spec to hand if you would find that convenient

dang, I was running this on my laptop overnight but it went to sleep!

ah okay I was trying using SQL (which I guessed you were using initially due to the stack trace including "msq" in your initial error. that's associated with SQL based ingestion)

Aah no all that time lost! Yes even my VM has discarded the druid server at times when I've forgotten to disown the process before logging off.

In other news, the 10-file ingestion operation succeeded, which means the failure point is somewhere between 11 and 13 inclusive.

I'm now going to see if I can generate an image of my disc and make that publicly available

let me back up a sec and say some stuff that may be clarifying

1) we have two ways of doing batch ingest (or three if you count hadoop, but let's ignore that, it's not common): native batch, and SQL. doc page describing the options is at <https://druid.apache.org/docs/latest/ingestion/index.html#batch|https://druid.apache.org/docs/latest/ingestion/index.html#batch>. In the web console you get SQL in the Query view

2) native batch is older and is associated with a bunch of usability and operational struggles: it requires tuning of memory, is slower, etc. SQL is newer and has an improved backend-- it was just released in the latest major Druid release, 24, with a goal of having ingest be much more usable and getting rid of the need to tune parameters to avoid OutOfMemoryErrors and the like. it does pretty well at this so far! but there are things we do run into here and there, some of which we're fixing in 25. we're very focused on making it rock solid and deprecating native batch as soon as we can. so i recommend using SQL for that reason :slightly_smiling_face:. we're going to try to figure out why you got that http error and ensure there's some kind of appropriate handling of whatever happened. if you run into any issues with SQL ingest using local files we will hunt those down too.

3) parallelism for both options is dictated by the number of worker tasks you use. you can use more than one task per physical machine, up to the number of task slots you have. you can see the number in the Servers tab under middleManagers. I think for a "small" setup the default is 3. There's an issue here with the single server configs: you don't get the full prallelism of your machine! You have 8 vCPUs but can use at most 3 of them for ingestion. the idea here is that in a prod setup, you likely don't want to devote the full resources of your machine to ingest. you likely want the majority available for querying. but for an ingest benchmark, or if you're not doing active querying, this isn't ideal. this is an area we are considering some improvements in, likely an update to the single server configs to increase the number of slots. you can also do this manually using the runtime properties file. if you're interested, let me know and i can advise you on some properties to use 

Goodness me you are too kind to give me such detailed and targeted help!

I'm only expecting queries during working hours and I'm expecting batches of input data only at the end of the day after everybody goes home, so it'd be great if I could sacrifice some query availability for the sake of ingestion performance

Yes I had noticed that the classic batch system was discouraged, but was uncertain whether the new SQL batching supported local files

I realise now that I can reuse the native batch input source logic for the SQL queries. My mistake!

I've now launched an SQL ingestion that's reading 13 files from the local disc - we'll see whether or not I get any OutOfMemory issues - if not then we can probably put the whole issue to bed :slightly_smiling_face:

btw, one of us on our end did try it with a single worker on a small server using https, and ran into a similar gzip thing eventually. so it's not just you!

&gt; Goodness me you are too kind to give me such detailed and targeted help!
no problem, as you can tell we're very motivated to make the batch ingest experience rock solid — big part of that is people should not run into weird errors!

&gt; I'm only expecting queries during working hours and I'm expecting batches of input data only at the end of the day after everybody goes home, so it'd be great if I could sacrifice some query availability for the sake of ingestion performance
yeah, for this, you'd definitely want to adjust the builtin configs. try setting `druid.worker.capacity=9` (instead of default: 3) in `conf/druid/single-server/small/middleManager/runtime.properties`, and setting `maxNumTasks: 9` with your SQL ingest (or setting "Max tasks: 9" in the web console). that should get you maxed out on the CPUs you have

the main risk of raising `druid.worker.capacity` is you might run out of memory if it's raised too high. but i think 9 would be safe on the `small` config -- however if you run into memory issues after raising it, let us know since it may mean i did my math wrong :slightly_smiling_face:

one thing we're working on for the near future is a script to do this math for you and spit out configs optimized for your hardware — <@U044GNYJSTA> has been looking into that in <https://github.com/apache/druid/pull/13365>. first version won't include an "ingestion optimized" mode but i am thinking this is a nice idea for a follow up

I ran the 'NYC Taxi Cabs' example ingest overnight. It processed all 74 files without any memory errors (although I did eventually run out of disk space - doh!)
I used the 'single-server/small' quickstart on aa 64g linux server.   My point being we can definitely rule out any issues with the files themselves and consider this to be environmental/resource related.
To summarise, you do not need to use SQL to ingest data, you can do native batch load (Load data -&gt; Batch Classic). Hopefully this will get you moving!

Funnily enough my ingest of 13 files also failed because I ran out of disk space haha.

I tried to do something clever by moving the local files into a storage bucket and linking it via FUSE but unfortunately I ran into an `AccessDeniedException` . I also tried using a symlink to try and convince Java to delegate file access to bash, which can access the mounted bucket just fine - but alas Druid doesn't error but also can't see the files.

I've compromised by keeping just those 13 files on my local disk and seeing if I can get them ingested. I've also cranked up the config to try and use all 8 cores - I'll report back with results :slightly_smiling_face:

I've managed to ingest another 26 files worth of data but I'm out of storage space now - I think I'm happy with the reliability of this ingestion :slightly_smiling_face:

Good to hear.  Best of luck with the rest of your work. we are here if you need us!

Stacktrace :
```java.lang.RuntimeException: org.apache.druid.msq.indexing.error.MSQException: BroadcastTablesTooLarge: Size of the broadcast tables exceed the memory reserved for them (memory reserved for broadcast tables = 223640812 bytes)
 at org.apache.druid.java.util.common.Either.valueOrThrow(Either.java:91)
 at org.apache.druid.frame.processor.FrameProcessorExecutor$1ExecutorRunnable.runProcessorNow(FrameProcessorExecutor.java:258)
 at org.apache.druid.frame.processor.FrameProcessorExecutor$1ExecutorRunnable.run(FrameProcessorExecutor.java:137)
 at org.apache.druid.msq.exec.WorkerImpl$1$2.run(WorkerImpl.java:634)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:251)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.druid.msq.indexing.error.MSQException: BroadcastTablesTooLarge: Size of the broadcast tables exceed the memory reserved for them (memory reserved for broadcast tables = 223640812 bytes)
 at org.apache.druid.msq.querykit.BroadcastJoinHelper.buildBroadcastTablesIncrementally(BroadcastJoinHelper.java:120)
 at org.apache.druid.msq.querykit.BaseLeafFrameProcessor.initializeSegmentMapFn(BaseLeafFrameProcessor.java:155)
 at org.apache.druid.msq.querykit.BaseLeafFrameProcessor.runIncrementally(BaseLeafFrameProcessor.java:106)
 at org.apache.druid.frame.processor.FrameProcessors$1FrameProcessorWithBaggage.runIncrementally(FrameProcessors.java:70)
 at org.apache.druid.frame.processor.FrameProcessorExecutor$1ExecutorRunnable.runProcessorNow(FrameProcessorExecutor.java:229)
 ... 8 more```

Request :

```REPLACE INTO "table1"
OVERWRITE WHERE __time &gt;= TIMESTAMP '2022-11-22 00:00:00' and __time &lt; TIMESTAMP '2022-12-01 10:00:00'
WITH source as (
  SELECT *
  FROM "table1"
  WHERE __time &gt;= TIMESTAMP '2022-11-22 00:00:00' and __time &lt; TIMESTAMP '2022-12-01 10:00:00'
),
events as (
  SELECT
    *
  FROM "events"
  WHERE __time &gt;= TIMESTAMP '2022-11-30 15:00:00'
)
SELECT
  source.__time,
  source.id,
  source.country,
  SUM(source."count") + SUM(events."count") as "count",
  GREATEST(MAX(source.click), MAX(events.click)) as "click",
  GREATEST(MAX(source.impression), MAX(events.impression)) as "impression"
FROM source
  LEFT JOIN events on events.id = source.id
GROUP BY 1, 2, 3
PARTITIONED BY HOUR```


Does this thread help : <https://www.druidforum.org/t/broadcasttablestoolarge/8089>

So it should be
```BigDataSource left join SmalDataSource on  ```


You might not need all the cols from events. That will probably reduce the size

Hi Karan, thank you for your detailed answers!

Unfortunately it’s already `BigDataSource left join SmalDataSource on`, and I need every columns from events, so I will try increasing the memory of the peon, and run the query multiple time on smaller intervals for my events table.

Screenshot 2022-12-01 at 4.11.15 PM.png

Please try increasing replicationThrottleLimit and maxSegmentsInNodeLoadingQueue in coordinator dynamic config
If they are already high and there's no progress, check if your segment loading queues are stuck in the servers page.

To how much i can increase  replicationThrottleLimit and maxSegmentsInNodeLoadingQueue

replicationThrottleLimit -&gt; 10000
maxSegmentsInNodeLoadingQueue -&gt; 500

Screenshot 2022-12-01 at 4.22.46 PM.png

we have given replicationThrottleLimit only 10... should i increase it to 1000 ?

And maxSegmentsInNodeLoadingQueue was 100

Sure. Even 10000 should be fine unless it affects the coordinator cycle too much

after increasing it should i restart any coordinator pod ?

And can i increase maxSegmentsInNodeLoadingQueue from 100 to 500 ?

These are dynamic configurations and the coordinator picks them up after each cycle

Actually we are doing these changes in prod cluster... so will there be any impact ?

You could leave maxSegmentsInNodeLoadingQueue at 100 but please increase replicationThrottleLimit

There won't be too much of an impact with 1000

ohh i increased maxSegmentsInNodeLoadingQueue to 500 and saved the changes

It can make coordinator cycles a bit slow but you can decrease it after nodes become available. Was replicationThrottleLimit not increased? (10 -&gt; 1000 / 5000 might help)

replicationThrottleLimit also i increased it to 1000

can i decrease  that maxSegmentsInNodeLoadingQueue from 500 to 100 right away ?

<@U0319HD8HEC> I was not seeing any improvement of datasource availability... will it take some time to recover ?

unavailable segments are still increasing

Could you please share your historical loading queues' snapshot from the servers tab?

&gt; unavailable segments are still increasing
Please try this query:
`select num_replicas, count(*) from sys.segments group by num_replicas`

Actually 3377 segments are unavailable it has replication factor 0

No actually its not full
```# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         125G   11G  114G   9% /
tmpfs            64M     0   64M   0% /dev
tmpfs           120G     0  120G   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
/dev/nvme2n1    1.7T   73G  1.6T   5% /mnt/datadisk2
tmpfs           120G   16K  120G   1% /vault/secrets
/dev/root       125G   11G  114G   9% /etc/hosts
/dev/nvme0n1    1.7T   73G  1.6T   5% /mnt/datadisk1
/dev/nvme3n1    1.7T   73G  1.6T   5% /mnt/datadisk3
/dev/nvme1n1    1.7T   74G  1.6T   5% /mnt/datadisk0
tmpfs           120G   12K  120G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           120G     0  120G   0% /proc/acpi
tmpfs           120G     0  120G   0% /proc/scsi
tmpfs           120G     0  120G   0% /sys/firmware
#troubleshooting ```

<@U0319HD8HEC> what is that current size and max size and where can we increase it... because we have disk space of 2TB on each node and we are using theree nodes for historical running as pods in kubernetes

You seem to have configured your historicals to use only 300GB though. Could you please try increasing it?

Where can i increase it... is it from druid console ?

<https://druid.apache.org/docs/latest/configuration/index.html#historical-general-configuration>

Oh <@U0319HD8HEC> you beat me to it :smile: :smile:

This would require restarting the historicals. You could also add extra historicals

<@U03EXS199PD> there is some other interesting bits in the basic cluster tuning guide - here’s where it mentions the setting that <@U0319HD8HEC> is pointing you to <https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html#segment-cache-size>

This was the parameter is it
```druid.segmentCache.locations=[{"path": "/mnt/datadisk0", "maxSize": 300000000000}, {"path": "/mnt/datadisk1", "maxSize": 300000000000}]```

<@U0319HD8HEC> we have given druid.server.maxSize as 37.5GB... but why in console it is showing as 300GB

<@U0319HD8HEC> After increasing the size it is recovering now... Thanks a lot

the best way is to do a test load and see how much space it takes up; compression can vary based on the dataset

a good rule of thumb is druid data is generally going to be slightly larger than parquet

so if your original is parquet you can use, like, 20–30% more than that as an estimate? (that will vary too; in both directions; so a test load is still useful)

note that the compression is improving soon due to <https://github.com/apache/druid/pull/12277>

will still be somewhat larger than parquet (we do have extra indexes, etc that parquet does not have) — but not as much larger as today

All great points by Gian. I will also point out that data locality matters too. Depending on how you are ingesting the data, you should use range partitioning (automatic if you are using SQL based ingest).
<https://imply.io/blog/multi-dimensional-range-partitioning>

While I have not directly verified this, range partitioning would make the dictionary compression (that Gian linked), even more effective. Because the dimensions with similar values are going to land in the same segment and the compression algorithm does much better when the values are similar.

Dictionary compression is going to be available in 25.0 which is just around the corner.

Brilliant! Thank you both very much for the valuable information!

IIRC this is a harmless error caused due to
<https://issues.apache.org/jira/browse/CURATOR-526>
Try removing the clientport from zoo.cfg

Try the actual server name instead of the localhost IP(0.0.0.0), I have seen some issues by using the 0.0.0.0

Hello, there is `druid.segmentCache.lazyLoadOnStart` does that fit the bill?  <https://druid.apache.org/docs/latest/configuration/index.html#storing-segments>

We're giving the option a try now. Thanks.

What data format are you using, and what is your observation in terms of ingested MB / second per task ? I’ve done a few large scale test experiments and I noticed ai can get about 2X troughput per task if I use protobuf instead of json

Also this was not obvious to me at the beginning but it seems like for best results you need to match number of tasks to number of Kafka partitions 

Mostly json and avro. Great point on trying out protobuf.

Avro definitely performed better than JSON for us. Probably worth trying out protobuf as well.

Been also playing with some Kafka consumer buffer settings but it did not seem to move the needle much 

&gt; Also this was not obvious to me at the beginning but it seems like for best results you need to match number of tasks to number of Kafka partitions
Yeah, number of Kafka partitions is the upper bound. We had to resort to increasing our Kafka partitions because we were maxed out on number of ingestion tasks.

a performance analysis workflow i use:

1) is CPU usage "high enough" that you're making good use of your hw? Note that you don't want it at 100%, since then you have no room to recover from failures or run queries or handle spikes. But you don't want it at 5–10% either, that's too low. Probably 50–80% range is healthy depending on your data patterns.

2) if CPU usage is not high enough, figure out why. you might need to adjust configs to use more CPUs. or you might be blocked on something that isn't CPU, like disk I/O or network.

3) OTOH, if CPU usage is "high enough" next step is figure out what CPU being used on. here, profiling is the best tool. use your favorite java profiling tool; or, if you don't have one, you can take flame graphs manually: <https://support.imply.io/hc/en-us/articles/360033747953-Profiling-Druid-queries-using-flame-graphs>. a brief plug for imply :slightly_smiling_face: — if you're working with us, we have a continuous profiling tool we can enable for your cluster that lets you look at aggregate profiling info across the fleet or drill down into individual servers/tasks/datasources/etc. very helpful for finding bottlenecks and is part of what we do when offering <https://imply.io/total-value-guarantee>

4) once you find what CPU is being spent on, you can take actions to optimize. for example, if it's mainly in JSON parsing, then you can consider switching to a diff format. you can also ask around here for further input