Apache Druid

when I used to manage a large druid cluster with many varied users on it, I'd approach this by doing a canary deploy and watching for any increase in errors or performance degradations

like: deploy one or two new historicals &amp; one or two new MMs, check in on their error rates and performance after about a day

if looks good then update the rest of the data servers 

ho… interesting. I guess I can replicate this on Kubernetes, just add 10% extra pods with the new Druid version and observe the error rate.

Us too. We want to rely on Kubernetes+Docker’s namespace and cgroup so we use the smallest number of task slots. Two per MM.

interesting, that'd result in a lot of middlemanager nodes for a large cluster

yes, but the memory behavior is predictable. If you have too many peons in a pod, one bad subtask can bulldoze the other peons, creating difficult to track errors.

temporary disk usage on EBS is more predictable as well because the mount is only shared by 2 peons

Based on our experience, it is better to have full control on compaction jobs. So we schedule them ourselves with Airflow.

Looks like, we were able to complete by running through periods through api calls. .

image.png

Is it possible that your compaction was within the "skip offset from latest" (default: 1 day)

in older versions there was also an "input segment size" limit that would cause compaction to not run if you had above a certain amount of bytes per time chunk

i believe it's been removed now, but if you aren't on the latest, yours might still have it

Yes you also need to include the `type`. Now sure what kind of DSL you are using there but I would expect adding something like:
```druid_auth_authenticator_basic_authorizerType: 'basic',```
should fix that error.

Thanks, will try that. Didn't see that option

This is our working configuration

druid.auth.authenticatorChain=["MyBasicPostgreSQLAuthenticator"]
druid.auth.authenticator.MyBasicPostgreSQLAuthenticator.type=basic
druid.auth.authenticator.MyBasicPostgreSQLAuthenticator.initialAdminPassword=&lt;ADmin Password&gt;
druid.auth.authenticator.MyBasicPostgreSQLAuthenticator.initialInternalClientPassword= &lt;Client password&gt;
druid.auth.authenticator.MyBasicPostgreSQLAuthenticator.credentialsValidator.type=metadata
druid.auth.authenticator.MyBasicPostgreSQLAuthenticator.skipOnFailure=false
druid.auth.authenticator.MyBasicPostgreSQLAuthenticator.authorizerName=MyBasicPostgreSQLAuthorizer
# Escalator
druid.escalator.type=basic
druid.escalator.internalClientUsername=druid_system
druid.escalator.internalClientPassword=&lt;Client Passowrd&gt;
druid.escalator.authorizerName=MyBasicPostgreSQLAuthorizer
druid.auth.authorizers=["MyBasicPostgreSQLAuthorizer"]
druid.auth.authorizer.MyBasicPostgreSQLAuthorizer.type=basic

added missing conf but getting same error:
```druid_auth_authenticator_basic_authorizerName: 'basic',                  druid_auth_authenticator_basic_authorizerType: 'basic',                  druid_auth_authenticatorChain: '["basic"]',```

```2022-09-25T18:46:30,681 INFO [main] org.eclipse.jetty.util.log - Logging initialized @7259ms to org.eclipse.jetty.util.log.Slf4jLog
2022-09-25T18:46:30,718 INFO [main] org.apache.druid.server.initialization.jetty.JettyServerModule - Creating http connector with port [8083]
2022-09-25T18:46:30,958 ERROR [main] org.apache.druid.cli.CliHistorical - Error when starting up.  Failing.
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error in custom provider, <http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>: server initialization exception
  at org.apache.druid.server.initialization.jetty.JettyServerModule.getServer(JettyServerModule.java:177) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.initialization.jetty.JettyServerModule)
  at org.apache.druid.server.initialization.jetty.JettyServerModule.getServer(JettyServerModule.java:177) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.initialization.jetty.JettyServerModule)
  while locating org.eclipse.jetty.server.Server

1 error
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1050) ~[guice-4.1.0.jar:?]
    at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:150) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:136) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:94) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:63) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.Main.main(Main.java:112) ~[druid-services-24.0.0.jar:24.0.0]
Caused by: <http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>: server initialization exception
    at org.apache.druid.server.initialization.jetty.JettyServerModule.makeAndInitializeServer(JettyServerModule.java:420) ~[druid-server-24.0.0.jar:24.0.0]
    at org.apache.druid.server.initialization.jetty.JettyServerModule.getServer(JettyServerModule.java:177) ~[druid-server-24.0.0.jar:24.0.0]
    at org.apache.druid.server.initialization.jetty.JettyServerModule$$FastClassByGuice$$ff9924d8.invoke(&lt;generated&gt;) ~[druid-server-24.0.0.jar:24.0.0]
    at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[guice-4.1.0.jar:?]
    ... 6 more
Caused by: com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Problem parsing object at prefix[druid.auth.authenticator.basic]: Missing type id when trying to resolve subtype of [simple type, class org.apache.druid.server.security.Authenticator]: missing type id property 'type'
 at [Source: UNKNOWN; line: -1, column: -1].
  at org.apache.druid.server.initialization.AuthenticatorMapperModule.configure(AuthenticatorMapperModule.java:54) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.initialization.AuthenticatorMapperModule)
  at org.apache.druid.server.initialization.AuthenticatorMapperModule.configure(AuthenticatorMapperModule.java:54) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.initialization.AuthenticatorMapperModule)
  while locating org.apache.druid.server.security.AuthenticatorMapper

1 error
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054) ~[guice-4.1.0.jar:?]
    at org.apache.druid.cli.QueryJettyServerInitializer.initialize(QueryJettyServerInitializer.java:99) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.server.initialization.jetty.JettyServerModule.makeAndInitializeServer(JettyServerModule.java:417) ~[druid-server-24.0.0.jar:24.0.0]
    at org.apache.druid.server.initialization.jetty.JettyServerModule.getServer(JettyServerModule.java:177) ~[druid-server-24.0.0.jar:24.0.0]
    at org.apache.druid.server.initialization.jetty.JettyServerModule$$FastClassByGuice$$ff9924d8.invoke(&lt;generated&gt;) ~[druid-server-24.0.0.jar:24.0.0]
    at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) ~[guice-4.1.0.jar:?]
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[guice-4.1.0.jar:?]
    ... 6 more
Caused by: java.lang.IllegalArgumentException: Missing type id when trying to resolve subtype of [simple type, class org.apache.druid.server.security.Authenticator]: missing type id property 'type'
 at [Source: UNKNOWN; line: -1, column: -1]```

Show us the final config that was generated in the pod.

configmap of cluster
```apiVersion: v1
data:
  DRUID_LOG4J: &lt;?xml version="1.0" encoding="UTF-8" ?&gt;&lt;Configuration status="WARN"&gt;&lt;Appenders&gt;&lt;Console
    name="Console" target="SYSTEM_OUT"&gt;&lt;PatternLayout pattern="%d{ISO8601} %p [%t]
    %c - %m%n"/&gt;&lt;/Console&gt;&lt;/Appenders&gt;&lt;Loggers&gt;&lt;Root level="info"&gt;&lt;AppenderRef ref="Console"/&gt;&lt;/Root&gt;&lt;Logger
    name="org.apache.druid.jetty.RequestLog" additivity="false" level="DEBUG"&gt;&lt;AppenderRef
    ref="Console"/&gt;&lt;/Logger&gt;&lt;/Loggers&gt;&lt;/Configuration&gt;
  DRUID_USE_CONTAINER_IP: "true"
  druid_auth_authenticator_basic_authorizerName: basic
  druid_auth_authenticator_basic_authorizerType: basic
  druid_auth_authenticator_basic_initialAdminPassword: 1234
  druid_auth_authenticator_basic_type: basic
  druid_auth_authenticatorChain: '["basic"]'
  druid_auth_authorizer_basic_type: basic
  druid_auth_authorizers: '["basic"]'
  druid_emitter: statsd
  druid_emitter_http_recipientBaseUrl: <http://druid_exporter_url>:druid_exporter_port/druid
  druid_emitter_logging_logLevel: debug
  druid_emitter_statsd_dogstatsd: "true"
  druid_emitter_statsd_dogstatsdConstantTags: '["cluster:cluster-test"]'
  druid_emitter_statsd_hostname: 10.3.111.222
  druid_emitter_statsd_includeHost: "true"
  druid_emitter_statsd_port: "8125"
  druid_extensions_directory: /opt/druid/extensions
  druid_extensions_hadoopDependenciesDir: /opt/druid/hadoop-dependencies
  druid_extensions_loadList: '["druid-histogram","druid-datasketches","postgresql-metadata-storage","druid-kinesis-indexing-service","druid-parquet-extensions","druid-basic-security","druid-s3-extensions","druid-kafka-indexing-service","statsd-emitter","druid-avro-extensions","druid-lookups-cached-global"]'
  druid_global_http_requestBuffersize: "16384"
  druid_indexer_logs_directory: /opt/data/indexing-logs
  druid_indexer_logs_disableAcl: "true"
  druid_indexer_logs_s3Bucket: druid-indexing-metadata-dev
  druid_indexer_logs_s3Prefix: druid/indexing-logs
  druid_indexer_logs_type: s3
  druid_indexer_task_baseDir: /opt/druid/var/tmp/
  druid_metadata_storage_connector_connectURI: jdbc:<postgresql://druid-metadata-storage-test.c12rj4a7dt4u.us-east-2.rds.amazonaws.com:5432/druid?autoReconnect=true>
  druid_metadata_storage_connector_password: 4321
  druid_metadata_storage_connector_user: druid
  druid_metadata_storage_type: postgresql
  druid_monitoring_emissionPeriod: PT15S
  druid_selectors_coordinator_serviceName: druid/coordinator
  druid_selectors_indexing_serviceName: druid/overlord
  druid_server_http_maxRequestHeaderSize: "16384"
  druid_startup_logging_logProperties: "true"
  druid_storage_baseKey: druid/segments
  druid_storage_bucket: druid-indexing-metadata-test
  druid_storage_disableAcl: "true"
  druid_storage_type: s3
  druid_zk_service_host: druid-cluster-test-zookeeper-headless:2181
  druid_zk_service_terminateDruidProcessOnConnectFail: "true"```

new error I got with above:
```2022-09-26T09:44:07,609 WARN [main] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 9) in 1,528ms.
com.fasterxml.jackson.core.JsonParseException: Input does not start with Smile format header (first byte = 0x7b) -- rather, it starts with '{' (plain JSON input?) -- can not parse
 at [Source: (byte[])"{"error":"Unauthorized request","errorMessage":null,"errorClass":null,"host":"druid-cluster-test-coordinator-767dfd47b6-bwdxn"}"; line: -1, column: 0]
    at com.fasterxml.jackson.dataformat.smile.SmileParserBootstrapper.constructParser(SmileParserBootstrapper.java:133) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory._createParser(SmileFactory.java:460) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory.createParser(SmileFactory.java:366) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory.createParser(SmileFactory.java:353) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory.createParser(SmileFactory.java:29) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3292) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:262) ~[?:?]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:192) ~[?:?]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:190) ~[?:?]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.initUserMaps(CoordinatorPollingBasicAuthenticatorCacheManager.java:289) ~[?:?]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.start(CoordinatorPollingBasicAuthenticatorCacheManager.java:108) ~[?:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_275]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
    at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:152) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:136) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:94) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:63) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.Main.main(Main.java:112) ~[druid-services-24.0.0.jar:24.0.0]
2022-09-26T09:44:09,146 WARN [main] org.apache.druid.java.util.common.RetryUtils - Retrying (2 of 9) in 1,203ms.
com.fasterxml.jackson.core.JsonParseException: Input does not start with Smile format header (first byte = 0x7b) -- rather, it starts with '{' (plain JSON input?) -- can not parse
 at [Source: (byte[])"{"error":"Unauthorized request","errorMessage":null,"errorClass":null,"host":"druid-cluster-test-coordinator-767dfd47b6-bwdxn"}"; line: -1, column: 0]
    at com.fasterxml.jackson.dataformat.smile.SmileParserBootstrapper.constructParser(SmileParserBootstrapper.java:133) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory._createParser(SmileFactory.java:460) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory.createParser(SmileFactory.java:366) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory.createParser(SmileFactory.java:353) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory.createParser(SmileFactory.java:29) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3292) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:262) ~[?:?]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:192) ~[?:?]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:190) ~[?:?]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.initUserMaps(CoordinatorPollingBasicAuthenticatorCacheManager.java:289) ~[?:?]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.start(CoordinatorPollingBasicAuthenticatorCacheManager.java:108) ~[?:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_275]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
    at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:152) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:136) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:94) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:63) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.Main.main(Main.java:112) ~[druid-services-24.0.0.jar:24.0.0]
2022-09-26T09:44:09,396 INFO [NodeRoleWatcher[COORDINATOR]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node [<http://10.3.105.8:8081>] of role [coordinator] detected.
2022-09-26T09:44:10,353 WARN [main] org.apache.druid.java.util.common.RetryUtils - Retrying (3 of 9) in 4,230ms.
com.fasterxml.jackson.core.JsonParseException: Input does not start with Smile format header (first byte = 0x7b) -- rather, it starts with '{' (plain JSON input?) -- can not parse
 at [Source: (byte[])"{"error":"Unauthorized request","errorMessage":null,"errorClass":null,"host":"druid-cluster-test-coordinator-767dfd47b6-bwdxn"}"; line: -1, column: 0]
    at com.fasterxml.jackson.dataformat.smile.SmileParserBootstrapper.constructParser(SmileParserBootstrapper.java:133) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory._createParser(SmileFactory.java:460) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory.createParser(SmileFactory.java:366) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory.createParser(SmileFactory.java:353) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory.createParser(SmileFactory.java:29) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3292) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:262) ~[?:?]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:192) ~[?:?]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:190) ~[?:?]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.initUserMaps(CoordinatorPollingBasicAuthenticatorCacheManager.java:289) ~[?:?]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.start(CoordinatorPollingBasicAuthenticatorCacheManager.java:108) ~[?:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_275]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
    at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:152) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:136) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:94) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:63) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.Main.main(Main.java:112) ~[druid-services-24.0.0.jar:24.0.0]```

am I missing also following conf?
```druid_auth_authenticator_basic_initialInternalClientPassword=1234
druid_escalator_authorizerName=basic
druid_escalator_internalClientPassword=1234
druid_escalator_internalClientUsername=druid_system
druid_escalator_type=basic```

changed config to
```apiVersion: v1
data:
  DRUID_LOG4J: &lt;?xml version="1.0" encoding="UTF-8" ?&gt;&lt;Configuration status="WARN"&gt;&lt;Appenders&gt;&lt;Console
    name="Console" target="SYSTEM_OUT"&gt;&lt;PatternLayout pattern="%d{ISO8601} %p [%t]
    %c - %m%n"/&gt;&lt;/Console&gt;&lt;/Appenders&gt;&lt;Loggers&gt;&lt;Root level="info"&gt;&lt;AppenderRef ref="Console"/&gt;&lt;/Root&gt;&lt;Logger
    name="org.apache.druid.jetty.RequestLog" additivity="false" level="DEBUG"&gt;&lt;AppenderRef
    ref="Console"/&gt;&lt;/Logger&gt;&lt;/Loggers&gt;&lt;/Configuration&gt;
  DRUID_USE_CONTAINER_IP: "true"
  druid_auth_authenticator_MyBasicMetadataAuthenticator_authorizerName: MyBasicMetadataAuthenticator
  druid_auth_authenticator_MyBasicMetadataAuthenticator_authorizerType: basic
  druid_auth_authenticator_MyBasicMetadataAuthenticator_credentialsValidator_type: metadata
  druid_auth_authenticator_MyBasicMetadataAuthenticator_initialAdminPassword: 1234
  druid_auth_authenticator_MyBasicMetadataAuthenticator_skipOnFailure: "false"
  druid_auth_authenticator_MyBasicMetadataAuthenticator_type: basic
  druid_auth_authenticator_basic_initialInternalClientPassword: 4321
  druid_auth_authenticatorChain: '["MyBasicMetadataAuthenticator"]'
  druid_auth_authorizer_MyBasicMetadataAuthenticator_type: basic
  druid_auth_authorizers: '["MyBasicMetadataAuthenticator"]'
  druid_emitter: statsd
  druid_emitter_http_recipientBaseUrl: <http://druid_exporter_url>:druid_exporter_port/druid
  druid_emitter_logging_logLevel: debug
  druid_emitter_statsd_dogstatsd: "true"
  druid_emitter_statsd_dogstatsdConstantTags: '["cluster:cluster-test"]'
  druid_emitter_statsd_hostname: 10.3.153.124
  druid_emitter_statsd_includeHost: "true"
  druid_emitter_statsd_port: "8125"
  druid_escalator_authorizerName: MyBasicMetadataAuthenticator
  druid_escalator_internalClientPassword: 1234
  druid_escalator_internalClientUsername: druid_system
  druid_escalator_type: basic
  druid_extensions_directory: /opt/druid/extensions
  druid_extensions_hadoopDependenciesDir: /opt/druid/hadoop-dependencies
  druid_extensions_loadList: '["druid-histogram","druid-datasketches","postgresql-metadata-storage","druid-kinesis-indexing-service","druid-parquet-extensions","druid-basic-security","druid-s3-extensions","druid-kafka-indexing-service","statsd-emitter","druid-avro-extensions","druid-lookups-cached-global"]'
  druid_global_http_requestBuffersize: "16384"
  druid_indexer_logs_directory: /opt/data/indexing-logs
  druid_indexer_logs_disableAcl: "true"
  druid_indexer_logs_s3Bucket: druid-indexing-metadata-test
  druid_indexer_logs_s3Prefix: druid/indexing-logs
  druid_indexer_logs_type: s3
  druid_indexer_task_baseDir: /opt/druid/var/tmp/
  druid_metadata_storage_connector_connectURI: jdbc:<postgresql://druid-metadata-storage-test.abcd.us-west-1.rds.amazonaws.com:5432/druid?autoReconnect=true>
  druid_metadata_storage_connector_password: 4321
  druid_metadata_storage_connector_user: druid
  druid_metadata_storage_type: postgresql
  druid_monitoring_emissionPeriod: PT15S
  druid_monitoring_monitors: '["org.apache.druid.java.util.metrics.JvmMonitor"]'
  druid_selectors_coordinator_serviceName: druid/coordinator
  druid_selectors_indexing_serviceName: druid/overlord
  druid_server_http_maxRequestHeaderSize: "16384"
  druid_startup_logging_logProperties: "true"
  druid_storage_baseKey: druid/segments
  druid_storage_bucket: druid-indexing-metadata-test
  druid_storage_disableAcl: "true"
  druid_storage_type: s3
  druid_zk_service_host: druid-cluster-test-zookeeper-headless:2181
  druid_zk_service_terminateDruidProcessOnConnectFail: "true"
kind: ConfigMap```
but getting
```
2022-09-26T14:03:20,344 WARN [main] org.apache.druid.java.util.common.RetryUtils - Retrying (3 of 9) in 4,526ms.
com.fasterxml.jackson.core.JsonParseException: Input does not start with Smile format header (first byte = 0x3c) and parser has REQUIRE_HEADER enabled: can not parse
 at [Source: (byte[])"&lt;html&gt;
&lt;head&gt;
&lt;meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/&gt;
&lt;title&gt;Error 401 Unauthorized&lt;/title&gt;
&lt;/head&gt;
&lt;body&gt;&lt;h2&gt;HTTP ERROR 401 Unauthorized&lt;/h2&gt;
&lt;table&gt;
&lt;tr&gt;&lt;th&gt;URI:&lt;/th&gt;&lt;td&gt;/druid-ext/basic-security/authentication/db/MyBasicMetadataAuthenticator/cachedSerializedUserMap&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;th&gt;STATUS:&lt;/th&gt;&lt;td&gt;401&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;th&gt;MESSAGE:&lt;/th&gt;&lt;td&gt;Unauthorized&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;th&gt;SERVLET:&lt;/th&gt;&lt;td&gt;default&lt;/td&gt;&lt;/tr&gt;
&lt;/table&gt;

&lt;/body&gt;
&lt;/html&gt;
"; line: -1, column: 0]
    at com.fasterxml.jackson.dataformat.smile.SmileParserBootstrapper.constructParser(SmileParserBootstrapper.java:133) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory._createParser(SmileFactory.java:460) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory.createParser(SmileFactory.java:366) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory.createParser(SmileFactory.java:353) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.dataformat.smile.SmileFactory.createParser(SmileFactory.java:29) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3292) ~[jackson-databind-2.10.5.1.jar:2.10.5.1]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:262) ~[?:?]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:192) ~[?:?]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:190) ~[?:?]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.initUserMaps(CoordinatorPollingBasicAuthenticatorCacheManager.java:289) ~[?:?]
    at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.start(CoordinatorPollingBasicAuthenticatorCacheManager.java:108) ~[?:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_275]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
    at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:152) ~[druid-core-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:136) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:94) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:63) ~[druid-services-24.0.0.jar:24.0.0]
    at org.apache.druid.cli.Main.main(Main.java:112) ~[druid-services-24.0.0.jar:24.0.0]```
not sure what I am doing wrong here

looks like I got it working with following config
```druid_auth_authenticator_MyBasicMetadataAuthenticator_authorizerName: MyBasicMetadataAuthorizer
  druid_auth_authenticator_MyBasicMetadataAuthenticator_authorizerType: basic
  druid_auth_authenticator_MyBasicMetadataAuthenticator_initialAdminPassword: 1234
  druid_auth_authenticator_MyBasicMetadataAuthenticator_initialInternalClientPassword: 1234
  druid_auth_authenticator_MyBasicMetadataAuthenticator_skipOnFailure: "false"
  druid_auth_authenticator_MyBasicMetadataAuthenticator_type: basic
  druid_auth_authenticatorChain: '["MyBasicMetadataAuthenticator"]'
  druid_auth_authorizer_MyBasicMetadataAuthorizer_type: basic
  druid_auth_authorizers: '["MyBasicMetadataAuthorizer"]'
  druid_escalator_authorizerName: MyBasicMetadataAuthorizer
  druid_escalator_internalClientPassword: 1234
  druid_escalator_internalClientUsername: druid_system
  druid_escalator_type: basic```

is there enough memory for 24 gb heap?  How much memory is available per node?

I think around 15GB, but I thought it would not get the 24g but whatever we have in `druid_processing_buffer_sizeBytes` by `druid_processing_numThreads` , I currently have 12 and 4 respectively which comes down to less than 15GB

This looks like the issue, I would make those much smaller as it is more than each node can allocate
```config: {
        DRUID_XMX: '24g',
        DRUID_XMS: '24g',```


This does not look right.
```[
{"path"\:"/opt/druid/var/druid/segment-cache","maxSize"\:"1754200000000"}, {"path"\:"/opt/druid/var/druid/segment-cache","maxSize"\:"1858900000000"}, {"path"\:"/opt/druid/var/druid/segment-cache","maxSize"\:"1858900000000"}, {"path"\:"/opt/druid/var/druid/segment-cache","maxSize"\:"1858900000000"}
]```
They are all pointing at the same path.

Shouldn't they be one entry in the array per replica? It's true there are 12 replicas and only 4 entries in the array

O should really be one entry in the array?

mm I am wrong here, should really be different paths yeah

if the historical logs just end, without it starting up successfully, for clues you can check dmesg (to see if it was oom-killed) or historical.stdout.log (for messages from jvm)

typically the reason is out of memory-- if it's configured to use more memory than you have available on the system, or more off heap memory than the jvm direct memory limit

It has one, but very basic -- looks at things more like "is there a filter", "is there a join", and "is there a subquery" rather than the specific filters and functions involved

I wonder if you can get what you're looking for using query priorities or timeouts?

`query priorities or timeouts` ? I am not sure I follow, custom timeout for every user? I don’t know about query priorities as well.

I was wondering if you knew, a priori, certain users are the ones that tend to perform "unsavory" queries

If you do then you can run those at lower priority or with shorter timeouts

If it could be _any_ user then my suggestion isn't as useful

well… we do know, after the fact, the usernames of these unsavory queries because we processed the request logs.

Actually it looks like we are using postgresql for metadata and the extension is loaded so the driver should be in the classpath however it fails to load the lookup due to an error with calcite

the actual error is
``` 2022-09-26T20:33:35,340 INFO [sql[cc91f75c-447f-437a-96b2-1053381b7c6c]] org.apache.druid.server.log.LoggingRequestLogger - 2022-09-26T20:33:35.328Z    10.3.120.136        {"sqlQuery/time":11,"sqlQuery/planningTimeMs":0,"sqlQuery/bytes":0,"s │
│ uccess":false,"identity":"admin","context":{"nativeQueryIds":"[]","sqlQueryId":"cc91f75c-447f-437a-96b2-1053381b7c6c"},"exception":"org.apache.druid.sql.SqlPlanningException: org.apache.calcite.runtime.CalciteContextException: From line 1, c │
│ olumn 22 to line 1, column 41: Object 'user_to_city' not found within 'lookup'"}    {"query":"SELECT \"k\", \"v\" FROM lookup.user_to_city LIMIT 5000","context":{"nativeQueryIds":"[]","sqlQueryId":"cc91f75c-447f-437a-96b2-1053381b7c6c"}}   │
│ 2022-09-26T20:33:35,342 DEBUG [qtp1747862622-162] org.apache.druid.jetty.RequestLog - 10.3.120.136 POST //10.3.116.255:8082/druid/v2/sql HTTP/1.1 400                                                                                             │
│ 2022-09-26T20:33:43,709 DEBUG [qtp1747862622-155] org.apache.druid.jetty.RequestLog - 10.3.114.17 GET //10.3.116.255:8082/status/health HTTP/1.1 200                                                                                              │                                                                                             │
│ 2022-09-26T20:34:19,409 INFO [MetadataSegmentView-Cache--0] org.apache.druid.sql.calcite.schema.MetadataSegmentView - polling published segments from coordinator                                                                                 │
│ 2022-09-26T20:34:23,709 DEBUG [qtp1747862622-123] org.apache.druid.jetty.RequestLog - 10.3.114.17 GET //10.3.116.255:8082/status/health HTTP/1.1 200                                                                                              │
│ 2022-09-26T20:34:24,771 DEBUG [qtp1747862622-131] org.apache.druid.jetty.RequestLog - 10.3.114.17 GET //10.3.116.255:8082/status/health HTTP/1.1 200                                                                                              │
│ 2022-09-26T20:34:30,409 DEBUG [qtp1747862622-159] org.apache.druid.jetty.RequestLog - 10.3.124.123 GET //10.3.116.255:8082/druid/listen/v1/lookups HTTP/1.1 200                                                                                   │
│ 2022-09-26T20:34:30,440 WARN [qtp1747862622-152] org.apache.druid.query.lookup.LookupUtils - Lookup [user_to_city] could not be serialized properly. Please check its configuration. Error: Cannot construct instance of `org.apache.druid.query │
│ .lookup.namespace.JdbcExtractionNamespace`, problem: java.lang.ClassNotFoundException: org.postgresql.Driver                                                                                                                                      │
│  at [Source: (byte[])":)                                                                                                                                                                                                                          │
│ ��versionAv1�lookupExtractorFactory��typeNcachedNamespace�extractionNamespace�BCjdbc�namespaceLuser_to_city�connectorConfig��connectURI�jdbc:<postgresql://druid-test.cluster-abcde4a12345.us-east-1.rds.amazonaws.com:5432/lookups>��userJtest │
│ admin�password1234��tableLuser_to_city�keyColumnFuser_id�valueColumnDemail�pollPeriodBP1D�tsColumnAts��firstCacheTimeout��injective#��"; line: -1, column: 366] (through reference chain: org.apache.druid.query.lookup.Looku │
│ pExtractorFactoryContainer["lookupExtractorFactory"]-&gt;org.apache.druid.query.lookup.NamespaceLookupExtractorFactory["extractionNamespace"])                                                                                                       │
│ 2022-09-26T20:34:30,440 DEBUG [qtp1747862622-152] org.apache.druid.jetty.RequestLog - 10.3.124.123 POST //10.3.116.255:8082/druid/listen/v1/lookups/updates HTTP/1.1 202                                                                          │
│ 2022-09-26T20:34:33,709 DEBUG [qtp1747862622-140] org.apache.druid.jetty.RequestLog - 10.3.114.17 GET //10.3.116.255:8082/status/health HTTP/1.1 200                                                                                              │```

I have `postgresql-metadata-storage` in extensions
```bash-5.1$ cd postgresql-metadata-storage/
bash-5.1$ ls
checker-qual-3.5.0.jar                  postgresql-42.4.1.jar                   postgresql-metadata-storage-24.0.0.jar
bash-5.1$ ls -l
total 1256
-rw-r--r--    1 druid    druid       214381 Sep 10 05:43 checker-qual-3.5.0.jar
-rw-r--r--    1 druid    druid      1045138 Sep 10 05:43 postgresql-42.4.1.jar
-rw-r--r--    1 druid    druid        19699 Sep 10 05:55 postgresql-metadata-storage-24.0.0.jar
bash-5.1$```

might need to copy `postgresql-42.4.1.jar` to `druid/lib` ?

If the data is in deep storage, it should load as per the rules.  Do you see the load queues taking on the segments or do you see in the logging if the older segments are being loaded?

<@U03TAQRMJPQ> I am not sure how to check, i am running query to validate this where i am still not able to older date data after extending retention.

Check your coordinator log to see if it is assigning segments to historicals.  Depending on when you changed the retention settings, it can take time to load the older segments.  You can also check the segments tab to find the earliest timestamp of the loaded segments

Once it's marked unused (by drop rules) it will not _automatically_ be marked used even if you change the rules

interesting, so the metadata still will have it marked as unused?

yeah, automated marking only goes one way

once it's been marked unused automatically, you can mark it used manually, as long as you haven't done a permanent deletion yet through kill task (<https://druid.apache.org/docs/latest/data-management/delete.html#permanently-kill-task>) or autokill

IIRC there's an action in the web console to re-mark a certain time interval as used

there is also an API for it of course; you can do it by interval or for an entire datasource

Thanks, these are very helpful inputs. i couldn't find any option in UI where i could mark it as used manually, segments are still available in s3 location.  also, to confirm i don't see segments in the date range which i extended, so i believe it has to be marked manually which i am trying to figure out an option in UI or how to submit through an Api. Any suggestions please. cc <@U0344FW86DD> <@U03TAQRMJPQ>

In the datasources view you can mark segments as used

Also be aware that you are on a fairly old Druid version (notice how my screenshot is blueish and yours is greenish), the UI evolved a ton so it is possible some options that people suggest via screenshots (like I did) might not be in the same place for you (or might not exist)

great, thanks for recommending, i will propose to move to latest internally. i was able to set interval as 2022-08-25T00:00:00.00/2022-09-13T00:00:00.00.  However, i see only 1 day data got loaded which was of 12th since more than 30 mins.  in coordinator node, i see an error, not sure if that has caused it not to reload-
```2022-09-27T15:27:54,368 WARN [ServerInventoryView-0] org.apache.druid.curator.inventory.CuratorInventoryManager - Exception while getting data for node /test-druid-base/druid/segments/stats-druid-historical-1.test-druid-historical.test-uat.svc.cluster.local:8083/test-druid-historical-1.test-druid-historical.test-uat.svc.cluster.local:8083_historical_tier1_2022-09-27T15:24:57.586Z_142b7f8576e44979a4c787f0f6472dbe230
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /test-druid-base/druid/segments/test-druid-historical-1.test-druid-historical.test-uat.svc.cluster.local:8083/test-druid-historical-1.test-druid-historical.test-uat.svc.cluster.local:8083_historical_tier```

in coordinator logs:
```2022-09-27T21:11:04,730 WARN [KinesisSupervisor-test-Reporting-0] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Exception while getting current/latest sequences                                             │
│ org.apache.druid.indexing.seekablestream.common.StreamException: com.amazonaws.AbortedException:                                                                                                                                                  │
│     at org.apache.druid.indexing.kinesis.KinesisRecordSupplier.wrapExceptions(KinesisRecordSupplier.java:120) ~[?:?]                                                                                                                              │
│     at org.apache.druid.indexing.kinesis.KinesisRecordSupplier.getPartitionTimeLag(KinesisRecordSupplier.java:991) ~[?:?]                                                                                                                         │
│     at org.apache.druid.indexing.kinesis.KinesisRecordSupplier.getPartitionsTimeLag(KinesisRecordSupplier.java:825) ~[?:?]                                                                                                                        │
│     at org.apache.druid.indexing.kinesis.supervisor.KinesisSupervisor.updatePartitionLagFromStream(KinesisSupervisor.java:340) ~[?:?]                                                                                                             │
│     at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.updateCurrentAndLatestOffsets(SeekableStreamSupervisor.java:3590) ~[druid-indexing-service-24.0.0.jar:24.0.0]                                                 │
│     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_275]                                                                                                                                                     │
│     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_275]                                                                                                                                                            │
│     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_275]                                                                                                       │
│     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_275]                                                                                                              │
│     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_275]                                                                                                                                             │
│     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_275]                                                                                                                                             │
│     at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]                                                                                                                                                                                       │
│ Caused by: com.amazonaws.AbortedException:                                                                                                                                                                                                        │
│     at com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:61) ~[aws-java-sdk-core-1.12.264.jar:?]                                                                                                                │
│     at com.amazonaws.internal.SdkFilterInputStream.markSupported(SdkFilterInputStream.java:125) ~[aws-java-sdk-core-1.12.264.jar:?]                                                                                                               │
│     at com.amazonaws.http.RepeatableInputStreamRequestEntity.isRepeatable(RepeatableInputStreamRequestEntity.java:139) ~[aws-java-sdk-core-1.12.264.jar:?]                                                                                        │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.shouldRetry(AmazonHttpClient.java:1640) ~[aws-java-sdk-core-1.12.264.jar:?]                                                                                                            │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1223) ~[aws-java-sdk-core-1.12.264.jar:?]                                                                                               │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165) ~[aws-java-sdk-core-1.12.264.jar:?]                                                                                                          │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[aws-java-sdk-core-1.12.264.jar:?]                                                                                                               │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[aws-java-sdk-core-1.12.264.jar:?]                                                                                                        │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[aws-java-sdk-core-1.12.264.jar:?]                                                                                                                 │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.12.264.jar:?]                                                                                                              │
│     at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[aws-java-sdk-core-1.12.264.jar:?]                                                                                                     │
│     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[aws-java-sdk-core-1.12.264.jar:?]                                                                                                                                 │
│     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[aws-java-sdk-core-1.12.264.jar:?]                                                                                                                                 │
│     at com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2980) ~[?:?]                                                                                                                                          │
│     at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2947) ~[?:?]                                                                                                                                            │
│     at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2936) ~[?:?]                                                                                                                                            │
│     at com.amazonaws.services.kinesis.AmazonKinesisClient.executeGetShardIterator(AmazonKinesisClient.java:1436) ~[?:?]                                                                                                                           │
│     at com.amazonaws.services.kinesis.AmazonKinesisClient.getShardIterator(AmazonKinesisClient.java:1405) ~[?:?]                                                                                                                                  │
│     at com.amazonaws.services.kinesis.AmazonKinesisClient.getShardIterator(AmazonKinesisClient.java:1453) ~[?:?]                                                                                                                                  │
│     at org.apache.druid.indexing.kinesis.KinesisRecordSupplier.getRecordsForLag(KinesisRecordSupplier.java:1024) ~[?:?]                                                                                                                           │
│     at org.apache.druid.indexing.kinesis.KinesisRecordSupplier.lambda$getPartitionTimeLag$18(KinesisRecordSupplier.java:1008) ~[?:?]                                                                                                              │
│     at org.apache.druid.indexing.kinesis.KinesisRecordSupplier.wrapExceptions(KinesisRecordSupplier.java:117) ~[?:?]                                                                                                                              │
│     ... 11 more                                                                                                    ```

is it possible your kinesis stream isn't receiving any data? sometimes, via kinesis api, that can cause timeouts while reading from it

That's unlikely, same kinesis stream is being read by another druid cluster just fine (0.23.0) and the stream receives over 6B records per day

Another messages I am seeing in the current task running are:
```Kinesis records are being processed slower than they are fetched. OrderedPartitionableRecord buffer full, storing iterator and retrying in [5000ms]```

not sure if that might be related to some of the below tuning conf for kinesis
```"httpTimeout": "PT10S",
"shutdownTimeout": "PT80S",
"recordBufferSize": 10000,
"recordBufferOfferTimeout": 5000,
"recordBufferFullWait": 5000,```

I think the issue is somehow to the fact that our k8s cluster lost connectivity somehow, well, not all the pods, but for example one coordinator did not have internet access whereas other had, similarly with some middleManagers .. All broker/router/historical did have connectivity tho, so it’s weird

The frequency of creating compaction tasks depends on coordinator indexing period (30 minutes by default). The actual number of tasks created would depend on the configured task slot ratio.

<https://druid.apache.org/docs/latest/configuration/index.html#coordinator-operation|https://druid.apache.org/docs/latest/configuration/index.html#coordinator-operation>

<https://druid.apache.org/docs/latest/operations/api-reference.html#post-4>
Use this API to set task slots for compaction

Another common trap which I can speak from experience is not setting the `Skip offset from latest` appropriately

<@U041J08RMFB>, Could you please share how many free task slots you have and the value being set for maxNumConcurrentSubTasks for the compact tasks?

maxNumConcurrentSubTasks is not set  Should we specify this for compaction spec also.  And what should be an ideal value?
And how do I get number of free task slots.. We have 1 spare worker for now.

Its default value is 1. Setting it too high a value can also mean that fewer compactions run simultaneously

Actually, this produces the same error:
```SELECT *
FROM "druid"."foobar"
WHERE false
ORDER BY "__time" DESC
LIMIT 10;```

Is this a bug? I mean, an unfulfillable WHERE condition should result in an empty result, not in an exception.

<https://github.com/apache/druid/issues/13152>

when you "explain" this SQL query in the console, what do you get?

<@U031A9D19NU> do you still need it (I saw you commented on the ticket)?

I am not aware of any druid snowflake connectors.

I haven't tried this. But in principle you should be able to use JDBC. I think the content of the EXTERN clause should look like the sql input source. e.g.:
```        "type": "sql",
        "database": {
            "type": "mysql",
            "connectorConfig": {
                "connectURI": "jdbc:<mysql://host>:port/schema",
                "user": "user",
                "password": "password"
            }
        },
        "sqls": ["SELECT * FROM table1 WHERE timestamp BETWEEN '2013-01-01 00:00:00' AND '2013-01-01 11:59:59'", "SELECT * FROM table2 WHERE timestamp BETWEEN '2013-01-01 00:00:00' AND '2013-01-01 11:59:59'"]```
and you will need to have the Snowflake JDBC driver in the classpath....
but upon <https://druid.apache.org/docs/latest/ingestion/native-batch-input-sources.html#sql-input-source|further reading> it does seem like you will also need an extension that enables the `connectorConfig` for a new `"database": { "type": "snowflake"....}` , so I don't think this is available out of the box.

note the SQL-based ingest supports all the same input sources as native; they are listed here <https://druid.apache.org/docs/latest/ingestion/native-batch-input-sources.html>. you can provide them all in <https://druid.apache.org/docs/latest/multi-stage-query/concepts.html#read-external-data-with-extern|EXTERN>

for reading larger amount of data from snowflake i think it'd be best to export first, perhaps using data unload <https://docs.snowflake.com/en/user-guide/data-unload-overview.html>

then we can read the exported data using input source like `s3` or `gcs` depending on your cloud

it's nice since it's automatically done in parallel

wow i haven't seen that UI in a long time!!

anyway, i bet you need to update your `druid.server.maxSize` and `druid.segmentCache.locations`

in newer versions, you only need to update `druid.segmentCache.locations`; other one is computed automatically

Thanks team, will update druid.server.maxSize and druid.segmentCache.locations. We will be planning to upgrade to latest UI, we have already moved to newer one, since i wasn't able to see volume in that UI, so referred here.

I used this for reference once :
<https://community.cloudera.com/t5/Community-Articles/Auth-to-local-Rules-Syntax/ta-p/245316>
maybe it will help

Active real-time ingestion holds a lock for time intervals that are being ingested. You're getting overlapping locks, because your load from S3 goes through 2022-09-29 which you are currently loading. If you limit the interval for your backfill up to 2022-09-28T00:00:00, you should be able to do the load.

ah I see, makes sense because realtime ingestion should already have covered `2022-09-29T00:00:00` if the realtime ingestion started on the 27th for example

I was also wondering two things:
• I started realtime ingestion, say on `2022-09-27T16:43:33` and I start S3 index parallel to backfill data, essentially from `2022-09-01T00:00:00` to `2022-09-28T00:00:00`, would there be any duplicate data? or there would be segment replaced by the index-parallel?
• Second question I have is about speeding up index-parallel, what settings should I tweak so that indexing 1 month worth of data from S3 takes a day or two? Taking into account that there is around 9B json records per day.

1. If you did not turn on append mode, then the last one will win and overwrite the existing time bucketed data.
2. If it’s dynamic partition, you can just spawn more subtasks, but later you need a compaction job to optimize the file size so that they are not too small

in case you can see something weird, the spec looks like:
```{
  "type": "index_parallel",
  "id": "index_parallel_analytics_heodejnl_2022-09-29T15:07:01.433Z",
  "groupId": "index_parallel_analytics_heodejnl_2022-09-29T15:07:01.433Z",
  "resource": {
    "availabilityGroup": "index_parallel_analytics_heodejnl_2022-09-29T15:07:01.433Z",
    "requiredCapacity": 1
  },
  "spec": {
    "dataSchema": {
      "dataSource": "analytics",
      "timestampSpec": {
        "column": "ts",
        "format": "millis",
        "missingValue": null
      },
      "dimensionsSpec": {
        "dimensions": [
          {
            "type": "string",
            "name": "user_id",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "ip",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "proxy",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "status_code",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "long",
            "name": "payload_size",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": false
          },
          {
            "type": "long",
            "name": "isAuthed",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": false
          }
        ],
        "dimensionExclusions": [
          "__time",
          "count",
          "ts"
        ],
        "includeAllDimensions": false
      },
      "metricsSpec": [
        {
          "type": "count",
          "name": "count"
        },
        {
          "type": "longSum",
          "name": "sum_payload_size",
          "fieldName": "payload_size"
        }
      ],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "DAY",
        "queryGranularity": "MINUTE",
        "rollup": true,
        "intervals": [
          "2022-09-01T00:00:00.000Z/2022-09-28T00:00:00.000Z"
        ]
      },
      "transformSpec": {
        "filter": null,
        "transforms": [
          {
            "type": "expression",
            "name": "isAuthed",
            "expression": "if(\"auth_id\" &gt; 0, 1, 0)"
          }
        ]
      }
    },
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "s3",
        "uris": null,
        "prefixes": [
          "<s3://analytics-cold-storage-v2/2022/09/>"
        ],
        "objects": null,
        "properties": null,
        "proxyConfig": null,
        "endpointConfig": null,
        "clientConfig": null
      },
      "inputFormat": {
        "type": "json",
        "flattenSpec": {
          "useFieldDiscovery": true,
          "fields": []
        },
        "featureSpec": {},
        "keepNullColumns": true
      },
      "appendToExisting": false,
      "dropExisting": false
    },
    "tuningConfig": {
      "type": "index_parallel",
      "maxRowsPerSegment": 5000000,
      "appendableIndexSpec": {
        "type": "onheap",
        "preserveExistingMetrics": false
      },
      "maxRowsInMemory": 1000000,
      "maxBytesInMemory": 0,
      "skipBytesInMemoryOverheadCheck": false,
      "maxTotalRows": null,
      "numShards": null,
      "splitHintSpec": null,
      "partitionsSpec": {
        "type": "dynamic",
        "maxRowsPerSegment": 5000000,
        "maxTotalRows": null
      },
      "indexSpec": {
        "bitmap": {
          "type": "roaring",
          "compressRunOnSerialization": true
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs",
        "segmentLoader": null
      },
      "indexSpecForIntermediatePersists": {
        "bitmap": {
          "type": "roaring",
          "compressRunOnSerialization": true
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs",
        "segmentLoader": null
      },
      "maxPendingPersists": 0,
      "forceGuaranteedRollup": false,
      "reportParseExceptions": false,
      "pushTimeout": 0,
      "segmentWriteOutMediumFactory": null,
      "maxNumConcurrentSubTasks": 5,
      "maxRetry": 3,
      "taskStatusCheckPeriodMs": 1000,
      "chatHandlerTimeout": "PT10S",
      "chatHandlerNumRetries": 5,
      "maxNumSegmentsToMerge": 100,
      "totalNumMergeTasks": 10,
      "logParseExceptions": true,
      "maxParseExceptions": 2147483647,
      "maxSavedParseExceptions": 0,
      "maxColumnsToMerge": -1,
      "awaitSegmentAvailabilityTimeoutMillis": 0,
      "maxAllowedLockCount": -1,
      "partitionDimensions": []
    }
  },
  "context": {
    "forceTimeChunkLock": true,
    "useLineageBasedSegmentAllocation": true
  },
  "dataSource": "analytics"
}```

``` "appendToExisting": false,
 "dropExisting": false```
ok, then the last one will win and overwrite the older data.

```"maxNumConcurrentSubTasks": 5```
This is too small, jack it up to 100 or 200

but then each of your zip file will be so small.
Therefore, you need a compaction job.
But data will be available very quickly.

would not that eat too much memory if there is  kinesis ingestion also happening?

Yes, you need to do that memory calculation very carefully.

we separated our Kafka MM vs batch MM, they don’t bulldoze each other.

given that this will probably be only one shot to backfill, I do not think I need to split them

just to double check, what parameters I need to check to calculate the memory? and also, is it possible to increase `maxNumConcurrentSubTasks` while already running for couple of hours?

you have to try ingesting again, you cannot modify currently running ingestion.

This is the most basic settings that’s relevant to you.
```# Indexer (worker/peon) Java options
# Make sure -Xmx is large enough for DRUIDMM_INDEXER_TASK_DEFAULTROWFLUSHBOUNDARY but not so big that we lose precious pods capacity.
# With our largest dataset, the -Xmx cannot be smaller than 19g.
DRUIDMM_INDEXER_RUNNER_JAVAOPTS: -server -Xms128m -Xmx19g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+ExitOnOutOfMemoryError -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Daws.region=us-west-2 --add-exports=jdk.management/com.sun.management.internal=ALL-UNNAMED --add-exports=java.base/jdk.internal.perf=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED

DRUIDMM_INDEXER_TASK_DEFAULTROWFLUSHBOUNDARY: "100000" # For ballpark comparison, each LARGEST_TABLE's row is about 2kb. So about 200mb is buffered inside peon indexer.

DRUIDMM_DAEMON_MAX_HEAP_SIZE: 36g    # -20% of max memory. E.g. 45GB - (20% x 45GB) = 36GB, each thread will get 18GB heap size

# processing
DRUIDMM_WORKER_CAPACITY: "2"
DRUIDMM_PROCESSING_BUFFER_SIZEBYTES: "100000000"
DRUIDMM_PROCESSING_NUMMERGEBUFFERS: "2"
DRUIDMM_PROCESSING_NUMTHREADS: "2"
DRUIDMM_PROCESSING_TMPDIR: var/ebs/middlemanager/processing
DRUIDMM_PROCESSING_STORAGE_TYPE: local

# Kubernetes
resources:
limits:
  # There are 2 workers. See: DRUIDMM_WORKER_CAPACITY
  cpu: 2
  # Each worker can consume 19GB RAM: 2 * 19 = 38.
  # See: -Xmx setting inside DRUIDMM_INDEXER_RUNNER_JAVAOPTS.
  # Now we need to add a little bit more RAM because the -Xmx should only be 80% of total RAM.
  # So the math is: (2 * 19) / 4 * 5 = 47.5
  memory: 47Gi
requests:
  cpu: 2
  memory: 47Gi```


all right, I checked `/druid/indexer/v1/workers` to find out capacity and there are few slots available really, from 12 MM, most of them already with up to 10 slots taken out of 12 slots available each

my MM looks smaller I think
```middleManager: {
    replicaCount: 12,
    config: {
        DRUID_XMX: '5g',
        DRUID_XMS: '5g',
        DRUID_MAXDIRECTMEMORYSIZE: '10g',
        druid_node_type: 'middleManager',
        druid_indexer_fork_property_druid_processing_buffer_sizeBytes: '300000000',
        druid_worker_capacity: '12',
        druid_service: 'druid/middleManager',
        druid_indexer_task_baseDir: '/opt/druid/var/tmp',
        druid_indexer_fork_property_druid_processing_numMergeBuffers: '2',
        druid_indexer_fork_property_druid_processing_numThreads: '2',
        druid_indexer_task_restoreTasksOnRestart: 'true',
        druid_indexer_task_gracefulShutdownTimeout: 'PT120S',
        druid_indexer_runner_javaOpts: '-server -Xmx5g -XX\:+IgnoreUnrecognizedVMOptions -XX\:MaxDirectMemorySize\=10g -Duser.timezone\=UTC -XX\:+PrintGC -XX\:+PrintGCDateStamps -XX\:+ExitOnOutOfMemoryError -XX\:+HeapDumpOnOutOfMemoryError -XX\:HeapDumpPath\=/mnt/tmp/druid-peon.hprof -Dfile.encoding\=UTF-8 -Djava.util.logging.manager\=org.apache.logging.log4j.jul.LogManager',
        druid_indexer_fork_property_druid_server_http_numThreads: '50'
    },
    persistence: {
        size: '50Gi'
    }
},```

wonder if I could get better S3 performance by increasing DRUID_XMX to 19g too and DRUID_MAXDIRECTMEMORYSIZE to 36g.

Wondering tho where did you get 47Gi for memory for k8s limits, in my case I currently have
```requests: {
    memory: '5g'
},
limits: {
    memory: '5g'
}```
you did `(2 * 19) / 4 * 5 = 47.5`  but not sure where the `4` and `5` came from

That comes from experience, JVM xmx setting is a soft limit, in reality JVM can use even more than what the xmx settings, so I give it 20% extra padding

I am not sure what you mean by this. Is the Zk being restarted gracefully ?

yes, 1 at a time and the ZK is also spread between multiple AZ

If there is always quorum, then coordinator should not be affected.
If that is not the case, coord logs would help us narrow down the isssue.

Historical also could not reconnect if Coordinator was down for a while.

other daemons reconnected properly, it’s just historical that failed to retry.

I have a dumb question that is possibly related to Coordinator not being good at detecting ZK restart.
Folks who deploy Druid in Kubernetes usually uses Bitnami Zookeeper chart.
In that chart, they teach users to use the Service object load balancer.

However, <https://support.imply.io/hc/en-us/articles/360015465773-Zookeeper-Best-Practices-for-Imply|Imply website> recommends users to connect to ZK pods directly.

Is it possible that my Coordinator is not HA because I use the Zookeeper’s Service object load balancer?

I saw a recent Curator patch being merged related to this issue: <https://github.com/apache/curator/pull/433|apache/curator#433>.

Maybe it is time to upgrade Curator?

I could see why Druid community have no love towards Curator.

oh, it is already the latest. <https://github.com/apache/druid/blob/acafd0d1e03a2f4827f1e300843dc178f95fdd14/pom.xml#L78>

There are quite a few fixes planned for curator 5.4.0 that I am excited to try out once it's released. 538 as you mentioned as well as <https://issues.apache.org/jira/plugins/servlet/mobile#issue/CURATOR-649|https://issues.apache.org/jira/plugins/servlet/mobile#issue/CURATOR-649> which look like they were both fixed by the same commit.

Does Maven have a concept of vendoring the source code like in Go?

Curator codebase looked small. If vendored inside Druid, Druid devs can fix Curator NPE immediately without waiting on a new release.

Just a small thought I have.

You could build Druid yourself specifying the `SNAPSHOT` artifact from Curator.

Usually, most projects, as well as Druid (I think), tend to avoid forking the code. Mostly because original code and forked code can diverge over time. So it carries some maintenance overhead. Then we will run into problems that non-standard techniques generally run into. E.g. if there is a CVE in the curator library, we won't be notified because we have our own fork instead of using a released version.

It seems you are using hash partitioning. I would advise you to use to range partitioning as it gives a better distribution of data amongst segments and also prevents such OOM failures.

Make sure that your indexer/middle manager sets sufficient heap size using `druid.indexer.runner.javaOptsArray` (<https://druid.apache.org/docs/latest/configuration/index.html#middlemanager-configuration>)

Also, since the failure is in the merge phase, you could try reducing `maxNumSegmentsToMerge` to a small value (<https://druid.apache.org/docs/latest/ingestion/native-batch.html#tuningconfig>)

I tried but had no luck with the `maxNumSegmentsToMerge`, I’m currently running with range partitioning to see it it handles the indexes better

noup, no luck with any of the options. I guess the only chance is to make the heap size bigger, but before trying that I’ll run the task with dynamic partitioning with best effort rollup, and later have a second job compact it with a better partitioning with full rollup

A couple comments:

1) I'm curious where exactly the task is running OOM. If you manage to capture a heap dump, the heaviest objects in the heap dump generally tell us something about what to do (to do it: use `-XX:+HeapDumpOnOutOfMemoryError` in your `druid.indexer.runner.javaOpts` or `druid.indexer.runner.javaOptsArray`)

2) As an alternative approach, in the last release (24.0) we introduced <https://druid.apache.org/docs/latest/multi-stage-query/index.html|SQL-based ingest and multi-stage query tasks> which is totally re-engineered and much better at memory management. This can be used for manual compaction using REPLACE: <https://druid.apache.org/docs/latest/multi-stage-query/concepts.html#overwrite-data-with-replace>. There are some examples: <https://druid.apache.org/docs/latest/multi-stage-query/examples.html>. If you use it, would love your feedback on this new feature! Note: we haven't yet built a way to use this for autocompaction, but we plan to in the future.

uhh, I’m really interested in that second one. I just set up MSQE for a different part of the project so it is relatively lightweight to test

it worked like a charm. Ran muuch faster (8 min vs several hours) and worked without errors. Thanks <@U0344FW86DD>

we'll want to make autocompaction use this as soon as we can :slightly_smiling_face:

I guess I’ll bump druid_emitter_logging_logLevel and see if it shows anything..

Check if your metricsJson has the task count metrics

<https://github.com/apache/druid/blame/master/extensions-contrib/prometheus-emitter/src/main/resources/defaultMetrics.json>

Right, I’m using a default configuration here, so that’s probably the reason

I guess “For most use-cases, the default mapping is sufficient.” from the documentation is a… rather optimistic assertion :smile:

Ideal CPU allocation will depend a lot on your requirements and your workload. A good way to understand utilization of resources in your current setup is to enable metrics. Here are a couple of docs on the subject:
<https://druid.apache.org/docs/latest/operations/metrics.html>
<https://druid.apache.org/docs/latest/configuration/index.html#enabling-metrics>

Since you're running in k8s, another good thing to look at would be the CPU usage in each pod

If it's low, you can get away with smaller CPU allocation

Historical, Broker, Router CPU needs depend mostly on your query workload

Coordinator, Overlord CPU needs depend mostly on the total number of segments in your cluster. You might be fine with 1 each instead of 2 each. Hard to say, so I'd look at the actual pod CPU usage

Not sure that this is what you are running into : <https://github.com/apache/druid/issues/4018>
but in that reported issue the problem seemed to originate from some other (non-Druid) service using the same port as the historical (8083)  (it was Debezium in that case).  So that when the broker tried to communicate with the historical it was actually talking to Debezium. Are you running anything else? Could you be having a port conflict?

We checked that Sergio and no other services are running other then druid on 8083

Hi Satish. Are you seeing bottlenecking on the brokers?

We see all the resources available On brokers . We had issues with ram and cpus we added more brokers and we have 4 brokers behind LB and all looks healthy

Thanks for that.

Queries are the other thing that come to mind. Have any queries changed recently? <https://support.imply.io/hc/en-us/articles/360033747953-Profiling-Druid-queries-using-flame-graphs|Flame graphs> might be worth a look.

You mentioned it happens when "running concurrent queries". Could you elaborate? Does it happen as soon as you run two concurrent queries? or what is the pattern?

We running around 4000 concurrent queries, we r seeing this after 3000 plus queries

Okay. I think a review of threads and connection parameters may help. Some relevant parameters and rules from <https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html|the basic cluster tuning guide>:
```For Historicals, druid.server.http.numThreads should be set to a value slightly higher than the sum of druid.broker.http.numConnections across all the Brokers in the cluster.```
On the Brokers:
```druid.server.http.numThreads on the Broker should be set to a value slightly higher than druid.broker.http.numConnections on the same Broker.```
If you are querying from real-time ingestion on Middle Manager Tasks:
```For Tasks, druid.server.http.numThreads should be set to a value slightly higher than the sum of druid.broker.http.numConnections across all the Brokers in the cluster.```
If none of these are an issue perhaps you are having trouble with some other resource contention on CPU or Memory. How is the utilization level on the Brokers, Historicals and MMs when this situation occurs?

<@U03149ALHN3> I think it is more a question of how big you want target the segments to be. It is not an exact row count cutoff as it is in dynamic partitioning.
It will sort the partitioning dimension values and calculate a distribution of rows across the values for each time chunk, then use the target rows per segment to find range boundaries (dimension values) that create segments that are closest to the target row count.

If I mistakenly put the number too high, I guess the only downside is that the segment file size is too big?

Yeah, I guess. You'll also want to check the resulting segments, if your partitioning dimension values have significant skew you will see larger segments when they end up with those values. One of the advantages of multi-dim is that you can add another dimension to alleviate the skew since it will find combinations of values as the boundaries and be able to better split rows across them.

Also, the processing of a segment cannot be parallelized. If you have really big segments, you won't be benefitting from parallel processing.

You can set it to the average number of rows in the chunk that is currently dynamically partitioned.
target rows  = (total number of rows) / (total number of segments)

with that, you should get the same number of segments but evenly balanced and better sized.

That seems easy to follow, thanks <@U031A9D19NU>

Can you hard refresh the druid docs once. It started working for me post that

all right just seeing <https://druid.apache.org/docs/latest/ingestion/tasks.html#lock-priority> - would it be harmless if I see a higher priority for S3 indexing than realtime kinesis indexing? Basically I want to prioritize backfill via S3

`[2022-09-01T00:00:00.000Z/2022-10-01T00:00:00.000Z] ` Looks like a hire priority task took the lock.

I see, yeah there is a kinesis ingestion going on in the same data source

The recommendation is to use it. Auto compaction on all data sources using real-time ingestion with multiple tasks is a good/best practice. Each parallel task generates segments independently so using auto-compaction to merge them is an important technique in <https://druid.apache.org/docs/latest/operations/segment-optimization.html|segment size optimization>.

And here's a consideration from the <https://druid.apache.org/docs/latest/data-management/automatic-compaction.html|docs>:

&gt; To set `skipOffsetFromLatest`, consider how frequently you expect the stream to receive late arriving data. If your stream only occasionally receives late arriving data, the auto-compaction system robustly compacts your data even though data is ingested outside the `skipOffsetFromLatest` window. For most realtime streaming ingestion use cases, it is reasonable to set `skipOffsetFromLatest` to a few hours or a day.

I remembered seeing a setting where if you turn it on, the JSON inside SQL request logs became the native JSON.
What was that setting again?

oh, it doesn’t change the request logs? Bummer… it would be super helpful for debugging if it does change the request log

can you share an example of what log you are referring to? I am not aware of a place where we log calcite logical plan

When SQL request is logged, I believe it just looks like this:
```{"query": "SELECT * from X"}```
It would be nice if it has the JSON explain as well, like this:
```{"query": "SELECT * from X", "json": {TopN...}}```


Since all queries get converted to native JSON queries anyway, you should be able to see them if request logging is turned on (on Broker logs).