Apache Druid

```EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
	at java.lang.Thread.run(Thread.java:748)```


Hi <@U04L6G64AM6>, middle managers hold the data in row form until they publish segments, either locally with intermediate persists or publish them to Deep Storage. I have heard of cases where more intermediate persists help performance because they are stored in segment form which are faster to query the the row form (the historicals only process in segment form). So you might want to try adjusting your streaming ingestion such that it does intermediate persists more often. The settings are `maxRowsInMemory` or `maxBytesInMemory` and time based  with `intermediatePersistPeriod`.
Hope this helps, let us know how it goes.

thanks <@U030K4UM3H7>, I will try these suggestions

<@U04JR9YAPDF> Please check the order of rules applied, if you have loadbyforever alongwith loadbyinterval, then loadforever should take precedency.

IMG_2262.jpg

<@U03R5H3B750> the loadforever is applied after loadbyinterval

<@U04JR9YAPDF> I believe you should not have LOADFOREVER rule set as you want only to load by interval.

<https://druid.apache.org/docs/latest/operations/rule-configuration.html#rule-structure>

The segements that satisfied rule1 will also satisfy rule 4 and hence will be loaded

I added load forever because apart from the specific intervals everything has to go to cold tier

Also these rules were working for multiple years
After we switch from tranquility to native kinesis ingestion (supervisor) facing this issue

Am I missing anything specific to supervisor ?

supervisor should not have anything to with load rules

may be coordinator log will have some more details regarding the load rules

The coordinator get rules API returns the latest rules, but it doesnt gets triggered

<@U03R5H3B750> here are coordinater logs. Does Throttling replication cause this issue?

*For the dates, which are supposed to move to Hot tier but are still in Cold tier, We're getting Throttling replication as below:*
```2023-01-20T12:48:51,439 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Throttling replication for segment [clickstream_2021-10-17T00:00:00.000Z_2021-10-18T00:00:00.000Z_2022-01-26T09:13:58.542Z_4] in tier [hot]
2023-01-20T12:48:51,439 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Throttling replication for segment [clickstream_2021-10-17T00:00:00.000Z_2021-10-18T00:00:00.000Z_2022-01-26T09:13:58.342Z_5] in tier [hot]```
For the dates, which are loaded in Hot tier as expectedly, we have't seen these Throtting replication they're getting dropped to Hot tier as expected:
```2023-01-20T01:33:29,131 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Dropping segment [clickstream_2021-10-18T00:00:00.000Z_2021-10-19T00:00:00.000Z_2022-01-27T09:15:12.255Z_2] on server [historical-hot-server-1] in tier [hot]
2023-01-20T04:18:34,666 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Dropping segment [clickstream_2021-10-18T00:00:00.000Z_2021-10-19T00:00:00.000Z_2022-01-27T09:15:12.255Z_5] on server [historical-hot-server-7] in tier [hot]```


Does changing `replicationThrottleLimit` help to resolve this?
We're using  `replicationThrottleLimit=10` .

<https://support.imply.io/hc/en-us/articles/360041635193>  please try this you can create role and add all permissions to it  and attach role to the required group with creating group mapping.

all segments are being shown as unvailable

Have you see if the historicals have enough storage space? in the capture they only show space configured but not real available space in the disk

<@U030MBK46BD> I am using kafka connect and adding data. I am not adding retention rules separately

<@U03DSHS8Z3N> what <@U030MBK46BD> try to say is if you can share the retention rules applied over the datasource

I am trying to understand why the 119 segments are unavailable. One reason might be they are not being loaded due to the retention rules.

You might want to have a look in the `coordinator` logs to understand how they are being handled

I did add retention rule. I am able to see some data now. I

it is possible tha oyur default retention rule is “drop forever”?

My doubt over here is why is it something need to asked explicitly to retent data. Isn't this expected by default as we need the db for analysis

image.png

the `_default` rule of a default cluster is `loadForever`- if this changed, somebody changed it explicitly.
Default rule in a default cluster:

out of the box it should be `loadForever` 2 replicas in the `_default_tier`

if you use the latest druid, the startup scripts tune everything for you based on available hardware

so I should not set environment variables and just trust the startup scripts will tune everything ?

<https://github.com/apache/druid/releases/tag/druid-25.0.0>

Druid was so complicated to tune that I didn't though that I should just remove the previously defined druid tuning environment variables.
Many thanks ! Will try that on Monday.

no problems now. Druid works like a charm.

Are you trying to flatten a column containing an array during ingestion? Could you post an example of the row/dataset?

Hi, I am facing a performance issue with *IN* clause in druid.  Here is my query and it time out after 1 min

select LOOKUP(fe.product_id,'dim_desc') as manf_desc ,fe.sales_month,fe.sales_year,fe.sales_quarter,sum(fe.dist_total_revenue) from fact_extrapolated_revenue fe where __time &gt;= TIME_PARSE ('2020-12-31T00:00:00.000Z') AND __time &lt; TIME_PARSE ('2022-11-30T00:00:00.000Z') and
fe.product_id IN ( _*&lt;1k product_ids&gt;*_)
GROUP BY LOOKUP(fe.product_id,'dim_desc') ,fe.sales_month,fe.sales_year,fe.sales_quarter

My understanding is that Druid is good at filtering even when list of items in IN clause is very large.

I recall that planning can sometimes take more time when there is a large IN filter involved. and <@U03NCEXDNJK> made some changes to optimize the same. there are a few context parameters that could be set to improve performance

Is the query timing out during planning? This was a known issue for IN clauses that contained ~10k value and there was a PR to help with this. (<https://github.com/apache/druid/pull/12357>)

Could you try to set the `inQueryThreshold` query to 0 to check if the query is faster? You can increase it to a higher appropriate value afterwards

hi <@U03NCEXDNJK> sure i will set this parameter in query context and try again.

Hi <@U03BJAMQ4H5> product_id is string but i am passing the product_ids without single quote. Example: select LOOKUP(fe.product_id,'dim_desc') as manf_desc ,fe.sales_month,fe.sales_year,fe.sales_quarter,sum(fe.dist_total_revenue) from fact_extrapolated_revenue fe where __time &gt;= TIME_PARSE ('2020-12-31T00:00:00.000Z') AND __time &lt; TIME_PARSE ('2022-11-30T00:00:00.000Z') and
fe.product_id IN _*( 1000015,1000112,1000140)*_ 
GROUP BY LOOKUP(fe.product_id,'dim_desc') ,fe.sales_month,fe.sales_year,fe.sales_quarter

the filter without quotes would be the bound filter (even for string fields). Putting the quotes changes it to a selector filter which will be faster. Please put the quotes and check

thank you <@U03BJAMQ4H5>. Putting the product_id in quotes really improved performance. Now the query is executing in 1.75 sec.

that's was the issue I had <https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1670808186666009?thread_ts=1670615072.343299&amp;cid=C0303FDCZEZ>

and maybe also <https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1674326908487929>
I think this can be fixed automatically by the planner, can't it?

Did that help/work?  A hacky workaround is to try replacing `myField in ("item1",..."item10001")` with `*MV_OVERLAP*(myField, ARRAY["item1",..."item 10001"])`.  Then calcite doesn't try to plan it, and sends it straight to druid, so you avoid the longer planning times.   Hopefully soon planning will be improved.

I don't think I was affect by long planning, because I had a special case for list of ints where my query interpolation just rendered then as a sequence of ints (without using the ? ), then when I cast to string (both sides, just to be sure), I got this insane improvment in the query time and huge decrease in cpu usage

Full Error
```2023-01-23T12:45:24,727 ERROR [main] org.apache.druid.emitter.prometheus.PrometheusEmitter - Unable to start prometheus HTTPServer
java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_275]
	at sun.nio.ch.Net.bind(Net.java:461) ~[?:1.8.0_275]
	at sun.nio.ch.Net.bind(Net.java:453) ~[?:1.8.0_275]
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222) ~[?:1.8.0_275]
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85) ~[?:1.8.0_275]
	at sun.net.httpserver.ServerImpl.&lt;init&gt;(ServerImpl.java:100) ~[?:1.8.0_275]
	at sun.net.httpserver.HttpServerImpl.&lt;init&gt;(HttpServerImpl.java:50) ~[?:1.8.0_275]
	at sun.net.httpserver.DefaultHttpServerProvider.createHttpServer(DefaultHttpServerProvider.java:35) ~[?:1.8.0_275]
	at com.sun.net.httpserver.HttpServer.create(HttpServer.java:130) ~[?:1.8.0_275]
	at io.prometheus.client.exporter.HTTPServer.&lt;init&gt;(HTTPServer.java:166) ~[simpleclient_httpserver-0.7.0.jar:?]
	at io.prometheus.client.exporter.HTTPServer.&lt;init&gt;(HTTPServer.java:180) ~[simpleclient_httpserver-0.7.0.jar:?]
	at io.prometheus.client.exporter.HTTPServer.&lt;init&gt;(HTTPServer.java:187) ~[simpleclient_httpserver-0.7.0.jar:?]
	at org.apache.druid.emitter.prometheus.PrometheusEmitter.start(PrometheusEmitter.java:75) [prometheus-emitter-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.emitter.service.ServiceEmitter.start(ServiceEmitter.java:61) [druid-core-0.23.0.jar:0.23.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_275]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) [druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) [druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:152) [druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:165) [druid-services-0.23.0.jar:0.23.0]
	at org.apache.druid.cli.CliPeon.run(CliPeon.java:306) [druid-services-0.23.0.jar:0.23.0]
	at org.apache.druid.cli.Main.main(Main.java:114) [druid-services-0.23.0.jar:0.23.0]```


Are you sending metrics to premetheus?   Has this been working? What value do you have set for druid.emitter.prometheus.port (<https://druid.apache.org/docs/latest/development/extensions-contrib/prometheus.html#configuration|See docs>)

Yes, I have added this property druid.emitter.prometheus.port=8078 in common.runtime.properties.
It is working, currently 3 ingestion task are running and for these 3 ingestion task automation are also running. Only two automation task are showing this exception while others are working.

If you are running multiple tasks in the same middle manager, each task is trying to start the prometheus emitter on the same port. This is why the recommendation is to use a prometheus gateway for the tasks. Check out <https://druid.apache.org/docs/latest/development/extensions-contrib/prometheus.html#override-properties-for-peon-tasks|the specifics here>.

BTW, the only reason I can think of that some of the tasks' emitters work and others do not, is because the first task to start on each MM will grab the port, other tasks that attempt it within the same MM will fail.

<@U04JDGL76QM> having the same issues on a fresh installation

Can someone provide a working version of k8s deploy for druid-operator (like the *tiny-cluster* example) with both `druid-kubernetes-extensions` and `druid-kubernetes-overlord-extensions` properly configured? (maybe <@U04NHCE780G>)

Tbh looking at the docs I don’t even understand if MM-less druid requires `druid-kubernetes-extensions` (like the extension names kinda suggest) loaded/running/no-zookeeper or if they’re incompatible.

Their documentation is far from simple and there are no examples for either. As an example of confusion, <https://druid.apache.org/docs/latest/development/extensions-contrib/k8s-jobs.html|Kubernetes extension doc> says you need
```druid.indexer.runner.type=httpRemote```
And <https://druid.apache.org/docs/latest/development/extensions-contrib/k8s-jobs.html|MM-less Druid> doc says you need
```druid.indexer.runner.type: k8s```
So… what it is? And can they work together?

(I’m seeing the same exceptions as the people on this thread btw)

<@U0322ET4ZGE> looks like could help on this one too.

for the k8s extension, put `druid.indexer.runner.type=httpRemote` in your common runtime properties

in your overlord put `druid.indexer.runner.type: k8s`

Hey <@U04NHCE780G>, thanks for your reply.

so all properties from the MM-less docs are for overlord only?
```    druid.indexer.runner.type=k8s
    druid.indexer.runner.namespace=druid
    druid.indexer.queue.maxSize=3
    druid.indexer.task.encapsulatedTask=true
    druid.processing.intermediaryData.storage.type=deepstore```
something like this? anything else?

yeah so the overlord k8s extension….should only have configs in the overlord.  The maxSize means you want a max of 3 concurrent tasks at a time?  That looks a bit small to be, otherwise looks good

yeah, I just want to see like a ‘tiny-cluster’ equivalent

sure no worries, i would bump that up to something a bit larger like 10, but the rest looks good to me

```    overlords:
      # Optionally specify for running coordinator as Deployment
      # kind: Deployment
      nodeType: overlord
      druid.port: 8088
      nodeConfigMountPath: "/opt/druid/conf/druid/cluster/master/overlord"
      replicas: 1
      runtime.properties: |
        druid.service=druid/overlord
        druid.indexer.runner.type=k8s
        druid.indexer.runner.namespace=druid
        druid.indexer.queue.maxSize=3
        druid.indexer.queue.startDelay=PT30S
        druid.indexer.task.encapsulatedTask=true
        druid.processing.intermediaryData.storage.type=deepstore
      extra.jvm.options: |-
        -Xmx512M
        -Xms512M```

```│ Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:                                                                                                                                                    ││                                                                                                                                                                                                                                                         ││ 1) Unknown provider [k8s] of Key[type=org.apache.druid.discovery.DruidNodeAnnouncer, annotation=[none]], known options [[curator]]                                                                                                                      ││   at org.apache.druid.guice.PolyBind.createChoiceWithDefault(PolyBind.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.curator.discovery.DiscoveryModule)     ││   while locating org.apache.druid.discovery.DruidNodeAnnouncer                                                                                                                                                                                          ││     for field at org.apache.druid.cli.ServerRunnable$DiscoverySideEffectsProvider.announcer(ServerRunnable.java:105)                                                                                                                                    ││   at org.apache.druid.cli.ServerRunnable.bindAnnouncer(ServerRunnable.java:93) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.cli.CliCoordinator$1)                   ││```

you should have something like loadList=[a,b,c] for common

then in your overlord loadList=[a,b,c,druid-kubernetes-overlord-extensions]

Yeah you’re right, I was reactivating the deployment with both extensions, but one was missing in the load list this time :face_palm:

ok, now I’m back at the old

```│ Exception in thread "main" java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:                                                                                                        ││                                                                                                                                                                                                                                                         ││ 1) Could not find a suitable constructor in org.apache.druid.indexing.common.config.TaskConfig. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.                      
││   at org.apache.druid.indexing.common.config.TaskConfig.class(TaskConfig.java:49)                                                                                                                                                                       ││   while locating org.apache.druid.indexing.common.config.TaskConfig                                                                                                                                                                                     ││     for the 3rd parameter of org.apache.druid.k8s.overlord.KubernetesTaskRunnerFactory.&lt;init&gt;(KubernetesTaskRunnerFactory.java:63)                                                                                                                      ││   at org.apache.druid.k8s.overlord.K8sOverlordModule.configure(K8sOverlordModule.java:63) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.k8s.overlord.K8sOverlordModule)                                               
││                                                                                                                                                                                                                                                         ││ 1 error                                                                                                                                                                                                                                                 ││     at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:88)                                                                                                                                                                           ││     at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:62)                                                                                                                                                                                  ││     at org.apache.druid.cli.Main.main(Main.java:112)```

hmm docs says it’s not required, the default would be a `-Xmx1g`

<https://druid.apache.org/docs/latest/development/extensions-contrib/k8s-jobs.html>

i messed up and didn’t specify that it was required, ill fix that

im pretty sure it has to be specified.  Looking at the code

so something as simple as this should do the trick?
```        druid.indexer.runner.javaOptsArray=["-Xmx1g"] ```

i can actually fix the code to make it work as desired

Hmmm all services are crashing, so looks like `druid-kubernetes-overlord-extensions` should exist _only_ in the load list for overlord, is that right?

yes we include it `only` in the overlord that extension

great, maybe small changes in the docs will make it easier to everyone!

yeah ill make a PR for the docs, its a bit weird since they are 2 separate extensions, the k8s discovery and overlord, but ill put a blurb in there about if you use the discovery, override these things in the overlord, is that what you were wanting?

i also fixed the javaOpts in the code in a PR that is open, when that gets merged, the docs will be truthful

I’d say
• maybe add all required fields to the properties table, so it’s very clear in a single place what’s expected and what’s configurable
• make it clear all configurations on for this extension are set only in Overlord section

we should thank you, great work, I believe this thing will be default behaviour in near future!

I just want to see this running, but I still didnt get it right…

what is the problem you are experiencing now?

Screenshot 2023-03-13 at 23.49.00.png

Screenshot 2023-03-13 at 23.47.49.png

tbh dunno, neither coordinator or overlord show me any log besides this (other services load just fine)
```│ 2023-03-13T23:41:53+00:00 startup service overlord                                                                                                                                                                                                                                                                                                             │
│ Setting druid.host=10.1.1.102 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties                                                                                                                                                                                                                                                        │
│ WARNING: An illegal reflective access operation has occurred                                                                                                                                                                                                                                                                                                   │
│ WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/druid/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)                                                                                                                  │
│ WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1                                                                                                                                                                                                                                            │
│ WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations                                                                                                                                                                                                                                                          │
│ WARNING: All illegal access operations will be denied in a future release```
But I can see something like this in other service logs and webconsole
```│ 2023-03-13T23:40:06,294 INFO [org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatchercoordinator] org.apache.druid.discovery.BaseNodeRoleWatcher - Node [<http://10.1.1.99:8081>] of role [coordinator] detected.                                                                                                                           │```
So I go to load data, submit a ingestion for wikipedia test data and… nothing happens, I cant even see a log as I get 404 status

druid.yaml

what do your overlord logs say when you grep for `wikipedia`

the overlord will log the task and what is it doing at each stage of the task

<@U04NHCE780G> <@U04L6JH4H2M> if all your efforts can be documented here <https://github.com/datainfrahq/druid-operator/issues/14>. It would be great help. Also we can add this in the operator e2e tests.
Thanks you :slightly_smiling_face:

For yesterday’s problem, looks like I can’t change the location here for either coordinator or overlord.
```      nodeConfigMountPath: "/opt/druid/conf/druid/cluster/master/coordinator-overlord"```
As soon as I set them back to this common value, services could start again, they coordinate using k8s API and not Zookeeper, which is great.

But as soon as I submit a the wikipedia ingest test, I got this

```│ 2023-03-14T12:02:09,548 ERROR [k8s-task-runner-0] org.apache.druid.indexing.overlord.TaskQueue - Failed to run task: {class=org.apache.druid.indexing.overlord.TaskQueue, exceptionType=class io.fabric8.kubernetes.client.KubernetesClientException, exceptionMessage=Failure executing: DELETE at: <https://10.96.0.1/apis/batch/v1/namespaces/druid/jobs/indexparallelwikipediafogdhebc20230314t120207965z>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. jobs.batch "indexparallelwikipediafogdhebc20230314t120207965z" is forbidden: 
││  User "system:serviceaccount:druid:default" cannot delete resource "jobs" in API group "batch" in the namespace "druid"., task=index_parallel_wikipedia_fogdhebc_2023-03-14T12:02:07.965Z, type=index_parallel, dataSource=wikipedia}                                                              ││ io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: DELETE at: <https://10.96.0.1/apis/batch/v1/namespaces/druid/jobs/indexparallelwikipediafogdhebc20230314t120207965z>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. jobs.batch "indexparallelwikipediafogdhebc20230314t120207965z" is forbidden: User "system:serviceaccount:druid:default" cannot delete resource "jobs" in API group "batch" in the namespace "druid".                                                                                        ││     at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:682) ~[?:?]                                                                                                                                                                                     ││     at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:661) ~[?:?]```
I have the suggested role + role binding, but maybe something is wrong here. I can’t say I understand these k8s roles and bindings very well.
```---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: druid-cluster
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - configmaps
      - jobs
    verbs:
      - '*'
---
kind: RoleBinding
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
metadata:
  name: druid-cluster
subjects:
  - kind: ServiceAccount
    name: default
roleRef:
  kind: Role
  name: druid-cluster
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>```

<https://github.com/apache/druid/blob/master/helm/druid/templates/overlord/role.yaml#L39-L44>

Interesting. Should this be the correct one?

```apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: druid-cluster
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - configmaps
      - jobs             &lt;--- required for the MM-less extension
    verbs:
      - '*'
  - apiGroups:
      - batch
    resources:
      - jobs
    verbs:
      - '*'```

```  - apiGroups:
      - ""```
This means the 'common' or 'base' api  (there is a specific term for it, i forget).  Jobs do not belong under this apiGroup

it *may* also need `pods/logs` under the first `apiGroups: [""]` for viewing logs.  I havent tested that yet, let me know if you do

So I’ve changed to
```apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: druid-cluster
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - configmaps
    verbs:
      - '*'
  - apiGroups:
      - batch
    resources:
      - jobs
    verbs:
      - '*'```
but now all pods crash during startup
```│ 2023-03-14T16:10:34,772 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [ANNOUNCEMENTS]                                                               ││ 2023-03-14T16:10:34,777 INFO [main] org.apache.druid.k8s.discovery.K8sDruidNodeAnnouncer - Announcing DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/router', host='10.1.1.251', bindOnHost=false, port=-1, plaintextPort=8088, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeRole='ROUTER', services={}}]                                                             ││ 2023-03-14T16:10:34,914 INFO [main] org.apache.druid.k8s.discovery.K8sDruidNodeAnnouncer - Json Patch For Node Announcement: [[{"op":"add","path":"/metadata/labels/druidDiscoveryAnnouncement-router","value":"true"},{"op":"add","path":"/metadata/labels/druidDiscoveryAnnouncement-id-hash","value":"1658736627"},{"op":"add","path":"/metadata/labels/druidDiscoveryAnnouncement-cluster-identifier","value":"druid-cluster"},{"op":"add","path":"/metadata/annotations/druidNodeInfo-router","value":"{\"druidNode\":{\"service\":\"druid/router\",\"host\":\"10.1.1.251\",\"bindOnHost\":false,\"plaintextPort\":8088,\"port\":-1,\"tlsPort\":-1,\"enablePlaintextPort\":true,\"enableTlsPort\":false},\"nodeType\":\"router\",\"services\":{}}"}]]                                                                                   ││ TRACE StatusLogger Log4jLoggerFactory.getContext() found anchor class org.apache.druid.java.util.common.logger.Logger                                                                                       ││ 2023-03-14T16:10:35,007 WARN [main] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 2) in 760ms.                                                                                              ││ <http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>: Failed to patch pod[druid/druid-druid-cluster-routers-0], code[403], error[{                                                                                          ││   "kind": "Status",
││   "apiVersion": "v1",
││   "metadata": {},
││   "status": "Failure",
││   "message": "pods \"druid-druid-cluster-routers-0\" is forbidden: User \"system:serviceaccount:druid:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"druid\"", "reason": "Forbidden",                                                                                                                                                                                    │
│   "details": {
│     "name": "druid-druid-cluster-routers-0",
│     "kind": "pods"
│   },
│   "code": 403                                                                                                                                                                                               
│ }].
│
│     at org.apache.druid.k8s.discovery.DefaultK8sApiClient.patchPod(DefaultK8sApiClient.java:71) ~[?:?]
│     at org.apache.druid.k8s.discovery.K8sDruidNodeAnnouncer.lambda$announce$0(K8sDruidNodeAnnouncer.java:112) ~[?:?]
(...)```


```kind: RoleBinding
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
metadata:
  name: druid-cluster
subjects:
  - kind: ServiceAccount
    name: default
roleRef:
  kind: Role
  name: druid-cluster
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>```

you are deploying into namespace `druid` ?

Screenshot 2023-03-14 at 16.18.45.png

and the pods are configured to use serviceAccount `default` ?

change:
```subjects:
  - kind: ServiceAccount
    name: default```
to
```subjects:
  - kind: ServiceAccount
    name: default
    namespace: druid```
in your roleBinding.  Not sure that will fix it, but that is the only diff i see between yours and mine

also confirm your podSpec is using `serviceAccount: default`

Screenshot 2023-03-14 at 16.25.54.png

&gt;  and the pods are configured to use serviceAccount `default` ?
I was assuming this to be true. But I’ve added `serviceAccount: default` and now I can see it in the pod, but I got the same errors

```│ Caused by: <http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>: Failed to patch pod[druid/druid-druid-cluster-overlords-0], code[403], error[{                                                                             ││   "kind": "Status",                                                                                                                                                                                         ││   "apiVersion": "v1",                                                                                                                                                                                       ││   "metadata": {},                                                                                                                                                                                           ││   "status": "Failure",                                                                                                                                                                                      ││   "message": "pods \"druid-druid-cluster-overlords-0\" is forbidden: User \"system:serviceaccount:druid:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"druid\"",             
││   "reason": "Forbidden",                                                                                                                                                                                    ││   "details": {                                                                                                                                                                                              
│     "name": "druid-druid-cluster-overlords-0",                                                                                                                                                              
│     "kind": "pods"                                                                                                                                                                                          
│   },                                                                                                                                                                                                        
│   "code": 403                                                                                                                                                                                               
│ }].```

yea if its not specified then its `default` ,  did you edit the roleBinding with the namespace?

I did. It’s like this now|

```kind: RoleBinding
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
metadata:
  name: druid-cluster
subjects:
  - kind: ServiceAccount
    name: default
    namespace: druid
roleRef:
  kind: Role
  name: druid-cluster
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>```

everything seems like it should be working

can you try deleting the pods that are failing just to make sure they are picking up the updated rbac

druid-cluster.yaml

I’m running kubectl delete and then apply every time, so everything (role, role binding and pods) is destroyed and recreated

i see from your config it should be in `logs/app.log`

Screenshot 2023-03-14 at 16.34.46.png

just to confirm, the role and roleBinding are both in the `druid` namespace ?

they used to be… :eyes: let me put them back

ok, they are namespace scoped resources, so they must be in the same namespace

Screenshot 2023-03-14 at 16.48.37.png

Thanks <@U04MRT25PSP> <@U04NHCE780G> <@U0322ET4ZGE>!

<@U04NHCE780G> can we add the role fix to the docs too?

Sure, its an OSS project you can just make a PR and add whatever you want to the docs.  I’ll approve

Hi, when do a division, i always get result as 0. My query is of type: (dist_total_revenue/total_revenue) where dist_total_revenue=long and total_revenue=long

here dist_total_revenue is awlays less than total_revenue

What error are you getting in the ingestion?

"taskId": "a42c79ce-48c7-4940-a311-cc8aae2fb34d",
   "state": "FAILED",
   "error": {
       "error": "Plan validation failed",
       "errorMessage": "org.apache.druid.java.util.common.ISE: Encountered null type for column[dist_total_revenue]",
       "errorClass": "org.apache.calcite.tools.ValidationException",
       "host": null
    }
}

Sounds like you have some null inputs.  Maybe you need something like NVL or ifnull, CASE, etc to replace any nulls with 0.

Hi ,when i ingested the data type as "float" instead of "floatSum" it worked.

interesting, gtk, thank you for the update

This is a discussion on how Kinesis works and how to achieve best throughput: <https://druid.apache.org/docs/latest/development/extensions-core/kinesis-ingestion.html#determining-fetch-settings>
Another interesting point is whether you are packing many rows into each message and using the deaggregation function at ingestion time. See here: <https://druid.apache.org/docs/latest/development/extensions-core/kinesis-ingestion.html#determining-fetch-settings>.

BTW... I haven't used it, but there is a suggestion in the docs that you use one fetchThread per shard, so you might want to tweak that too.

<@U030K4UM3H7> Is it possible to use the deaggregate function outside of ingestion time?

No. Unless I am missing something, deaggregate takes multiple rows packed into a single message and processes them as individual rows during ingestion. There is no mechanism to store a packed set of rows at ingestion time as a single row they should always be processed as individual rows at ingestion.

Following up on why aggregation. It seems to me that using aggregation on the producer side will have a potentially large effect on throughput. Here's a bit more info on the producing side for aggregation: <https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-concepts.html#kinesis-kpl-concepts-aggretation>

```BTW... I haven't used it, but there is a suggestion in the docs that you use one fetchThread per shard, so you might want to tweak that too.```
Is the fetchThreads the total threads for all the tasks or the total number of threads per task? I am assuming its the total number of threads for all the tasks?

I made the same assumption, but it’s a good question.

Let us know how it goes, we're always interested in seeing how folks are optimizing their ingestions.

Hi Didip, I believe it’s based on the cycle time of the coordinator ( <https://druid.apache.org/docs/latest/design/coordinator.html> )

data will be there in deep storage old versions will be marked as unused and dropped from druid  it will be there  in but deep storage . You will need to run kill tasks to delete those segments
<https://medium.com/nmc-techblog/data-retention-and-deletion-in-apache-druid-74ffd12398a8>
<https://druid.apache.org/docs/latest/tutorials/tutorial-delete-data.html>

You can configure kill tasks  or manually  triggered.

awesome, thanks folks. Looks like whenever the automatic kill job is running, it will correctly delete the unused versions.

I actually have an internal PR for hadoop ingestion that allows reading data from one cluster and write to other cluster. 

arrr… I would love to have that but it won’t work for our use-case because of org-wide policy of not sharing PG access between k8s clusters.

Well, all you need at the end of the day is the list of files from deep storage of Datasource A that you want to ingest. You could possibly execute sql against postgres for cluster 1, build the static path of the spec listing the files, and then submit the spec to do the batch ingest on cluster 2. Of course, it would be nice if Druid could provide that out of the box which shouldn't be too hard to implement, too.

The most unfortunate part is that data blob inside druid_segments, one has to deserialize/serialize the blob just to change the s3 bucket name. I wish those info are stored in real PG columns.

oh! perhaps I can simplify my problem by having the 2 clusters sharing the same s3 bucket. then it’s just pg_dump and pg_restore process.

My need happened to be for setting up a DR cluster for 0 downtime maintenance.

Hi, I am facing an issue with building SQL query in druid ( version =24.0.0) for percentage revenue for a given manufacturer for given year, month and quarter.

Below is my table: fact_extrapolated_revenue

with below query, i can get total revenue of manufacturer per year,month,quarter