Apache Druid

you should check out the specific upgrade notes for each of the releases (e.g. <https://github.com/apache/druid/issues/12510#0.23.0-upgrading-to-0.23.0>) to see if there is anything in your specific setup that you should be mindful of

OOM Killer killed it. The memory usage configuration is not correct.

<@U03149ALHN3> I had given these parameters as per druid recomendation only
``` value: -server -Xms8G -Xmx8G -XX:MaxDirectMemorySize=12G -Duser.timezone=UTC
            -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
            -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Djute.maxbuffer=16777216

resources:
          limits:
            cpu: "10"
            memory: 100Gi

# Processing threads and buffers
    druid.processing.buffer.sizeBytes=500000000
    druid.processing.numMergeBuffers=3
    druid.processing.numThreads=9
    druid.processing.tmpDir=var/druid/processing```

<@U03EXS199PD> , have you defined `requests`  memory?, as said <@U03149ALHN3>, 137 exit code is an OOM killer from system to the process caused by high memory request

if you update to Druid 25, there’s a script that auto tunes memory configs for you

Issue got resolved... Actually issue is not with memory... issue is with cores

Do you see any errors in the overlord log?

I snagged the entire block of config properties I set for reference,
```  druid_zk_service_enabled: "false"
  druid_serverview_type: "http"
  druid_discovery_type: "k8s"
  druid_discovery_k8s_clusterIdentifier: "test-druid"
  druid_discovery_k8s_podNameEnvKey: "HOSTNAME"
  druid_coordinator_loadqueuepeon_type: "http"
  druid_indexer_runner_type: "httpRemote"
  druid_indexer_storage_type: "metadata"```
It took some trial and error to get them correct.

This is what I got
```druid.discovery.k8s.clusterIdentifier=druid-prod
druid.zk.service.enabled=false
druid.discovery.type=k8s
druid.serverview.type=http
druid.coordinator.loadqueuepeon.type=http
druid.indexer.runner.type=httpRemote```

I see no obvious error in the coordinators

```2023-01-30T22:35:42,109 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Asking taskRunner to run: index_xxxxxx.sum_16a5545416f873a_gbgdoeib
2023-01-30T22:35:42,109 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner - Adding pending task[index_kafka_xxxxxx.sum_16a5545416f873a_gbgdoeib].
...
2023-01-30T22:35:47,399 WARN [KafkaSupervisor-xxxxx-Worker-1] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Ignoring task [index_kafka_xxxxxx_16a5545416f873a_gbgdoeib], as probably it is not started running yet
...
2023-01-30T22:36:40,901 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner - Assigned a task[index_kafka_xxxxxsum_16a5545416f873a_gbgdoeib] that is known already. Ignored.```

Does the task appear in the Middle Manager logs?

no worries at all, I feel like a newbie everyday :smile:

I guess all I needed, is talk to someone! thank you! All is good now :slightly_smiling_face:

I see what you mean by http thread leak :slightly_smiling_face:

Could this be related:
```"The worker that this task was assigned disappeared and did not report cleanup within timeout[PT15M]...."```

Potentially.  If the worker running the task had the pod evicted and didn't have the chance to announce shutdown that would be a possible result

No pod eviction. Tasks run inside the middler manager. No?

Are you running with middlemanagers or indexers?  We use indexers, the tasks run there

With middleManager. I'll switch to indexer

indexers work much better under k8s compared to middlemanagers.

Instead of forking JVMs it uses a threadpool, so it is much more efficient and can be scaled much more easily.

Damn! new feature I guess. DruidOperator doesn't seem to support that!

Oh I didn't know that, we don't use the operator.

Maybe I should stop using it, and just use the helm chart!

I was wrong, looking at the Operator code, it is using the Indexer. I moved to use that too.
Thanks :slightly_smiling_face:

we switched to using roaring bitmaps for indexes by default in druid 0.18, it could be related to that if you have lots of super high cardinality columns

<https://github.com/apache/druid/pull/9548>

performance is like a ton better with roaring though so the size is probably worth it unless you were never filtering by those columns

yeah, roaring bitmap is definitely preferable. I don’t want to switch it off

actually if you’re never filtering on the column, the better option is to not write indexes for it, there is a flag on dimension schema to control

Another related question, could imperfect rollup impacted the smoosh file?

also column order can play a pretty big role

if you’re on the latest version of druid and feeling adventurous and looking to make segments smaller, i did a thing that seems pretty cool in tests so far that added an option at ingest time to make the strings stored in segments smaller, particularly if they have lots of shared prefixes, such as urls

<https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1671170726474159> has details

its not really backwards compatible, so only make segments like that if you don’t need to read them on any older versions of druid

also definitely experiment with your workload to ensure there are no performance regressions that i’ve missed before using in production

theta skecth is an approximation so the reult will change. However it will give exact cound upto the size of the sketch

APPROX_COUNT_DISTINCT_DS_THETA(expr, [size])

you specify the size…larger size will take more memory and disk space

also error in the estimate depends on size <https://datasketches.apache.org/docs/Theta/ThetaErrorTable.html>

you can use init container to run curl and then copy the file to the place

or build a custom image and download at build time

building custom image is safer option as the curl/s3 could fail during the init container

Thanks, that what I figured :disappointed:

I think the *cachedNamespace* you can configure onHeap or offHeap bc it uses mapDb

but the Kafka Lookups is uses the underline local cache, so that depends on the cache config

the default is <https://github.com/ben-manes/caffeine|Caffeine>, that uses heap
<https://druid.apache.org/docs/latest/configuration/index.html#cache-configuration>

Hi <@U038XPQFJHM> where can i configure *cachedNamespace to off-heap?*

check out <https://druid.apache.org/docs/latest/development/extensions-core/lookups-cached-global.html#cache-settings>

it's not on the json where you set up the lookup, it's in the cluster (Broker, Peon, Historical) config

thanks <@U038XPQFJHM> will try that. So if i have say a lookup which is in heap and after this config changes, will it be removed and kept in offheap files?

yes, after the reload, they will not be stored anymore on heap

it still count as memory used, tho, but I'm not sure if it count on the jvm, or just at the kernel level (as it mmap)

<https://mapdb.org/blog/mmap_files_alloc_and_jvm_crash/> i dunno the implementation ,but there's a lot of warnings and reasons that this could go wrong

thank you <@U038XPQFJHM> for sharing the above link. I have gone through it and what i understand is JVM crash happens mainly during write and read shouldn't cause a crash.

And if offheap is enabled and when the lookups are created, it should crash during write and hence we can schedule the lookup creation during off hours right?

I do not use this extension, so I'm not sure, I think it re-scan periodically, but don't do always do a full-scan

I only use static lookups with a few thousand keys

if what you are doing is for joining, I recommend you to re-import the data with the lookup during the ingestion, as usually this doesn't change that much
but if the lookups changes, that's one of the hard problems, you gotcha need more RAM or use another solution for the query (spark?)

are you talking about disk usage or historical fill level? Are all the historicals on identical hardware?

<@U030W50CNAC>
Yes, all are identical have the same hardware.   When i look at output of /druid/coordinator/v1/servers?simple maxSize is ~6TB for all histroicals, but current size is much higher on few historicals, ie they are not balanced

```    "tier": "_default_tier",
    "type": "historical",
    "priority": 0,
    "currSize": 6247982185627,
    "maxSize": 6248103673856

    "tier": "_default_tier",
    "type": "historical",
    "priority": 0,
    "currSize": 3927430171420,
    "maxSize": 6248103673856 ```

```curl -sL <http://druid-coordinator:8080/druid/coordinator/v1/servers?full> | jq '.[]|{host:.host, segments: [.segments|to_entries[] | .key]}' | jq -c '[.host, (.segments|length)]'
["druidHistoricalBroker-6:8100",1438]
["druidHistoricalBroker-15:8100",18128]
["druidHistoricalBroker-9:8100",18988]
["druidHistoricalBroker-14:8100",18755]
["druidHistoricalBroker-8:8100",12589]
["druidHistoricalBroker-3:8100",17947]
["druidHistoricalBroker-12:8100",18283]
["druidHistoricalBroker-10:8100",17882]
["druidHistoricalBroker-13:8100",28791]
["druidHistoricalBroker-5:8100",29536]
["druidHistoricalBroker-1:8100",17969]
["druidHistoricalBroker-11:8100",19848]
["druidHistoricalBroker-2:8100",29918]
["druidHistoricalBroker-4:8100",18312]
["druidHistoricalBroker-7:8100",20744]
["druidHistoricalBroker-16:8100",19488]```


<@U04MC3QH8SH> You may try setting : `druid.coordinator.balancer.strategy`

Use `diskNormalized` to distribute segments among processes so that the disks fill up uniformly and use `random` to randomly pick processes to distribute segments.
<https://druid.apache.org/docs//0.15.1-incubating/configuration/index.html>

<@U03R5H3B750> i changed this setting and restarted coordinators waiting 8 hours nothing changed.

May be try to restart the master node and check the log for further information.

If you are counting something dynamic, and you are not concerned about retrieving the _actual_ value, have you considered using a sketch? <https://druid.apache.org/docs/latest/development/extensions-core/datasketches-hll.html>

Hi <@U030MBK46BD>
we can tolerate certain level of error, however I’m not sure I understand how sketches are answer my need
I mean I need simple counters and not some unique distinct counters, so why to use sketches?
my problem is that number of “keys” in dictionary of counters is not known beforehead and I can’t define static schema for those “keys”

Ah - ok - I misunderstood. Are you looking at the narrow in a narrow vs wide (<https://en.wikipedia.org/wiki/Wide_and_narrow_data> ) scenario?

ok, I understand, thanks!
this means that I need to model key to be part of dimension

IMG_20230201_154330.jpg

Likely your historicals are running out of CPU or memory

IMG_20230201_155912.jpg

Okay Vijay i am checking this our configuration for historicals

you can add a filter in the ingestion in the transformSpec

thanx, that's what I thought of. but by which column?
and won't it fail on the parsing before the filter anyway?

add a placeholder value for the __time and then filter out rows with that placeholder value

basically I think you can add a default time stamp where the timestamp is null.

Like Vijay said: order of operations is important
&gt; Conceptually, after input data records are read, Druid applies ingestion spec components in a particular order: first `flattenSpec` (if any), then `timestampSpec`, then `transformSpec`, and finally `dimensionsSpec` and `metricsSpec`. Keep this in mind when writing your ingestion spec.
So you need to have a default time set to something, then you can filter it with a `transformSpec`

That filter says you _only_ want `null` rows - you probably want a `not`
```  "filter": {
    "type": "not",
    "field": {
      "type": "selector",
      "dimension": "column3",
      "value": null
    }
  }```

Hi, i have a query that need to scan multiple years ( max upto 7 years) and do group by

SELECT SUM(fes.dist_total_revenue) as total_rev,sales_year,sales_month,sales_quarter FROM fact_revenue fes
where fes.__time BETWEEN TIME_PARSE ('2019-01-31T00:00:00.000Z') AND TIME_PARSE ('2022-11-30T00:00:00.000Z')
group by sales_year,sales_month,sales_quarter order by sales_year,sales_month,sales_quarter

This should ideally depend on your ingestion data and the querying needs. Partitioning by month would allow a better pruning during the query time, however in case there is less data for a month, it would lead to a blowup in the amount of segments. From the query given, I think that the data would be identically distributed across the months, and it might make more sense to partition by month, if there is not much historic data. Ideally, the segment sizes are kept around 5mn rows (or 3mn for SQL based ingestion), so if there is a lot less data than this, then it would make sense to increase the granularity to a YEAR.

If your only filter is on __time and you want to be able to filter by year or month, then I would think to have segment granularity at the month level.

To your last question -- if you are willing to denormalize a bit and add more filter conditions, you could create a new field  'month_num' and range partition on it to take advantage of secondary pruning, and potentially perfect rollup for your group bys.

Hi <@U04C4593VDY> thank you. When you say 'month_num', you mean 01 for Jan etc? For example, 2019-01-31 will have 'month_num' as 01?

also could you give an example of range partition using month_num please?

Hi <@U04K60E9N7L>, yes that's what I was thinking, generate a column that contains the numeric value for month, then you could issue queries with where conditions like "... and month between 1 and 3 ..."

For the example I just copied from the Druid doc page and modified the columns, e.g.:

```"partitionsSpec": {
  "type": "range",
  "partitionDimensions": ["month_num"],
  "targetRowsPerSegment": 5000000
}```
Although I would recommend looking for other secondary columns you would also filter on along with month_num, and add those to the list.

what's the source? Do you have many files?

Also, the recommendation for  `druid.indexer.fork.property.druid.processing.numThreads` is 1 or 2. 10 seems high, but I doubt that is the issue.

Actually, aren't you using Indexers? I don't think the "fork" properties apply, but I could be wrong. Instead you control the total processing thread pool with `druid.processing.numThreads` which should be set to the same value as the `druid.worker.capacity`  which should be set to (CPU cores - 1) if the Indexer runs on its own pod.

Still, none of that explains the fact that you get a single subtask, if you only have one file in your input source, that does explain it.

```{
  "type": "index_parallel",
  "spec": {
    "dataSchema": {
      "dataSource": "test",
      "timestampSpec": {
        "column": "timestamp_hourly",
        "format": "posix",
        "name": "__time"
      },
      "dimensionsSpec": {
        "dimensions": [
          ...
        ]
      },
      "metricsSpec": [ { "type" : "arrayOfDoublesSketch", ...},{ "type" : "longSum"...},{"type":"count"...}],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "hour",
        "queryGranularity": "hour",
        "rollup": true
      }
    },
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "s3",
        "prefixes": ["s3://...."]
      },
      "inputFormat": {
        "type": "parquet"
      },
      "appendToExisting": false
    },
    "tuningConfig": {
      "type": "index_parallel",
      "maxNumConcurrentSubTasks":4,
      "maxRowsPerSegment": 20000,
      "maxRowsInMemory": 20000
    }
  }
}```

image.png

What does the "index_parallel" task log say, it should start with some decision around how many tasks it will use.

I'm assuming that the prefixes include the s3 path(s) that contain the parquet files.
One thing that pops out at me is that there is no "<https://druid.apache.org/docs/latest/ingestion/native-batch.html#partitionsspec|partitionsSpec>" which governs the parallel task behavior. I'm not sure if that affects it this way though. Since you are doing rollup, a partitionSpec makes sense to get perfect rollup, Range partitioning is probably best to get even segments sizes and pruning on the partitioning dimension(s).

Also, have you considered using <https://druid.apache.org/docs/latest/multi-stage-query/index.html|SQL based ingestion> to do this, it uses a different, faster and more robust processing algorithm.  You can start one of those directly from the console Query view, by using "Connect external data" and specify your S3 input source there and then just adjust the generated SQL to do the rollup by just coding it as a SQL aggregation query. You control the parallelism of the ingestion directly in the UI or in the query context:

Oh, I didn't even know that existed!!!! I will check it out now! Thanks

Damn it! Found the issue. We have many files but not big enough. Druid decided that it can just read them all in 1 task. Which makes sense!
Added the following to test my theory
```      "splitHintSpec": {
        "maxSplitSize": 1000000,
        "type": "maxSize"
      },```


let us know how that goes. I was reading that as a max for each task before splitting into another task. But I didn't realize that it would also be used to control the number of tasks used. It's a good clarification for the docs. 

Yeap, it does control how many tasks by looking at the file size. tested and verified. :slightly_smiling_face:

Hi <@U04KYLUGGGK>, I think there is likely more nuance here.
I ran a few tests with Druid 25.0 on a nano-quickstart laptop deployment which I adjusted to have a middle manager with 4 task slots:

testing 2 small files, maxNumConcurrentSubtasks = 2
```dynamic, no splinthint  - 1 index_parallel task, 2 single_phase_sub_task
hashed, no splithint - 1 index_parallel, 2 partial_dimension_cardinality, 2 partial_index_generate, 1 partial_index_generic_merge```
I did not have to change the split hint to get parallel tasks, I did need two input files.

testing 2 small files, maxNumConcurrentSubtasks = 3
```hashed, no splithint - 1 index_parallel, 2 partial_dimension_cardinality, 2 partial_index_generate, 1 partial_index_generic_merge```
In this case it was limited by the number of input files, as expected.

testing 3 small files, maxNumConcurrentSubtasks = 3
```dynamic, no splithint - 1 index_parallel, 3 single_phase_sub_task
hashed, no splithint - 1 index_parallel, 3 partial_dimension_cardinality, 3 partial_index_generate, 1 partial_index_generic_merge```
Verified by adding another file and I see 3 tasks running in the input processing phases for either dynamic or hashed partitioning

I specified the list of files as a list of individual URIs, not sure if this has an effect. I'll try testing with like you did S3 prefixes instead.

So, yes, when using S3 prefixes in the input source definition, it does not automatically generate more tasks unless the set of files it finds, exceed the splitHint specified (by default 1000 files or 1GiB total bytes).

druid does exactly once ingestion from kafka. The best way to verify this is to load a static dataset into kafka and then ingest into druid and verify row counts.

The only reason druid data won’t match kafka is if there are parse errors which will be logged (if you have rollup in your ingestion then you need to have “count” as a metric and then do a sum(“count”)

there is no limitation but lookups are stored on heap and you need to keep that in mind

Hi <@U03BJAMQ4H5> if i make the config (druid.lookup.namespace.cache.type) as offheap as per this document, will it still store on heap? <https://druid.apache.org/docs/latest/development/extensions-core/lookups-cached-global.html#cache-settings>

in that case they will be stored off heap but the lookups will still take up memory in the page cache (so ultimately the available RAM will be a hard limit).

One other question related to druid.processing.buffer.sizeBytes. currently we have this configured as 1Gb. Will there be a query performance improvement if we keep the value higher at say 3 GB?

max buffer size is 2GB. Setting buffer to max 2GB could help but there is a trade off as it will reduce available memory for memory mapping segments and hence segments will be paged from disk

Untitled

Looks like someone else had this issue about 10 days ago in here but didn't get a response:
<https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1674483696642769>
CC <@U04JDGL76QM>

So I was able to dig in a bit.  One thing I noticed is the above error is about overlord so I went and took a look at the overlord pod which was failing its liveness check.  The logs showed basically nothing except what config was being loaded.  I had to exec into the pod and `cat /opt/druid/log/*` to find actual useful logs.  These should probably be getting output to stdout or stderr for the purposes of running in a container

I saw that overlord could not talk to postgres which was an error on my part.  I got that fixed and now it talks to postgres without issue.  The new issue is about listing the pods.

```2023-02-02T17:57:15,421 ERROR [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Failed to manage: {class=org.apache.druid.indexing.overlord.TaskQueue, exceptionType=class io.fabric8.kubernetes.client.KubernetesClientException, exceptionMessage=Operation: [list]  for kind: [Pod]  with name: [null]  in namespace: [druid1]  failed., restartDelay=30000}
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Pod]  with name: [null]  in namespace: [druid1]  failed.```

The pod is actually in a healthy state now though passing both liveness and readiness checks.  So maybe this is just an error because there are no jobs currently?  It probably should be getting handled better though

'Fixing' this however has had no effect on the errors initially described for `coordinator` and `historical`  pods.  Those still exist

hmmm that looks like a role / rolebinding issue

also one thing i noticed, if you are migrating a cluster from mm to mm-less with real-time indexing, it is wise to pause your supervisors then roll the upgrade, then unpause

I added the serviceAccount annotation (using iam role linked service accounts) and restarted the historical pod

I can see the pod now gets annotations for AWS credentials but it still gives the same error

```        - name: AWS_STS_REGIONAL_ENDPOINTS
          value: regional
        - name: AWS_DEFAULT_REGION
          value: us-west-2
        - name: AWS_REGION
          value: us-west-2
        - name: AWS_ROLE_ARN
          value: 'arn:aws:iam::REDACTED:role/druid-s3'
        - name: AWS_WEB_IDENTITY_TOKEN_FILE
          value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token```

I'll create a sample pod using this serviceaccount/namespace just to be sure I can access the configured s3 bucket

unless there are AWS permissions outside of the s3 setup that I need ?

<https://druid.apache.org/docs/latest/development/extensions-core/kubernetes.html>

probably good to add “jobs” to the resource as well

OK, i was able to test against the namespace/serviceAccount druid is using and successfully write to the configured druid s3 bucket using a test pod

Yes, my role already has the jobs permission

```$ kubectl -n druid1 get role -o yaml
apiVersion: v1
items:
- apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
  kind: Role
  metadata:
    name: druid-cluster
    namespace: druid1
    resourceVersion: "75299544"
    uid: 1cc3f0b5-4456-4880-a739-9a9d4b97f83e
  rules:
  - apiGroups:
    - ""
    resources:
    - pods
    - configmaps
    - jobs
    verbs:
    - '*'
kind: List
metadata:
  resourceVersion: ""```

```$ kubectl -n druid1 get rolebinding -o yaml
apiVersion: v1
items:
- apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
  kind: RoleBinding
  metadata:
    name: druid-cluster
    namespace: druid1
  roleRef:
    apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
    kind: Role
    name: druid-cluster
  subjects:
  - kind: ServiceAccount
    name: default
kind: List
metadata:
  resourceVersion: ""```

which version of k8s are you running with?

This is a brand new druid deployment.  never been used or had a task run

Sure... but to be completely transparent,  I've never used druid before.  I'm attempting to deploy this for a developer group  :slightly_smiling_face:   So i'll have to figure out how to do that

there are two sections here, `supervisors` and `tasks`

click submit json task and paste what i gave you

then lets track that task through the overlord logs

At the top I get `Task submitted successfully` , but then an error pops up
```
Unknown exception / Failed to talk to leader node at [/druid/indexer/v1/tasks]. Error code [500], description [Server Error]. / <http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>```

no, both `coordinator` and `historical` pods cannot start due to the error in the snippet at the top of this thread

the kubernetes-overlord extension is `only` loaded for the overlord

the rest of the services can’t have that

ugh. i freaking *knew* it.  The docs suggest that but it wasn't clear how to do that for a specific service

then in your overlord, you’ll have [a,b,c,druid-kubernetes-overlord-extensions]

no worries, if its not clear, i can add that to the docs

you could not have a better 'dummy' than me.  I know literally nothing about this stuff haha :sweat_smile:

lol, no worries, we will get you up to speed

cool, once you apply that fix, you can test again with the “noop” task json i gave you

that is a good canary to check if everything works

do you want me to attempt and start coordinator pods first ?

updating the configmap won't do that automatically

well... they aren't dying right away.  that is a big improvement

definitely lets work on getting druid in a good healthy state, then we can worry about the rest

OK, so coordinator has some errors in its log file.  Some things to note:
• these logs from `/opt/druid/log/*` do not make it to stderr/stdout 

So I have to exec into the pod to look at them

take your time, once druid is up…lets continue

• The log file seems to be a variable that is not being interpolated:   `bash-5.1$ cat /opt/druid/log/\$\{sys\:druid.node.type\}.log`

I made the change you noted and then deleted the pod once the configmap was updated

do you use the druid operator to do your deployments?

no, i am using the helm chart in the main repo

that does make things a lot easier in k8s

but kind of wishing i went and used the operator

:slightly_smiling_face: no big deal…. i just use the operator and am not really that familiar with the helm charts

I didn't want to have to write a helm chart to deploy the `Druid` custom resource,  so i went with this

so the historical pod is still not starting

but the error is now hidden.  its not producing the above message anymore

yeah that error should go away once the extension is gone

I get logging up to the config-load and the warning about reflexive access

nothing in the logs that show why things are not working?

and then the pod dies with no other information

so you shouldn’t need helm to deploy the crd right?  Its pretty much a static file you can just do a `kubectl apply -f crd.yaml`

yea, but infrastructure as code, gitops, etc. etc.

you can see the coordinator cycle runs in raw metrics -&gt; loadqueue

thanks <@U03TAQRMJPQ> we are not using Imply solution
load queue however is metric exposed by coordinator isn’t it?
I mean if historical is hanging for any reason it wont reflect it’s ‘liveness’

that is correct, it gets it from the coordinator

it would be cool if someone added that as a column to the `"sys"."server_segments"` table!

The broker is not generally involved with ingestion, its does parse SQL Based Ingestion commands to convert them into native ingestion instructions, but then the work is done by the Overlord and MM/Indexers. Look for problems in the overlord and MM/indexer/task logs. It could also be a problem with coordinator at the time of handoff to the historicals.
How are you determining that only half the data was ingested?

Hi Sergio, thanks for the reply. We determined only *half of the total rows being ingested is becoming available in the datasource* by referencing the ingestion task report of the multiple ingestion tasks that complete and save data into the datasource in question. The sum of the processed metrics for all the corresponding ingestion task reports was equal to exactly double of the number of rows shown in the resulting data source

One of my teammates gave me the advice to start scaling down the number of rows being ingested and determining how that affects the data leak. The more data being ingested the greater the proportion of data that was missing in the resulting datasource with a complete run resulting in a 50% leak

We referenced the logs you referred to but didn't find any clear answers. I can run the service that ingests data into the datasource with the leak on demand and provide you with specific logs if that helps

Yes. That would help. Particularly the task logs. But back to the question, how do you know that data is missing. Is is the result of a query? What’s the query? Is it by row count? If so, are you using roll up at ingestion?

Yea, we have a data connector that our devs built that connects an internal visualization tool to the Druid server. One of the internal stakeholders noticed that the visualizations were off so I looked into it and ran some basic queries which verified that there was data leakage

Are you going to be around for a bit? I can run the service and get whichever logs you need

A common mistake with rollup ingestion is to  compare the count of rows processed at ingestion to the count of rows actually ingested. Since there is aggregation, these ingestions usually calculate a “count” column that indicates how many source rows were rolled up into the target row. So a sum(“count”) is used to verify the rows ingested. Don’t know if this is your case…yes, I’m around.

Gotcha, but also verified that there's missing rows by running the queries like I mentioned. Its product event data so looked for basics like daily active users which was clearly off which wasn't the case previously

Also noticed that when I set the maxBytesInMemory parameter to -1 and rely on the maxRowsInMemory parameter value, the data leak decreased

Ok finished. ~880k processed. 575k rows in the data source

When I didn't set the -1 flag, 553k rows in the datasource

Not unless there is a bug. So, maxRowsInMemory/maxBytesInMemory control the real-time ingestion buffer. Once this limit is reached, the ingestion does a local temporary persist and continues ingesting into a new buffer. I don't think it would have a direct effect. After the segment level thresholds are reached, the ingestion task merges all the temporary persists into segment files and publishes them.
Let's take a look at the logs and then also check that the coordinator/historicals are loading all the segments that the ingestion produced. Do you see full availability in the data sources view?

did you say this was a rollup ingestion? or not rollup?

task-log-index_parallel_production_event_data_updated_PublicProduction_test_iiilfgap_2023-02-02T23_02_07.324Z.log

I didn't include any rollup specs in the task

task-log-index_parallel_production_event_data_updated_PublicProduction_test_hoiajnpm_2023-02-02T23_06_08.186Z.log

To close out this thread... we did a bit of troubleshooting offline and discovered that there were two ingestion jobs for the same data source running one after the other with `appendToExisting:false` and covering the same overall timeframe. This caused the second job to overwrite any segments (most of them) that covered the same time interval, by switching `appendToExisting:true`, the second job added segments instead of replacing them and then we got the full row count.

I opened up the service account role to full permissions and it still doesn't help

Hey <@U030MNRGBNG>, sorry for pinging you again but I consider you the `druid-kubernetes-extension` expert. Can you help?

Thanks <@U030K4UM3H7>
So, I was able to work past the immediate issue above by adding an annotation via the helm chart supplied in the `apache/druid` repo.  It seems the error comes from there being no existing annotations on the statefulset pods.  I added `druid: true` to the annotations and I was able to move to the next error. The next error seems to be the same in nature  though:
```2023-02-02T22:56:47,927 INFO [main] org.apache.druid.k8s.discovery.K8sDruidNodeAnnouncer - Json Patch For Node Announcement: [[{"op":"add","path":"/metadata/labels/druidDiscoveryAnnouncement-historical","value":"true"},{"op":"add","path":"/metadata/labels/druidDiscoveryAnnouncement-id-hash","value":"1962150723"},{"op":"add","path":"/metadata/labels/druidDiscoveryAnnouncement-cluster-identifier","value":"druid1"},{"op":"add","path":"/metadata/annotations/druidNodeInfo-historical","value":"{\"druidNode\":{\"service\":\"druid/historical\",\"host\":\"10.233.87.196\",\"bindOnHost\":false,\"plaintextPort\":8083,\"port\":-1,\"tlsPort\":-1,\"enablePlaintextPort\":true,\"enableTlsPort\":false},\"nodeType\":\"historical\",\"services\":{\"dataNodeService\":{\"type\":\"dataNodeService\",\"tier\":\"_default_tier\",\"maxSize\":10737418240,\"type\":\"historical\",\"serverType\":\"historical\",\"priority\":0},\"lookupNodeService\":{\"type\":\"lookupNodeService\",\"lookupTier\":\"__default\"}}}"}]]
2023-02-02T22:56:47,960 INFO [main] org.apache.druid.k8s.discovery.K8sDruidNodeAnnouncer - Announced DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/historical', host='10.233.87.196', bindOnHost=false, port=-1, plaintextPort=8083, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeRole='HISTORICAL', services={dataNodeService=DataNodeService{tier='_default_tier', maxSize=10737418240, serverType=historical, priority=0}, lookupNodeService=LookupNodeService{lookupTier='__default'}}}]
2023-02-02T22:56:47,960 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Successfully started lifecycle [module]
2023-02-02T22:56:48,064 INFO [qtp138617961-59] org.apache.druid.server.coordination.SegmentLoadDropHandler - Starting...
2023-02-02T22:56:48,066 WARN [qtp138617961-59] org.apache.druid.guice.LifecycleScope - Caught exception when trying to create a[Key[type=org.apache.druid.server.coordination.SegmentLoadDropHandler, annotation=[none]]]
java.lang.reflect.InvocationTargetException: null```

ah. there is more to the error a bit below
```Caused by: org.apache.druid.java.util.common.IOE: Cannot create directory [/var/druid/segment-cache/info_dir]```

that directory is not set by default in the helm chart but it is shown as the example

<https://github.com/apache/druid/blob/master/helm/druid/values.yaml#L196>
`/opt/druid/var/druid/` is what is mounted for the statefulset... so maybe something in there is more appropriate?   unclear

Cool. It is probably defaulting to /var/druid/segment-cache

yea the helm chart does not match that for the mount location.  Honestly there is a lot to be desired about the helm chart.  It also does not allow for specifying serviceAccounts which is a pretty big deal (you have to use the default which means your only level of segregation becomes the namespace and all pods get same permissions)

the mount location for the above is hard-coded
maybe the operator is the direction things are going though and I should be using that?

I'm happy to make a PR against the current helm chart but I see a few that have been sitting open for a while.  Let me know if its welcomed or if the operator is where the focus should be.

By all means! We need a better helm chart, so any help from the community in making it better is very welcome.

<https://github.com/apache/druid/pull/13746>
<https://github.com/apache/druid/pull/13747>