Apache Druid

Hi <@U030K4UM3H7> I worked with vishal, thanks for responding, Here is the asked files

There's a "druid doctor" but that also not for the config itself

the time interval being weird was being due to the segment granularity set to ALL, i have dropped it and reingested with segment granularity set to DAY. Now i see proper values, but still compaction seeing same issue. Awaiting first run

if I run the query with just a 'primary key' (mm) and max(last_seen) and count, the results are correct

if I add an column (other than the one being filtered) the mm=3617382237 got a different max() value

if I filter by it, I also got the right result

If i filter using the max query, I got this timestamp

if I filter (mm) I got the correct result

there's just two rows for this object, both are on the historicals considering the __time

(without, I got 1970-02-11T22:09:09.414Z)

filtering by 'having' or adding other having ops does not change it

Using a having(max(last_seen) &gt;0 ) does indeed fixes it
(both with or without using MM IN)

If I add group by some column using mm (eg: mm=0) it also got the right result

if I remove (the column, or just the group by) the result is back to *1970-02-11T22:09:09.414Z*

still happening today, there's a new device

if I add an fixed column, it solves the bug

<@U038XPQFJHM> it's running on k8s. Segment size is around 20 KB each. I have attached the JVM metrics in case it's of any help. I don't see any thing wrong in logs.

One more forgot to mention when
`druid.emitter.composing.emitters=["logging","prometheus"]`
    `#druid.emitter=prometheus` configs are mentioned together we are getting below error

I got the below response along with the HTTP 400 status code:thinking_face:

Hi,

Thanks for comments.

When the Druid is started, for first 12 minutes, coordinator is not starting at all.
Below one is the output of oc get po command output -

`druid-tsdb-apache-druid-brokers-656d576d7d-shps9             0/1     Running            0          11m`
`druid-tsdb-apache-druid-historicals-0                        0/1     Running            0          11m`
`druid-tsdb-apache-druid-historicals-1                        0/1     Running            0          11m`
`druid-tsdb-apache-druid-middlemanagers-0                     0/1     Running            0          11m`
`druid-tsdb-apache-druid-middlemanagers-1                     0/1     Running            0          11m`

After 12 minutes, the coordinator pod is coming up -

`druid-tsdb-apache-druid-brokers-656d576d7d-shps9             1/1     Running            0          20m`
`druid-tsdb-apache-druid-coordinators-5ccf8c468f-g742f        1/1     Running            0          7m`
`druid-tsdb-apache-druid-historicals-0                        1/1     Running            0          20m`
`druid-tsdb-apache-druid-historicals-1                        1/1     Running            0          20m`
`druid-tsdb-apache-druid-middlemanagers-0                     1/1     Running            0          20m`
`druid-tsdb-apache-druid-middlemanagers-1                     1/1     Running            0          20m`
`druid-tsdb-apache-druid-routers-787d75c9ff-b9md8             1/1     Running            0          5m40s`

Please refer to the last column which denotes the pod up time.

I have enabled the logs with debug level, and attached the logs of brokers, historicals, middlemanagers and coordinators.
Please guide how to proceed further.

Thanks!

what error do you get &amp; what version are you on? seems to work for me

This is how I have the retention rules configured:

as for the replica, it's configured via dynamic settings, per datasource, you can use druid console to edit clicking in the pencil icon on the retention column
eg: to have the last day 3 replicas (so 2 nodes can crash, still have the data consistent for that period)

My middlemanager is not running ig and there is logs also. when i checked the describe pod getting this  `Warning  Unhealthy  61s (x22 over 3m59s)  kubelet            Readiness probe failed: Get "<http://10.42.2.217:8088/status/health>": dial tcp 10.42.2.217:8088: connect: connection refused`

Yeah, set druid.port: 8088.....  We can give any port under druid.port?

Tried setting log level to Trace also no output on logs same like this <https://apachedruidworkspace.slack.com/files/U03M70PELMN/F03NXM030HF/image.png>

And while I get 504 Druid Doctor tells that everything is perfect…

this what log tells at the end i don't know what its suppose to mean

Could you please double check that the top left schema selection says "druid" and not "sys"?

somehow we have mixed partitioning strategy

yup, the dropdown is druid. all other datasources exist except that one.

Do you have enough task slots? You can see that in the services tab:

It's not showing the task slots. Is there another way to check it ?

Same as the OP, no MiddleManager but a Peon instance.