Luke Foskey
05/01/2025, 3:21 AMMahesha Subrahamanya
05/04/2025, 5:47 AMMahesha Subrahamanya
05/06/2025, 9:20 PMLis Shimoni
05/06/2025, 9:22 PMdruid-scaling
automatic process and need some guidance. Is there an API that allows me to stop requests for historical and broker nodes before removing them? For example, something like:
curl -X POST "<http://localhost:8081/druid/coordinator/v1/servers/10.120.122.122:8083/disable>"
I've tried this approach, but it didn't work. Any help would be appreciated!Stefanos Pliakos
05/07/2025, 7:01 AMUdit Sharma
05/07/2025, 9:35 AM"groupBy_JoinDataSource{left=event-trst-36745eaf-8215-4dd2-baed-a6bc123e3b02, right=InlineDataSource{signature={d0:LONG}}, rightPrefix='__j0.', condition=("long42" == "__j0.d0"), joinType=INNER, leftFilter=null}_[2025-04-04T00:00:00.000Z/2025-04-05T00:00:00.000Z]_54f468cf-e44b-4835-895a-e3f48cec0db4" #205 daemon prio=5 os_prio=0 cpu=67305835.68ms elapsed=763356.58s tid=0x00007b2f00492800 nid=0x10f runnable [0x00007b2c06af4000]
java.lang.Thread.State: RUNNABLE
at org.apache.druid.segment.join.HashJoinEngine$1JoinCursor.matchCurrentPosition(HashJoinEngine.java:193)
at org.apache.druid.segment.join.HashJoinEngine$1JoinCursor.advanceUninterruptibly(HashJoinEngine.java:223)
at org.apache.druid.segment.join.HashJoinEngine$1JoinCursor.advance(HashJoinEngine.java:180)
at org.apache.druid.segment.join.HashJoinEngine$1JoinCursor.initialize(HashJoinEngine.java:157)
at org.apache.druid.segment.join.HashJoinEngine.makeJoinCursor(HashJoinEngine.java:254)
at org.apache.druid.segment.join.HashJoinSegmentStorageAdapter.lambda$makeCursors$1(HashJoinSegmentStorageAdapter.java:339)
at org.apache.druid.segment.join.HashJoinSegmentStorageAdapter$$Lambda$1184/0x00000008009bf440.apply(Unknown Source)
at org.apache.druid.java.util.common.guava.Sequences$$Lambda$653/0x0000000800aa7040.apply(Unknown Source)
at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40)
at org.apache.druid.java.util.common.guava.FilteringAccumulator.accumulate(FilteringAccumulator.java:41)
at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40)
at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44)
at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
at org.apache.druid.java.util.common.guava.FilteredSequence.accumulate(FilteredSequence.java:45)
at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
at org.apache.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:42)
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40)
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40)
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
at org.apache.druid.query.spec.SpecificSegmentQueryRunner$1.accumulate(SpecificSegmentQueryRunner.java:103)
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
at org.apache.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:190)
at org.apache.druid.query.spec.SpecificSegmentQueryRunner.access$100(SpecificSegmentQueryRunner.java:45)
at org.apache.druid.query.spec.SpecificSegmentQueryRunner$2.wrap(SpecificSegmentQueryRunner.java:170)
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:110)
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:252)
at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:239)
at java.util.concurrent.FutureTask.run(java.base@11.0.7/FutureTask.java:264)
at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:251)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.7/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.7/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
Mahesha Subrahamanya
05/08/2025, 2:25 AMsuhas panchbhai
05/08/2025, 8:16 AMmehrdadbn9
05/10/2025, 7:33 PMMahesha Subrahamanya
05/12/2025, 9:50 PMSoman Ullah
05/13/2025, 11:23 PMKonstantin Minevskiy
05/14/2025, 5:15 PMloading lookup
. Global cached lookup (JDBC, specifically) works fine, but it’s becoming a problem due to the size of the data being retrieved from the db. We’re on the latest Druid (33.0.0). More info in the 🧵Jvalant Patel
05/14/2025, 10:42 PMnull
to the latest Druid version where legacy mode is not supported, just wanted to get some help from here to know what should be the best strategy to upgrade druid if we have null
and ""
strings in the datasources and our queries rely on the legacy behavior. If we want to rewrite queries to handle three valued logic for null
comparisons, what should be the strategy ? is there any generalized way to modify the queries ? we are still using native Druid query language.Rohen
05/19/2025, 1:14 PMRohen
05/19/2025, 1:14 PMUdit Sharma
05/19/2025, 1:43 PMselect distinct customer from events where __time BETWEEN TIMESTAMP '2025-03-20 12:30:00'
AND TIMESTAMP '2025-05-19 13:00:00' AND
customer IN (
'2140', '1060', '2207', '1809', '2985',
'3026', '2947', '2955', '2367', '2464',
'899', '355', '3284', '3302', '1034',
'3015', '2127', '2123', '2731', '2109',
'2832', '2479', '2702', '2387', '1804',
'1018', '1364', '3467', '1028', '850'
)
While this seems to return the right results.
select distinct custId from events where __time BETWEEN TIMESTAMP '2025-03-20 12:30:00'
AND TIMESTAMP '2025-05-19 13:00:00' AND
custId IN (
'2140', '1060', '2207', '1809', '2985',
'3026', '2947', '2955', '2367', '2464',
'899', '355', '3284', '3302', '1034',
'3015', '2127', '2123', '2731', '2109',
'2832', '2479', '2702', '2387', '1804',
'1018', '1364', '3467', '1028', '850'
)
Druid Version : 26.0.0JRob
05/22/2025, 5:53 PMCannot construct instance of `org.apache.druid.data.input.protobuf.FileBasedProtobufBytesDecoder`, problem: Cannot read descriptor file: file:/tmp/metrics.desc at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 1090] (through reference chain: org.apache.druid.indexing.kafka.KafkaSamplerSpec["spec"]->org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec["ioConfig"]->org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig["inputFormat"]->org.apache.druid.data.input.protobuf.ProtobufInputFormat["protoBytesDecoder"])
I suspect that Druid is trying to download the filer over HTTP but we would never expose /tmp
to the internet. Why doesn't it just grab the file locally?
For example, this works:
{
"type": "index_parallel",
"spec": {
"ioConfig": {
"type": "index_parallel",
"inputSource": {
"type": "local",
"baseDir": "/tmp/",
"filter": "metrics.desc"
}
},
"tuningConfig": {
"type": "index_parallel"
}
}
}
However, I can't get this working with inputFormat
Utkarsh Chaturvedi
05/23/2025, 10:13 AMBrindha Ramasamy
05/23/2025, 6:30 PMRohen
05/26/2025, 1:44 PMJRob
05/28/2025, 9:14 PM1) No implementation for org.apache.druid.server.metrics.TaskCountStatsProvider was bound.
while locating org.apache.druid.server.metrics.TaskCountStatsProvider
for the 1st parameter of org.apache.druid.server.metrics.TaskCountStatsMonitor.<init>(TaskCountStatsMonitor.java:40)
Hardik Bajaj
05/29/2025, 7:12 PMSeki Inoue
06/02/2025, 4:47 PM.coordinator-issued_kil...
had 265 bytes length and it exceeds the XFS limit of 255 bytes.
Do you know any work around to forcibly kill those segments?
2025-05-30T22:10:42,465 ERROR [qtp214761486-125] org.apache.druid.indexing.worker.WorkerTaskManager - Error while trying to persist assigned task[coordinator-issued_kill_<deducted_long_datasource_name_119_bytes>]
java.nio.file.FileSystemException: var/tmp/persistent/task/workerTaskManagerTmp/.coordinator-issued_kill_<deducted_long_datasource_name_119_bytes>_dfhlgdae_2024-07-10T23:00:00.000Z_2024-07-18T00:00:00.000Z_2025-05-30T22:10:42.417Z.2aababbd-02a6-4002-9b9f-cba30bbea8a7: File name too long
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) ~[?:?]
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:181) ~[?:?]
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298) ~[?:?]
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357) ~[?:?]
at org.apache.druid.java.util.common.FileUtils.writeAtomically(FileUtils.java:271) ~[druid-processing-33.0.0.jar:33.0.0]
...
Asit
06/03/2025, 4:25 AMJon Laberge
06/04/2025, 5:35 AMkubernetes-overlord-extensions
, however I see this error when the overlord is trying to start:
Caused by: org.apache.commons.lang3.NotImplementedException: this druid.indexer.logs.type [class org.apache.druid.storage.google.GoogleTaskLogs] does not support managing task payloads yet. You will have to switch to using environment variables
Is there something I should be changing in my task template?Jimbo Slice
06/06/2025, 9:55 PMSELECT
COUNT(*) As Entries,
SUM(packets) as Packets,
SUM(bytes) as Bytes,
(SUM(bytes) / SUM(packets)) as AvgPacketSizeBytes,
MIN(__time) as FirstSeen,
MAX(__time) as LastSeen,
TIMESTAMPDIFF(SECOND, MIN(__time), MAX(__time)) as DurationSeconds,
(SUM(bytes) * 8 / TIMESTAMPDIFF(SECOND, MIN(__time), MAX(__time))) as AvgMbps,
"pkt-srcaddr", "pkt-dstaddr", "protocol"
FROM "AWSLogsVPC"
WHERE "log-status"!='NODATA' AND "pkt-srcaddr"!='-' AND "action"='ACCEPT'
GROUP BY "pkt-srcaddr", "pkt-dstaddr", "protocol"
But when i remove the TIMESTAMPDIFF section for AvgMbps this does not happen:
SELECT
COUNT(*) As Entries,
SUM(packets) as Packets,
SUM(bytes) as Bytes,
(SUM(bytes) / SUM(packets)) as AvgPacketSizeBytes,
MIN(__time) as FirstSeen,
MAX(__time) as LastSeen,
TIMESTAMPDIFF(SECOND, MIN(__time), MAX(__time)) as DurationSeconds,
(SUM(bytes) * 8) as AvgMbps,
"pkt-srcaddr", "pkt-dstaddr", "protocol"
FROM "AWSLogsVPC"
WHERE "log-status"!='NODATA' AND "pkt-srcaddr"!='-' AND "action"='ACCEPT'
GROUP BY "pkt-srcaddr", "pkt-dstaddr", "protocol"
I've tried removing the "WHERE" because != is bad practice, no difference, I believe there is an issue here with subquerying (druid.server.http.maxsubqueryrows) - however this is not a subquery, this is a simple calculation in a simple query.
This query runs perfectly without TIMESTAMPDIFF(SECOND, MIN(__time), MAX(__time))
being called in AvgMbps.
Any ideas on what could be wrong???Ben Krug
06/06/2025, 10:07 PMvenkat
06/07/2025, 8:50 AMvenkat
06/07/2025, 8:54 AMsandy k
06/09/2025, 4:43 AM