Jvalant Patel
04/11/2024, 6:50 PM2024-04-10T22:09:12,058 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Uncaught Throwable while running task[AbstractTask{id='partial_index_generic_merge_<datasource>_bnopakcn_2024-04-10T22:08:49.696Z', groupId='coordinator-issued_compact_<datasource>_dhfpacdl_2024-04-10T21:51:25.920Z', taskResource=TaskResource{availabilityGroup='partial_index_generic_merge_<datasource>_bnopakcn_2024-04-10T22:08:49.696Z', requiredCapacity=1}, dataSource='pnr_page_request', context={useLineageBasedSegmentAllocation=true, storeCompactionState=true, appenderatorTrackingTaskId=coordinator-issued_compact_<datasource>_dhfpacdl_2024-04-10T21:51:25.920Z, priority=25, forceTimeChunkLock=true}}]
java.lang.OutOfMemoryError: Cannot reserve 65536 bytes of direct buffer memory (allocated: 5368678031, limit: 5368709120)
at java.nio.Bits.reserveMemory(Bits.java:178) ~[?:?]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121) ~[?:?]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332) ~[?:?]
at org.apache.druid.segment.CompressedPools$4.get(CompressedPools.java:102) ~[druid-processing-29.0.0.jar:29.0.0]
at org.apache.druid.segment.CompressedPools$4.get(CompressedPools.java:95) ~[druid-processing-29.0.0.jar:29.0.0]
at org.apache.druid.collections.StupidPool.makeObjectWithHandler(StupidPool.java:184) ~[druid-processing-29.0.0.jar:29.0.0]
at org.apache.druid.collections.StupidPool.take(StupidPool.java:156) ~[druid-processing-29.0.0.jar:29.0.0]
We have configured peon tasks to run with 8 GB Heap and 6 GB Direct Memory. Tasks fail within the 2 minutes of the initialization. Does anyone know how to solve this? Any pointers will be super helpful. TIA.Sharmin Choksey
04/11/2024, 7:23 PMAditya Randive
04/11/2024, 11:44 PMapache-druid-29.0.1
directory.
curl -X 'POST' -H 'Content-Type:application/json' -d @quickstart/tutorial/wikipedia-index.json <http://localhost:8081/druid/indexer/v1/task>
After the task fails, on the console the error details is:
failed to run task with an exception. See middleManager or indexer logs for more details.
Can someone help guide me what may be going wrong?Bharat
04/12/2024, 8:53 AMBharat
04/12/2024, 8:54 AMAmit Jain
04/12/2024, 10:30 AMPio Salvatore MORRONE (KEBULA)
04/12/2024, 1:29 PMlukas
04/15/2024, 5:51 PM{
"type": "kill",
"id": "api-issued_kill_<datasource>_oicdplgc_1000-01-01T00:00:00.000Z_3000-01-01T00:00:00.000Z_2024-04-15T17:39:07.818Z",
"dataSource": "<datasource>",
"interval": "1000-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z",
"context": {
"forceTimeChunkLock": true,
"useLineageBasedSegmentAllocation": true
},
"batchSize": 100,
"limit": null,
"groupId": "api-issued_kill_<datasource>_oicdplgc_1000-01-01T00:00:00.000Z_3000-01-01T00:00:00.000Z_2024-04-15T17:39:07.818Z",
"resource": {
"availabilityGroup": "api-issued_kill_<datasource>_oicdplgc_1000-01-01T00:00:00.000Z_3000-01-01T00:00:00.000Z_2024-04-15T17:39:07.818Z",
"requiredCapacity": 1
}
}
The task runs successfully, but no unused segments have been deleted (neither in Postgres not in S3 in this case).
I can query them like this in Postgres:
select *
from druid_segments
where datasource = '<datasource>' and used = false
limit 10
and I still get results. Do you know what might be the problem?Jvalant Patel
04/15/2024, 8:03 PMJiaojiao Fu
04/16/2024, 7:54 AM2024-04-15T16:19:20,946 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Exception while running task[AbstractTask{id='index_parallel_custom_lists_bhnobale_2024-04-15T16:17:13.511Z', groupId='index_parallel_custom_lists_bhnobale_2024-04-15T16:17:13.511Z', taskResource=TaskResource{availabilityGroup='index_parallel_custom_lists_bhnobale_2024-04-15T16:17:13.511Z', requiredCapacity=1}, dataSource='custom_lists', context={forceTimeChunkLock=true, useLineageBasedSegmentAllocation=true}}]
java.lang.RuntimeException: org.apache.druid.rpc.HttpResponseException: Server error [400 Bad Request]; body: {"error":"Task[single_phase_sub_task_custom_lists_hdihfjom_2024-04-15T16:17:19.655Z] already exists!"}
at org.apache.druid.common.guava.FutureUtils.getUnchecked(FutureUtils.java:82) ~[druid-core-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.common.task.batch.parallel.TaskMonitor.submitTask(TaskMonitor.java:320) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.common.task.batch.parallel.TaskMonitor.submit(TaskMonitor.java:254) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexPhaseRunner.submitNewTask(ParallelIndexPhaseRunner.java:246) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexPhaseRunner.run(ParallelIndexPhaseRunner.java:135) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runNextPhase(ParallelIndexSupervisorTask.java:314) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runSinglePhaseParallel(ParallelIndexSupervisorTask.java:612) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runTask(ParallelIndexSupervisorTask.java:513) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) ~[druid-indexing-service-25.0.0.jar:25.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_221]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_221]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_221]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_221]
Caused by: org.apache.druid.rpc.HttpResponseException: Server error [400 Bad Request]; body: {"error":"Task[single_phase_sub_task_custom_lists_hdihfjom_2024-04-15T16:17:19.655Z] already exists!"}
at org.apache.druid.rpc.ServiceClientImpl$1.onSuccess(ServiceClientImpl.java:201) ~[druid-server-25.0.0.jar:25.0.0]
at org.apache.druid.rpc.ServiceClientImpl$1.onSuccess(ServiceClientImpl.java:183) ~[druid-server-25.0.0.jar:25.0.0]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1181) ~[guava-16.0.1.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_221]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_221]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_221]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_221]
... 3 more
Vadim Sohin
04/16/2024, 11:30 AMSiva
04/17/2024, 3:06 PMAmit Gold
04/18/2024, 7:15 AMred-analytics
and it worked fine. To complete some investigation, I created a new source using “Load Data -> Paste Data” option. After using it for a join
on the :8888
port “query” tab, I went to delete it in the Datasources tab using “mark as unused all segments -> delete unused segments” and it is no longer in that tab. However I still see it in the :80
port in the data cubes list…
And even worse, now the red-analytics
cube keeps renaming itself to red-analytics1
and back every couple minutes??? How do I revert all this 🫠Jason Witkowski
04/18/2024, 9:23 PMreport.json
is not generated in the expected directory attemptDir
with the status.json
, instead it is being generated in /opt/druid/var/druid/task/attempt/1/report.json
. Overlord jobs therefore do not see it as they look in a directory like /opt/druid/var/druid/task/query-cf1408bf-e244-4dc7-8f28-6c1ac302a360/attempt/1/report.json
. Has anyone seen this? Is this a bug? Am I doing something that is altering the path druid writes report.json
to?JRob
04/19/2024, 5:52 PMWITH constants AS (SELECT '2024-02-15' AS start_date, '2024-02-19' AS end_date)
SELECT ...
FROM datasource CROSS JOIN constants
WHERE __time >= start_date AND __time < end_date
The error I get is:
> requireTimeCondition is enabled, all queries must include a filter condition on the __time column
Aside, I didn't need to use CROSS JOIN until Druid 28.0.0. The original query used a comma list like:
FROM datasource, constants
amir kalantar
04/21/2024, 11:01 AM1) No implementation for org.apache.hadoop.conf.Configuration annotated with @org.apache.druid.guice.Hdfs() was bound.
at com.google.inject.Guice.createInjector(Guice.java:62)
at org.apache.druid.initialization.ExtensionInjectorBuilder.build(ExtensionInjectorBuilder.java:49)
while locating org.apache.hadoop.conf.Configuration annotated with @org.apache.druid.guice.Hdfs()
at org.apache.druid.initialization.ServerInjectorBuilder.build(ServerInjectorBuilder.java:118)
at org.apache.druid.initialization.ServerInjectorBuilder.makeServerInjector(ServerInjectorBuilder.java:73)
at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:85)
... 2 more
for the 1st parameter of org.apache.druid.storage.hdfs.HdfsDataSegmentKiller.<init>(HdfsDataSegmentKiller.java:52)
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: namenode
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:475)
at org.apache.druid.storage.hdfs.HdfsStorageDruidModule.configure(HdfsStorageDruidModule.java:84) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.storage.hdfs.HdfsStorageDruidModule)
at org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:134)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:374)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:308)
at org.apache.hadoop.hdfs.DistributedFileSystem.initDFSClient(DistributedFileSystem.java:204)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:189)
2) No implementation for org.apache.hadoop.conf.Configuration annotated with @org.apache.druid.guice.Hdfs() was bound.
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
while locating org.apache.hadoop.conf.Configuration annotated with @org.apache.druid.guice.Hdfs()
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3712)
for the 2nd parameter of org.apache.druid.storage.hdfs.HdfsDataSegmentPusher.<init>(HdfsDataSegmentPusher.java:68)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3663)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290)
at org.apache.druid.storage.hdfs.HdfsStorageDruidModule.configure(HdfsStorageDruidModule.java:96)
at org.apache.druid.storage.hdfs.HdfsStorageDruidModule.configure(HdfsStorageDruidModule.java:83) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.storage.hdfs.HdfsStorageDruidModule)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
at com.google.inject.spi.Elements.getElements(Elements.java:110)
at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:198)
at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
3) No implementation for org.apache.hadoop.conf.Configuration annotated with @org.apache.druid.guice.Hdfs() was bound.
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
at com.google.inject.spi.Elements.getElements(Elements.java:110)
while locating org.apache.hadoop.conf.Configuration annotated with @org.apache.druid.guice.Hdfs()
at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
... 9 more
for the 1st parameter of org.apache.druid.storage.hdfs.HdfsFileTimestampVersionFinder.<init>(HdfsFileTimestampVersionFinder.java:42)
Caused by: java.net.UnknownHostException: namenode
at org.apache.druid.storage.hdfs.HdfsStorageDruidModule.configure(HdfsStorageDruidModule.java:79) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.storage.hdfs.HdfsStorageDruidModule)
... 30 more
4) An exception was caught and reported. Message: java.net.UnknownHostException: namenode
at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:198)
4 errors
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107)
at com.google.inject.Guice.createInjector(Guice.java:99)
at com.google.inject.Guice.createInjector(Guice.java:73)
at com.google.inject.Guice.createInjector(Guice.java:62)
at org.apache.druid.initialization.ExtensionInjectorBuilder.build(ExtensionInjectorBuilder.java:49)
at org.apache.druid.initialization.ServerInjectorBuilder.build(ServerInjectorBuilder.java:118)
at org.apache.druid.initialization.ServerInjectorBuilder.makeServerInjector(ServerInjectorBuilder.java:73)
at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:85)
... 2 more
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: namenode
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:475)
at org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:134)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:374)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:308)
at org.apache.hadoop.hdfs.DistributedFileSystem.initDFSClient(DistributedFileSystem.java:204)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:189)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3712)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3663)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290)
at org.apache.druid.storage.hdfs.HdfsStorageDruidModule.configure(HdfsStorageDruidModule.java:96)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
at com.google.inject.spi.Elements.getElements(Elements.java:110)
at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:198)
at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
at com.google.inject.spi.Elements.getElements(Elements.java:110)
at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
... 9 more
Caused by: java.net.UnknownHostException: namenode
... 30 more
Slack ConversationMohit Kumar
04/22/2024, 6:08 AMDruidGroupByQuery.DruidGroupByQueryBuilder builder = getAQIAggregation();
List<OrderByColumnSpec> orderByColumnSpecs = Collections.singletonList(
new OrderByColumnSpecMap("deviceID", false, null)
);
DefaultLimitSpec limitSpec=new DefaultLimitSpec(pageSize,orderByColumnSpecs);
String dataSource = "test-dataset";
DruidGroupByQuery query = builder
.dimensions(Collections.singletonList(new SimpleDimension("deviceID")))
.dataSource(dataSource)
.intervals(Collections.singletonList(interval))
.filter(inFilter)
.limitSpec(limitSpec)
.granularity(aggregationGranularity) // Apply the aggregation granularity
.build();
// Execute the query using Druid HTTP client
how can I achieve the offset thing in the following codeUday Singh Matta
04/22/2024, 8:26 AM"SELECT __time as \"timestamp\", {parameters} from \"{dataSource}\" WHERE \"__time\" > '{startDate}' and \"__time\" <= '{endDate}' and \"deviceID\" IN ({deviceID})"
Noor
04/22/2024, 10:07 AMJay G
04/22/2024, 3:48 PMDavid Adams
04/23/2024, 12:44 AMinconsistent metadata state
error:
org.apache.druid.java.util.common.ISE: Failed to publish segments because of [java.lang.RuntimeException: Inconsistent metadata state. This can happen if you update input topic in a spec without changing the supervisor name.
I'll include the state in a comment to this thread to avoid clutter, but essentially....we aren't changing anything. The supervisor name hasn't been modified. I know V29 had a change that was intended to help alleviate race conditions that caused this error to appear, but it's still happening. Any thoughts on what data we could be collecting to learn more about the problem? I've hit a dead end.
Additional info: We use an AWS Serverless Aurora MySQL instance for our database. Our ingestion has a replication of 2.Youngsol Koh
04/23/2024, 3:51 AMLancCJ chen
04/23/2024, 9:38 AMSumedh
04/23/2024, 1:02 PMMaytas Monsereenusorn
04/24/2024, 6:18 PMPrabir Choudhury
04/25/2024, 5:37 AMAman Kothari
04/25/2024, 12:35 PM0.18.1
Trying to enable logging so I can see what people are querying on Druid.
Made the below code change in */druid-0.18.1/conf/druid/cluster/_common/*common.runtime.properties
file.
druid.request.logging.type=file
druid.request.logging.dir=/goshposh/log
druid.request.logging.durationToRetain=P3D
And I started seeing the logs.
But then I wanted to filter the logs to avoid seeing SegmentMetadata and hence added the below line.
*druid.request.logging.mutedQueryTypes*=["segmentMetadata"]
But don’t see it getting filtered. Any idea why?
Also I eventually wanted to push it to S3. Any idea how?Julian Reyes
04/25/2024, 4:00 PMprometheus-emitter
in the load list)
druid_emitter: "composing"
druid_emitter_composing_emitters: '["statsd","prometheus"]'
druid_emitter_prometheus_strategy: "exporter"
I can see metrics in DataDog through statsd
however I cant see them in prometheusMahesha Subrahamanya
04/25/2024, 9:58 PMPriya Dhingra
04/26/2024, 1:03 AM