Eric Chin
06/08/2023, 7:56 AMkafka-emitter
like shown below:
druid.request.logging.type=emitter
druid.request.logging.feed=druid-requests
druid.emitter=kafka
druid.emitter.kafka.bootstrap.servers=kafka:9092
druid.emitter.kafka.request.topic=druid-request-logs
It works, but I noticed that after running queries it gives me all logs tracing up from the router to the historical. Is there a way to filter the logs just to see those with sql queries? I tried using the filtered
logging type with the emitter
as the delegated logging type. I was able to filter based off the queryTimeThresholdMs
, but is there a way to filter based off whether the log contains the actual query? Thank you.Kai Sun
06/08/2023, 11:47 PMquery/segment/time
is closely tied to the spikes of query time. In fact, by adding more historical servers we reduced the query time. Mostly likely it is that the segments files are competing for the memory space (to be mapped) in the Historicals. So here, I have the following questions:
1/ What is the strategy that the historical maps the segment file to the process memory? Is there a limit of how many segment files? (Thus, it would not fully use all the memory space of the java process) Or the process would just map as many segments files as possible if it has memory spaces?
2/ Is there a way to examine what segment files are mapped to the process? Any logs or statistics?jp
06/12/2023, 5:52 AMDuc Ạnh Nguyen
06/12/2023, 8:46 AMjakubmatyszewski
06/12/2023, 10:52 AMprometheus-emitter
extension. I used to use druid-exporter which conveniently expanded metrics with suffix describing units as per prometheus recommendation.
However with prometheus-emitter
it seems to me like there is no straight-forward way to do this (expand metrics names with unit suffix), and I end up with default names. Am I missing something or is that in fact how this extension operates at the moment?Oleg Yamin
06/12/2023, 8:42 PMbin/run-druid router
but you cant check status or like you can with
bin/broker.sh status
and it doesnt start in background automaticalyjp
06/13/2023, 12:47 AMjp
06/13/2023, 5:40 AMSrinivas Narava
06/13/2023, 12:10 PMFATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.ClassCastException: org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat cannot be cast to org.apache.hadoop.mapreduce.OutputFormat
cactus
06/13/2023, 3:45 PMMargo Ratte
06/13/2023, 3:52 PMCaner Kürtür
06/14/2023, 1:05 PMReena Leone
06/14/2023, 3:54 PMSrinivas Narava
06/16/2023, 4:55 AM# Schedule the metadata management store task for every hour:
druid.coordinator.period.metadataStoreManagementPeriod=P1H
# Set a kill task to poll every day to delete Segment records and segments
# in deep storage > 4 days old. When druid.coordinator.kill.on is set to true,
# you can set killDataSourceWhitelist in the dynamic configuration to limit
# the datasources that can be killed.
# Required also for automated cleanup of rules and compaction configuration.
druid.coordinator.kill.on=true
druid.coordinator.kill.period=P1D
druid.coordinator.kill.durationToRetain=P4D
druid.coordinator.kill.maxSegments=1000
# Poll every day to delete audit records > 30 days old
druid.coordinator.kill.audit.on=true
druid.coordinator.kill.audit.period=P1D
druid.coordinator.kill.audit.durationToRetain=P30D
# Poll every day to delete supervisor records > 4 days old
druid.coordinator.kill.supervisor.on=true
druid.coordinator.kill.supervisor.period=P1D
druid.coordinator.kill.supervisor.durationToRetain=P4D
# Poll every day to delete rules records > 4 days old
druid.coordinator.kill.rule.on=true
druid.coordinator.kill.rule.period=P1D
druid.coordinator.kill.rule.durationToRetain=P4D
# Poll every day to delete compaction configuration records
druid.coordinator.kill.compaction.on=true
druid.coordinator.kill.compaction.period=P1D
# Poll every day to delete datasource records created by supervisors > 4 days old
druid.coordinator.kill.datasource.on=true
druid.coordinator.kill.datasource.period=P1D
druid.coordinator.kill.datasource.durationToRetain=P4D
killDataSourceWhitelist=Joon
Srinivas Narava
06/16/2023, 4:55 AMAnant Sharma
06/16/2023, 7:21 AMAnant Sharma
06/16/2023, 7:23 AMAnkit
06/18/2023, 12:07 PMdruid.service: broker-uat
still the same error. My questions are: 1) should I give druid/broker-uat or just broker-uat? 2) Router will not only look for broker but for also coordinator and other services, how can I override the default service names for all? 3) I think i would need to make this change for all the services. I mean for coordinator I need to give all services with -uat and similarly for MM and Historical and broker? ThanksViraj Raul
06/19/2023, 5:54 AMError from upstream druid server: Time ordering is not
supported for a Scan query with 51 segments per time chunk
and a row limit of 9,223,372 ,036,854,775,807. Try reducing
your query limit below maxRowsQueuedForOrdering (currently
100,000), or using compaction to reduce the number of
segments per time chunk, or raising
maxSegmentPartitionsOrderedinMemory (currently 50) above
the number of segments you have per time chunk.
can ayone please guideDuc Ạnh Nguyen
06/19/2023, 11:45 AMAshok Kumar Ragupathi
06/19/2023, 12:47 PMSoman Ullah
06/19/2023, 10:10 PMJonathan Du
06/19/2023, 11:53 PMSaurabh Pande
06/20/2023, 5:29 AMChandu
06/20/2023, 11:29 AMChandu
06/20/2023, 12:53 PMCaner Kürtür
06/20/2023, 3:56 PMSiddharth Gautam
06/20/2023, 6:30 PMJRob
06/20/2023, 8:31 PMChandu
06/21/2023, 4:38 AM