Frank Cao
05/30/2023, 5:43 AMjp
05/30/2023, 6:26 AMdu
command, I notice that the total size is growing by a few megabytes even though the maximum number of rows has not been reached. what could be the reason for this behavior?Harsh Upparwal
05/30/2023, 12:07 PMHarsh Upparwal
05/30/2023, 12:08 PMDuc Ạnh Nguyen
05/30/2023, 11:46 PMSrinivas Narava
05/31/2023, 3:28 AMYang Dark
05/31/2023, 6:29 AMjp
05/31/2023, 7:27 AMPrabir Choudhury
05/31/2023, 7:51 AMdruid-kubernetes-overlord-extensions
/ druid-kubernetes-extensions
to support MM less and Zookeeper less setup and also we want to enable hot and cold tier for historicals. Will helm-chart support this or we have to go with druid-operator?Gururaj K.P
05/31/2023, 6:33 PMJRob
05/31/2023, 6:54 PMbroadcastForever
. However, I just noticed today in the Druid documentation it states not to use this in a production environment:
Druid extensions use broadcast rules to load segment data onto all brokers in the cluster. Apply broadcast rules in a test environment, not in production.What is the reason for this? Should I be using a
loadForever
rule instead? If so, should I specify a number of replicas equal to the number of historical nodes I have to ensure each node has a copy of the data?Ashok Kumar Ragupathi
06/01/2023, 6:45 AMSrinivas Narava
06/01/2023, 10:48 AMRoger Gabarit
06/01/2023, 10:56 AMJRob
06/01/2023, 2:00 PMstring_to_array
string_to_array(\"test\", '|')
^ does not work. Undocumented but str2 is a regular expression. So I tried:
string_to_array(\"test\", '\\|')
^ also does not work. It's unclear to me why. So I started messing around and finally settled on:
string_to_array(\"test\", '\\\\|')
So the delimiter is a regex and I think it goes through multiple translations:
1. Druid will read the delimiters to convert this string to:
a. string_to_array("test", '\\|')
2. What isn't clear is it seems the regex library goes through ANOTHER interpolation so \\|
becomes \|
3. Finally, the regex parser will know that you want an explicit |
rather than the OR operator.
What I don't get is why the delimiter is going through multiple stages of interpolation. Is this intended?Ashok Kumar Ragupathi
06/01/2023, 2:11 PMAshok Kumar Ragupathi
06/01/2023, 2:12 PMJosh Sylvestre
06/01/2023, 5:24 PManoop kumar
06/02/2023, 10:59 AMPaddy Green
06/02/2023, 2:44 PMAVG(value) FILTER(WHERE condition)
AVG(CASE WHEN condition THEN value END)
JRob
06/02/2023, 3:08 PMSELECT COUNT(*)
FROM (
SELECT 1
FROM "datasource"
WHERE __time >= '2023-05-01' AND __time < '2023-06-01'
GROUP BY "Field1"
)
Took 1.16s
SELECT APPROX_COUNT_DISTINCT("Field1")
FROM "datasource"
WHERE __time >= '2023-05-01' AND __time < '2023-06-01'
Took 18.41s
Srinivas Narava
06/02/2023, 5:17 PMSrinivas Narava
06/02/2023, 5:40 PMSrinivas Narava
06/02/2023, 5:50 PManoop kumar
06/06/2023, 6:29 AMSiddharth Shirish Nahar
06/06/2023, 8:09 AM2023-06-06T07:15:55,730 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[PERF_NEO_CHETAK_ADS_DCC_PROD_2023-06-04T21:00:00.000Z_2023-06-04T21:30:00.000Z_2023-06-06T07:15:29.653Z_11] at existing path[/m3data-druid-prod-rt-be/segments/fk-m3data-druid-prod-rt-be-middlemanager-13:8102/fk-m3data-druid-prod-rt-be-middlemanager-13:8102_indexer-executor__default_tier_2023-06-06T07:15:46.011Z_b12fe58e484845b6af8f958f862605ee0]
2023-06-06T07:15:56,157 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[PERF_NEO_CHETAK_ADS_DCC_PROD_2023-06-04T21:30:00.000Z_2023-06-04T22:00:00.000Z_2023-06-06T07:15:29.781Z_11] at existing path[/m3data-druid-prod-rt-be/segments/fk-m3data-druid-prod-rt-be-middlemanager-13:8102/fk-m3data-druid-prod-rt-be-middlemanager-13:8102_indexer-executor__default_tier_2023-06-06T07:15:46.011Z_b12fe58e484845b6af8f958f862605ee0]
2023-06-06T07:15:56,486 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[PERF_NEO_CHETAK_ADS_DCC_PROD_2023-06-04T22:00:00.000Z_2023-06-04T22:30:00.000Z_2023-06-06T07:15:30.053Z_11] at existing path[/m3data-druid-prod-rt-be/segments/fk-m3data-druid-prod-rt-be-middlemanager-13:8102/fk-m3data-druid-prod-rt-be-middlemanager-13:8102_indexer-executor__default_tier_2023-06-06T07:15:46.011Z_b12fe58e484845b6af8f958f862605ee0]
2023-06-06T07:15:56,806 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[PERF_NEO_CHETAK_ADS_DCC_PROD_2023-06-04T22:30:00.000Z_2023-06-04T23:00:00.000Z_2023-06-06T07:15:30.272Z_11] at existing path[/m3data-druid-prod-rt-be/segments/fk-m3data-druid-prod-rt-be-middlemanager-13:8102/fk-m3data-druid-prod-rt-be-middlemanager-13:8102_indexer-executor__default_tier_2023-06-06T07:15:46.011Z_b12fe58e484845b6af8f958f862605ee0]
2023-06-06T07:15:57,122 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[PERF_NEO_CHETAK_ADS_DCC_PROD_2023-06-04T23:00:00.000Z_2023-06-04T23:30:00.000Z_2023-06-06T07:15:30.515Z_11] at existing path[/m3data-druid-prod-rt-be/segments/fk-m3data-druid-prod-rt-be-middlemanager-13:8102/fk-m3data-druid-prod-rt-be-middlemanager-13:8102_indexer-executor__default_tier_2023-06-06T07:15:46.011Z_b12fe58e484845b6af8f958f862605ee0]
2023-06-06T07:15:57,499 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[PERF_NEO_CHETAK_ADS_DCC_PROD_2023-06-04T23:30:00.000Z_2023-06-05T00:00:00.000Z_2023-06-06T07:15:30.679Z_11] at existing path[/m3data-druid-prod-rt-be/segments/fk-m3data-druid-prod-rt-be-middlemanager-13:8102/fk-m3data-druid-prod-rt-be-middlemanager-13:8102_indexer-executor__default_tier_2023-06-06T07:15:46.011Z_b12fe58e484845b6af8f958f862605ee0]
2023-06-06T07:16:44,704 ERROR [MonitorScheduler-0] org.apache.druid.java.util.metrics.cgroups.CpuAcct - Unable to fetch snapshot
<http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>: Hierarchy for [cpuacct] not found
at org.apache.druid.java.util.metrics.cgroups.ProcCgroupDiscoverer.getCgroupEntry(ProcCgroupDiscoverer.java:93) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.metrics.cgroups.ProcCgroupDiscoverer.discover(ProcCgroupDiscoverer.java:63) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.metrics.cgroups.ProcSelfCgroupDiscoverer.discover(ProcSelfCgroupDiscoverer.java:37) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.metrics.cgroups.CpuAcct.snapshot(CpuAcct.java:77) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.metrics.CpuAcctDeltaMonitor.doMonitor(CpuAcctDeltaMonitor.java:75) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.metrics.AbstractMonitor.monitor(AbstractMonitor.java:48) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.metrics.BasicMonitorScheduler.lambda$startMonitor$0(BasicMonitorScheduler.java:56) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$4.run(ScheduledExecutors.java:163) [druid-core-0.22.1.jar:0.22.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_172]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
2023-06-06T07:16:44,709 INFO [MonitorScheduler-0] org.apache.druid.java.util.metrics.CpuAcctDeltaMonitor - Detected first run, storing result for next run
2023-06-06T07:30:57,590 WARN [HttpClient-Netty-Worker-47] org.apache.druid.java.util.http.client.pool.ResourcePool - Resource at key[<http://fk-m3data-druid-prod-rt-be-master-2:8090>] was returned multiple times?
2023-06-06T07:30:57,591 WARN [task-runner-0-priority-0] org.apache.druid.discovery.DruidLeaderClient - Request[<http://fk-m3data-druid-prod-rt-be-master-2:8090/druid/indexer/v1/action>] failed.
org.jboss.netty.handler.timeout.ReadTimeoutException: null
at org.jboss.netty.handler.timeout.ReadTimeoutHandler.<clinit>(ReadTimeoutHandler.java:84) ~[netty-3.10.6.Final.jar:?]
at org.apache.druid.java.util.http.client.NettyHttpClient.go(NettyHttpClient.java:172) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.http.client.AbstractHttpClient.go(AbstractHttpClient.java:33) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.discovery.DruidLeaderClient.go(DruidLeaderClient.java:143) ~[druid-server-0.22.1.jar:0.22.1]
at org.apache.druid.discovery.DruidLeaderClient.go(DruidLeaderClient.java:127) ~[druid-server-0.22.1.jar:0.22.1]
at org.apache.druid.query.lookup.LookupReferencesManager.fetchLookupsForTier(LookupReferencesManager.java:576) ~[druid-server-0.22.1.jar:0.22.1]
at org.apache.druid.query.lookup.LookupReferencesManager.tryGetLookupListFromCoordinator(LookupReferencesManager.java:429) ~[druid-server-0.22.1.jar:0.22.1]
at org.apache.druid.query.lookup.LookupReferencesManager.lambda$getLookupListFromCoordinator$4(LookupReferencesManager.java:407) ~[druid-server-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.query.lookup.LookupReferencesManager.getLookupListFromCoordinator(LookupReferencesManager.java:397) ~[druid-server-0.22.1.jar:0.22.1]
at org.apache.druid.query.lookup.LookupReferencesManager.getLookupsList(LookupReferencesManager.java:374) ~[druid-server-0.22.1.jar:0.22.1]
at org.apache.druid.query.lookup.LookupReferencesManager.loadAllLookupsAndInitStateRef(LookupReferencesManager.java:357) ~[druid-server-0.22.1.jar:0.22.1]
at org.apache.druid.query.lookup.LookupReferencesManager.start(LookupReferencesManager.java:157) ~[druid-server-0.22.1.jar:0.22.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_172]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_172]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_172]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172]
at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:143) ~[druid-core-0.22.1.jar:0.22.1]
at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:115) ~[druid-services-0.22.1.jar:0.22.1]
at org.apache.druid.cli.CliPeon.run(CliPeon.java:304) ~[druid-services-0.22.1.jar:0.22.1
I am trying to ingest data in druid in kafka-based ingestion but I am seeing this error. From what I am understanding, Segments are generated but some timeout configuration is wrongly configured. Can someone share some insights what can be the issue ?Ashok Kumar Ragupathi
06/06/2023, 2:59 PMjp
06/07/2023, 5:42 AMSrinivas Narava
06/07/2023, 9:37 AMZibo Gong
06/07/2023, 6:37 PM