Uday Singh Matta
04/11/2024, 4:35 AM{
"queryType": "groupBy",
"dataSource": "EnergyMeterST501",
"intervals": [
"2023-12-23T00:00+03:00/2024-03-23T23:59:59+03:00"
],
"granularity": {
"type": "period",
"period": "P1D",
"timeZone": "Asia/Riyadh"
},
"dimensions": [
"deviceID"
],
"metric": "count",
"aggregations": [
{
"type": "count",
"name": "count",
"fieldName": "deviceID"
}
],
"having": {
"type": "filter",
"filter": {
"type": "in",
"dimension": "deviceID",
"values": [
"2020070100"
]
}
}
}
Slack ConversationSwapnil Jokhakar
04/11/2024, 2:26 PMSELECT
dim_field_1,
LOOKUP(CONCAT(dim_field_1, ''), 'lookup_1') AS "lookup_1_val",
LOOKUP(CONCAT(dim_field_2, ''), 'lookup_2') AS "lookup_2_val",
LOOKUP(CONCAT(dim_field_2, ''), 'lookup_3') AS "lookup_3_val",
dim_field_2,
LOOKUP(CONCAT(dim_field_2, ''), 'lookup_4') AS "lookup_4_val",
(SUM(cost)) AS "cost"
FROM test_table
WHERE
dim_field_2 IN (2040)
AND __time >= '2024-03-01 00:00:00' AND __time <= '2024-03-31 23:59:59'
AND dim_field_1 IN ('1608826050', '1608826054', '1608826048', '1608818855', '1608818825', '1609091475', '1608818895', '1608818852', '1608818865', '1608818753', '1608818775', '1608818724', '1609027835', '1608273535', '1608916347', '1608268938', '1608818799', '1608818766', '1608818852', '1609569765', '1608273487', '1609347997', '1609477098', '1608826096', '1609559940', '1608916420')
GROUP BY
LOOKUP(CONCAT(dim_field_2, ''), 'lookup_3'),
dim_field_1,
LOOKUP(CONCAT(dim_field_1, ''), 'lookup_1'),
LOOKUP(CONCAT(dim_field_2, ''), 'lookup_3'),
LOOKUP(CONCAT(dim_field_2, ''), 'lookup_4'),
dim_field_2
In above query I have added all fields available in the select clause except the field on with aggregation function in group by clause in the different order then the field added to the select clause and in this case this query returns records with 1:M mapping between dim_field_2 field (key) and lookup_2 value, if I change the order of the group by and use the same order in which fields are added in the select clause then query returns records without including 1:M mapping. This seems to be an odd behavior. Could anyone please help here to understand the reason why the above query is returning records with 1:M mapping between dim_field_2 field (key) and lookup_2 value?
I have also tried to query the lookup_2 by adding the filter on value field in whare clause with the list of values which I am getting as 1:M mapping in the above query result and this query return result with 1:1 mapping only, this is an expected behavior as Druid Lookups can have 1:1 mapping between the key and value.Ashish Kumar
04/12/2024, 7:10 AMindex-parallel
& msqe
in two different tables.
The Total data size in the table which is ingested through index-parallel is ~ 190mb although the total data size in the table which is i*ngested using the msqe is ~166 mb.*
Do anyone knows why the data size is different in both the table, despite count of rows are same.Bharat
04/12/2024, 8:56 AMAmit Jain
04/12/2024, 10:30 AMPio Salvatore MORRONE (KEBULA)
04/12/2024, 1:25 PMPINNURI SAIKRISHNA PRASAD
04/13/2024, 5:11 AMIgor Berman
04/15/2024, 10:22 AMMax ZorN
04/15/2024, 11:39 AMsriramdas sivasai
04/16/2024, 6:31 AMNimrod Lahav
04/16/2024, 9:42 AM29.0.1
)
• https://hub.docker.com/r/apache/druid
• docker file: https://github.com/apache/druid/blob/master/distribution/docker/Dockerfile
and security scans showing 35 Critical
and 84 High
wanted to ask how did you manage to deploy this on an enterprise / prod env that wont allow critical findings, or is there any other image we can use?AR
04/16/2024, 3:15 PMKai Sun
04/17/2024, 12:03 AMdruid.processing.buffer.sizeBytes
my understanding of this config is the size of intermediate buffer or merge buffer size. The question is this, if we set a small value, say 10M, will it cause groupby or topN query failure?
The underlying reason is that each thread in the processing pool when processing the hydrants/segment, it would use one piece of this buffer. Thus a large configuration of threads say 100 with buffer size as 500M would use 50G. That is pretty huge, meaning one middle manager server may hold only one or two peons.AR
04/17/2024, 5:23 AMAR
04/17/2024, 5:32 AMJRob
04/17/2024, 3:35 PMOlivier JOUSSET
04/18/2024, 9:11 AMАлексей Ясинский
04/18/2024, 3:25 PMYoungsol Koh
04/19/2024, 7:42 AMFor better query performance, you will want to ensure a good (Hi, any suggestion for ratio to start with? which metrics should I look for? in order to tune? Segment Query time? Thank you./ total size of allfree system memory
) ratio so that a greater proportion of segments can be kept in memory.druid.segmentCache.locations
Parth Agrawal
04/19/2024, 9:15 AMNiraj Pal
04/19/2024, 6:48 PMOVERWRITE ALL
for a datasource. Simultaneously, I also have DAGs that trigger MSQE job with
OVERWRITE WHERE __time >= TIMESTAMP '{start_time}' AND __time < TIMESTAMP '{end_time}'
. Would this cause any issues on how druid handled multiple ingestion jobs on same datasources resulting in failures?Maria Taramigkou
04/19/2024, 7:58 PMDavid Glasser
04/19/2024, 9:58 PMIgor Berman
04/21/2024, 9:56 AMsqlQuery/time
?
e.g. something like : sqlQuery/time
= query/segments/count
* query/segment/time
+ sqlQuery/planningTimeMs
Mohit Kumar
04/22/2024, 6:08 AMDruidGroupByQuery.DruidGroupByQueryBuilder builder = getAQIAggregation();
List<OrderByColumnSpec> orderByColumnSpecs = Collections.singletonList(
new OrderByColumnSpecMap("deviceID", false, null)
);
DefaultLimitSpec limitSpec=new DefaultLimitSpec(pageSize,orderByColumnSpecs);
String dataSource = "test-dataset";
DruidGroupByQuery query = builder
.dimensions(Collections.singletonList(new SimpleDimension("deviceID")))
.dataSource(dataSource)
.intervals(Collections.singletonList(interval))
.filter(inFilter)
.limitSpec(limitSpec)
.granularity(aggregationGranularity) // Apply the aggregation granularity
.build();
// Execute the query using Druid HTTP client
how can I achieve the offset thing in the following codeAR
04/22/2024, 2:52 PMАлексей Ясинский
04/23/2024, 9:43 AMKiran Kumar Puttakota
04/23/2024, 11:34 AMJianshu Chi
04/25/2024, 6:30 PMsegmentMetadata
query. Any advises on what could be wrong, and which method/interface i'm missing? Thanks.JRob
04/26/2024, 5:19 PM