Ashi Bhardwaj
01/05/2025, 9:38 AMMaytas Monsereenusorn
01/07/2025, 8:38 PMMaytas Monsereenusorn
01/18/2025, 1:18 AMHazmi
01/20/2025, 10:03 AMSuraj Goel
01/27/2025, 5:10 PMSuraj Goel
01/28/2025, 7:02 AMinfo Advisionary
01/30/2025, 11:08 AMMaytas Monsereenusorn
01/31/2025, 6:05 AMSuraj Goel
02/13/2025, 10:13 AMAshwin Tumma
02/21/2025, 1:10 AMJamie Chapman-Brown
02/27/2025, 6:23 PMdruid_emitted_metrics{metric_name="ingest-rabbit-lag"}
. We've recently switched to using the Prometheus plugin, but I can't find any rabbit ingest stats.
Am I missing anything? Can anyone point me to what I would need to change to get these stats back?Mikhail Sviatahorau
03/05/2025, 3:51 PMcoordinator-issued
prefix, which helps identify auto-compaction and could be used to check and reject publishing segments that don’t match the current cluster state. The problem is that this prefix isn’t accessible on the level where segments are being chosen. The options are to add some preprocessing of the input source at generateAndPublishSegments in IndexTask or to send some flag to the InputSourceProcessor.prosess() method, but both feel like a last resort.
Curious to hear your thoughts — if not there, where do you think this could be best handled?Maytas Monsereenusorn
03/24/2025, 7:07 PMMaytas Monsereenusorn
04/16/2025, 9:38 PMMaytas Monsereenusorn
04/23/2025, 7:36 PMAbhishek Balaji Radhakrishnan
05/07/2025, 12:31 AMjson_merge()
function when someone gets a chance: https://github.com/apache/druid/pull/17983. Thanks!Abhishek Balaji Radhakrishnan
05/12/2025, 9:09 PMAbhishek Balaji Radhakrishnan
05/23/2025, 1:42 AMMaytas Monsereenusorn
05/28/2025, 2:03 AMint toAdjust = 0
if (violatesPeriodThreshold) {
toAdjust += adjustment;
}
if (violatesDurationThreshold) {
toAdjust += adjustment;
}
if (violatesSegmentThreshold) {
toAdjust += adjustment;
}
if (violatesSegmentRangeThreshold) {
toAdjust += adjustment;
}
if (toAdjust != 0) {
final int adjustedPriority = theQuery.context().getPriority() - toAdjust;
return Optional.of(adjustedPriority);
}
• What if we can set the adjustment value for each Threshold seperately?
i.e.
int toAdjust = 0
if (violatesPeriodThreshold) {
toAdjust += periodThresholdAdjustment;
}
if (violatesDurationThreshold) {
toAdjust += durationThresholdAdjustment;
}
if (violatesSegmentThreshold) {
toAdjust += segmentThresholdAdjustment;
}
if (violatesSegmentRangeThreshold) {
toAdjust += segmentRangeThresholdAdjustment;
}
if (toAdjust != 0) {
final int adjustedPriority = theQuery.context().getPriority() - toAdjust;
return Optional.of(adjustedPriority);
}
The motivation for the first change is that if a query that violate N thresholds, it should be penalize more (not equal) to another query that violate N-1 thresholds.
The motivation for the second change is that some violate are worst than other. i.e. periodThreshold is not that bad compare to segmentRangeThreshold.
The prioritization value would then carry over to the Historical and can help with resources prioritization on Historical processing threadpool (related to this discussion https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1745436989786489). CC:@Gian Merlino @Clint WylieAbhishek Balaji Radhakrishnan
05/28/2025, 3:02 AMSoman Ullah
05/28/2025, 7:15 PMJesse Tuglu
06/03/2025, 11:11 PMDataSourceMetadata
are solely supervisor-based (either materialized view or seekable stream). ObjectMetadata
seems to only be used in tests. The way I see it there are ≥ 2 options:
• Commit a datasource metadata row per supervisor (likely the easiest, but will take some re-workings on the SegmentTransactionalInsertAction
API and others, who assume these rows are keyed by datasource
) – I'm currently doing this and it seems to work fine.
• Commit a single row per datasource, storing partitions per supervisor ID and doing merges in the plus
minus
methods
◦ Something like the payload being:
▪︎ map[supervisor_id] = SeekableStreamSequenceNumbers
◦ This might suffer from write contention since N supervisors * M tasks per supervisor will be attempting to write new updates in the commit payload to this row in the DB.Allen Madsen
06/10/2025, 9:39 PMJesse Tuglu
06/13/2025, 1:46 AMJesse Tuglu
06/17/2025, 7:31 PM[INFO] ------------------< org.apache.druid:druid-quidem-ut >------------------
[INFO] Building druid-quidem-ut 34.0.0-SNAPSHOT [80/80]
[INFO] from quidem-ut/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-multi-stage-query/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-datasketches/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-orc-extensions/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-parquet-extensions/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-avro-extensions/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-protobuf-extensions/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-s3-extensions/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-kinesis-indexing-service/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-azure-extensions/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-google-extensions/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-hdfs-storage/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-histogram/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/druid-aws-common/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/druid-processing/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/druid-sql/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/druid-indexing-service/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/druid-indexing-hadoop/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/mysql-metadata-storage/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-kafka-indexing-service/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-basic-security/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-lookups-cached-global/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/druid-testing-tools/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/extensions/simple-client-sslcontext/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/druid-services/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/druid-server/34.0.0-SNAPSHOT/maven-metadata.xml>
Downloading from datasets: <https://raw.githubusercontent.com/kgyrtkirk/datasets/repo/org/apache/druid/druid-gcp-common/34.0.0-SNAPSHOT/maven-metadata.xml>
Wondering if you knew about this, and whether this was intentional
cc @Gian MerlinoSoman Ullah
07/08/2025, 9:52 PMCristian Daniel Gelvis Bermudez
07/09/2025, 4:00 PMJesse Tuglu
07/15/2025, 1:53 AMListFilteredDimensionSpec
not allowed to have null
as an element in its values
array? Or is this a bug?
Example:
{
"dimensions": [
{
"type": "listFiltered",
"delegate": "based_on",
"values": [
null,
"A",
"B"
]
}
],
"aggregations": [
{
"type": "doubleSum",
"fieldName": "value",
"name": "value"
}
],
"intervals": [
"2025-01-01T00:00:00.000Z/2025-07-10T23:59:59.999Z"
],
"queryType": "groupBy",
"granularity": "all",
"dataSource": "datasource_A"
}
will fail with
Error: RUNTIME_FAILURE (OPERATOR)
Cannot invoke "String.getBytes(String)" because "string" is null
java.lang.NullPointerException
The line in question is this.
Passing in empty string in-place of the NULL
returns null values, so this is a partial work-around for now.Jesse Tuglu
07/17/2025, 6:20 PMNULL
into the segment when druid.generic.useDefaultValueForNull=true
. Is this the expected behavior?
See the data loader photo attached, where it appears to show parsing row 3's string_value
as an empty string. However, post-segment creation, I took a dump of the segment:
{"__time":1704070860000,"title":"example_1","string_value":"some_value","long_value":1,"double_value":0.1,"float_value":0.2,"multi_value":["a","b","c"],"count":1,"double_value_doubleSum":0.1,"float_value_floatSum":0.2,"long_value_longSum":1}
{"__time":1704070920000,"title":"example_2","string_value":"another_value","long_value":2,"double_value":0.2,"float_value":0.3,"multi_value":["d","e","f"],"count":1,"double_value_doubleSum":0.2,"float_value_floatSum":0.3,"long_value_longSum":2}
{"__time":1704070980000,"title":"example_3","string_value":null,"long_value":0,"double_value":0.0,"float_value":0.0,"multi_value":null,"count":1,"double_value_doubleSum":0.0,"float_value_floatSum":0.0,"long_value_longSum":0}
{"__time":1704071040000,"title":"example_4","string_value":null,"long_value":0,"double_value":0.0,"float_value":0.0,"multi_value":null,"count":1,"double_value_doubleSum":0.0,"float_value_floatSum":0.0,"long_value_longSum":0}
you can see that in row 3, the string_value
column has replaced what I'd expect to be an ""
with a null
. This is running on version v31, with druid.generic.useDefaultValueForNull=true
. I've tested that running on v33 with druid.generic.useDefaultValueForNull=false
produces the expected result (""
stored instead of null
).Chris Warren
07/24/2025, 4:23 PMRoleProvider
interface that is defined in the druid-basic-security extension, but if I try to make my extension a standalone extension (using the provided
scope for the druid-basic-security
dependency in my pom.xml
) I get Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/druid/security/basic/authorization/RoleProvider
... however if I shade the jar, then it does work. Also if I build it into druid-basic-security
that also works. Am I missing something with how the interface might be implemented outside of this core extension package? Anyone have experience with this?