Hello, anyone else noticed druid.struct.ai hasn't been available for the past few days?

hi all. how do you specify for a data source that it should monitor a directory for new files and ingest them? thanks!

i added this bit to the data source's spec file, but it doesn't seem to work.

"spec": {
    "ioConfig": {
        "watcher": {
          "type": "file",
          "pollPeriod": "PT10M"
        }

Hi folks, please review this PR to upgrade pac4j extension: https://github.com/apache/druid/pull/18259 This major upgrade is needed to fix CVE-2023-52428 which requires upgrading nimbus-jose-jwt to

9.37.2

which is not compatible with pac4j v4.

Druid combined with an AI. Natural language queries are then possible with Druid.

Are there any plans around supporting histogram ingestion in Druid? Specifically, I’m referring to a way to ingest and store histograms as a native metric type, similar to Prometheus histogram. The idea is to push pre-aggregated histogram data into Druid, not compute them at query time. Thanks

Hello everyone

We are hiring Druid Developer from Asia or Africa. It is fully remote position.

Roles & Responsibilities At least 3 years of experience on Druid. · Experience in design and architecture of druid for large scale streaming data process systems. · Experience in setup and configuration of druid for production environments. · Experience in working with druid data modelling · Experience in performance tuning of druid configuration, query optimization · Experience in working with Kafka, Pulsar and druid for real time streaming data ingestion · Hands-on experience in monitor and troubleshooting of the issues Experience in working with cloud platforms and Kubernetes

Please respond me if anyone interested

Hello, this is about ingestion json parsing. We have a large Kafka topic which is split to multiple data sources on the Druid side, using filter transformation. This means there's compute overhead of parsing the same events across all supervisors (of course, in addition to network overhead). Before I go and benchmark it, does anyone know how likely is it to be a large overhead?

Hello, question about timeseries vs timeboundary query performance. I have a table containing emails and customer IDs. I want to get the oldest time for an email and the oldest time for the same email but filtered on a customer ID. My app will have to do 2 timeboundary minTime queries (in parallel) to get those results, they should be fast because they're using the metadata. Whereas I can get the 2 results in only one timeseries query, but it should be slower ? Am I correct ? Which strategy is the fastest ?

hello, can someone tell me how to create users and roles on druid for specific permissions

hello, can someone help me out to increase the query timeout from 1minute to xyz minutes tried below parameters druid_server_http_maxQueryTimeout=120000 druid_server_http_defaultQueryTimeout=120000 also tried adding connectionTimeout also still same issue

Hi all, Could someone help to review this fix PR: https://github.com/apache/druid/pull/18415 The handling of null objects in QueryCache is problematic.

Hi team. Wanted to understand how compactions and ingestions in compacted datasources are expected to be working. 1. I have a datasource with data from 2024. Granularity set to Day. I set up its auto compaction with granularity set to Month. Offset day set to 10. Even so on Aug 19 : The segments seem to be month granularity up till July and then Day granularity till Aug 19. Is this expected behaviour? Are these Day level segments going to be there till Aug end? 2. Now I run an ingestion for a time period July 25 - Aug 5. Firstly this ingestion breaks as OVERWRITE WHERE clause identified interval which is not aligned with

PARTITIONED BY granularity.

This I figure is because the date range is split between month level segments and day level segments. So I break the ingestion into 2 : Before the month level change and after the month level change. So I run an ingestion July 25 - July 31. This works but only with DAY granularity. So this makes me uncertain about whether or not the ingestion earlier was breaking because of the underlying segment granularity. 3. Now the ingestion for July 25 - July 31. creates 7 day level segments : But they are not getting compacted : Compaction is saying 100% compacted except for last 10 days. Not seeing these uncompacted segments. Shouldn't these segements be relevant for compaction? If anybody who understands compaction well, can help with this. Would be appreciated.

Hi team, I was looking to see. Is there any specific difference in processing times between doing a compaction job, where we change a data source from hour granularity to day granularity, this is for a hashed partitioned data set, in contrast to using an index job re-ingesting it from a druid dataSource as input to a new dataSource. For reference we are looking to test reducing from an hourly granularity to daily granularity as currently we are storing 560 segments per day for 26gb of data and the high count of segments appears to be significantly impacting the ability for caching to succeed and subsequently making queries slower. From testing re-ingesting a days data from a druid dataSource its taking much too long to do, i'm wondering if its different at all from doing it in a compaction job. We don't want to impact the current data in the table by running a compaction job so I'm limited in my capability to test this currently.

Hi Team, I am using Druid v25 and trying to connect with SSL enabled postgres for metadata storage. I have ca-cert.crt file and configured below configs to common.runtime.properties file

# Metadata storage configuration
druid.extensions.loadList=["postgresql-metadata-storage"]
druid.metadata.storage.type=postgresql
druid.metadata.storage.connector.connectURI=jdbc:<postgresql://your-db-host:5432/your-db-name>
druid.metadata.storage.connector.user=your-db-user
druid.metadata.storage.connector.password=your-db-password
 
# SSL-specific configuration
druid.metadata.postgres.ssl.useSSL=true
druid.metadata.postgres.ssl.sslMode=verify-full
druid.metadata.postgres.ssl.sslRootCert=/path/to/ca-cert.crt

It's not working. Errors are in screenshots. If I am trying to connect with ssl Disabled postgres then Druid is running (commented all ssl related configs)

Hello everybody, I'm looking for a bit of help with Druid SQL query stats. Is there any way to get the amount of data scanned by a SQL query? I couldn't find anything in the documentation.

Hello guys, we are thinking of building a real-time BI system that could be used by any of our employees. Currently we have a Power BI system that is linked directly to our production database, so any complex queries significantly increase the resource usage on the server, and we would like to separate the BI side from ops. We are thinking of using Apache Druid for that, but the only problem is that the db schema is very decoupled because it is auto-generated by our ERP and often changes, so it would be hard to use Debezium with CDC for our db (SQL Server) as source. Maybe someone faced such a problem and can recommend something? The architecture we currently have in mind is: SQL Server -> Airbyte -> dbt -> Apache Druid -> Metabase

Hello, i am learning about flattenSpec and it looks like it supports most of the JSONPath features such as applying filters, and maps etc, but i found that JSON_QUERY functions support only limited set of JSONPath features. Is that intentional(may be to not effect query performance) or is there a plan to support that in future?

Can anyone help me understand difference between query/time metrics in broker/historical and sqlQuery/time log. https://druid.apache.org/docs/latest/operations/request-logging#:~:text=The%20following%20shows%20an%20example%20log%20emitter%20output%3A

I am trying to get this working with mysql but am getting stuck with: org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLException: Cannot create PoolableConnectionFactory (Unknown system variable 'query_cache_size'). We are using mysql 8.0.42, druid 34, and mysqlconnector 9.4

I can't find anything documented on how to set the mysql version

Hi, I have a bunch of files (10000+, 50 MB each) that my system writes into S3 every minute. What would be the recommendation to have near real time ingestion for this ? I can't use Kafka because my files are big. Should I go with batch ingestion every 5 minutes or so ?

Hi everyone, I’m currently working on enabling basic authentication for a Druid instance running within a Kubernetes cluster. Secrets are being generated and managed via HashiCorp Vault. The initial user creation process is functioning correctly, but I’m encountering challenges with secret rotation. Specifically, I’m looking for a way to rotate the admin user credentials other than using the API, as I no longer have access to the previous secret once it’s regenerated. Would anyone be able to share insights or suggestions on how to approach this? Any help would be greatly appreciated. Thank you in advance! Best regards

Hi Team, Is there a feature in Druid Supervisor to allow pre-ingestion kafka header-based filtering ? Druid's

transformSpec

filtering operates after row deserialization and data ingestion and the above feature can help in saving a lot of computation.

Is there going to be a 2025 Druid summit???

Hi Team, Issue: Data Length Mismatch During S3 Ingestion in Apache Druid

Overview

We are facing an issue while ingesting a large dataset from S3 into Apache Druid. The ingestion process fails during the segment building phase with a data length mismatch error.

Error Message

java.lang.IllegalStateException: java.io.IOException: com.amazonaws.SdkClientException: Data read has a different length than the expected: dataLength=9404416; expectedLength=1020242891; includeSkipped=true; in.getClass()=class com.amazonaws.services.s3.AmazonS3Client$2; markedSupported=false; marked=0; resetSinceLastMarked=false; markCount=0; resetCount=0
at org.apache.commons.io.LineIterator.hasNext(LineIterator.java:108)
at org.apache.druid.data.input.TextReader$1.hasNext(TextReader.java:73)
at org.apache.druid.data.input.IntermediateRowParsingReader$1.hasNext(IntermediateRowParsingReader.java:60)
at org.apache.druid.java.util.common.parsers.CloseableIterator$2.findNextIteratorIfNecessary(CloseableIterator.java:74)
at org.apache.druid.java.util.common.parsers.CloseableIterator$2.next(CloseableIterator.java:108)
at org.apache.druid.java.util.common.parsers.CloseableIterator$1.next(CloseableIterator.java:52)
at org.apache.druid.indexing.common.task.FilteringCloseableInputRowIterator.hasNext(FilteringCloseableInputRowIterator.java:68)
at org.apache.druid.data.input.HandlingInputRowIterator.hasNext(HandlingInputRowIterator.java:63)
at org.apache.druid.indexing.common.task.InputSourceProcessor.process(InputSourceProcessor.java:95)
at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:891)
at org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:500)

Context

• The error occurs while ingesting a large JSON file from S3. • Data read has a length of 9,404,416 bytes, while the expected length is 1,020,242,891 bytes. • The error happens in the BUILD_SEGMENTS phase. • The same large dataset is ingested successfully from our local Druid setup without any issues. • Other datasets of smaller sizes are being ingested successfully.

Questions / Request for Support

We are looking for guidance and support on the following points: 1. Is this a known issue when ingesting large files from S3 into Druid? 2. Are there recommended configurations or best practices to handle such issues? 3. Should we consider splitting files, adjusting timeouts, or configuring retries to better handle large file ingestion? 4. Are there troubleshooting steps, patches, or workarounds that can help resolve this problem?

Additional Information

• Druid version, ingestion spec, and sample files can be provided upon request. • We are happy to share more logs and configuration details as needed. Thank you for your support!

Hi Team, Please review this PR to add the functionality of kafka header based filtering. Thanks