It keeps in around 15 20 days . When I restart server everything become normal

Please help

Hi We have a dev cluster run on Kubernetes (with many nodes of large EC2s) which is costing us a lot of money. We would like to run it on standalone cluster with few nodes. Original setup has s3 for deep storage and Postgres for metadata. We want to run the new cluster on completely new instances. I tried copying the segments data for old s3 to new s3 and also segments table in old postgres to the new one. Cluster is up but there is an error in coordinator "No available [_default_tier] servers or node capacity to assign primary segment". Any help is highly appreciated.

Hi, for https://druid.apache.org/docs/latest/development/extensions-core/kafka-extraction-namespace.html, is this param

connectTimeout

is in millis?

This message was deleted.

Hi All , Hope you all are well Compaction is Stuck on running , Cant se any logs in coordinator , ive applied the same setting in my DEV and Prod environment and its working fine ,But just for QC it's not working Compaction config

{
  "dataSource": "sensors_data",
  "taskPriority": 25,
  "inputSegmentSizeBytes": 100000000000000,
  "maxRowsPerSegment": null,
  "skipOffsetFromLatest": "P1D",
  "tuningConfig": {
    "maxRowsInMemory": 25000,
    "appendableIndexSpec": null,
    "maxBytesInMemory": null,
    "maxTotalRows": null,
    "splitHintSpec": null,
    "partitionsSpec": {
      "type": "dynamic",
      "maxRowsPerSegment": 5000000,
      "maxTotalRows": null
    },
    "indexSpec": null,
    "indexSpecForIntermediatePersists": null,
    "maxPendingPersists": null,
    "pushTimeout": null,
    "segmentWriteOutMediumFactory": null,
    "maxNumConcurrentSubTasks": null,
    "maxRetry": null,
    "taskStatusCheckPeriodMs": null,
    "chatHandlerTimeout": null,
    "chatHandlerNumRetries": null,
    "maxNumSegmentsToMerge": null,
    "totalNumMergeTasks": null,
    "maxColumnsToMerge": null,
    "type": "index_parallel",
    "forceGuaranteedRollup": false
  },
  "granularitySpec": null,
  "dimensionsSpec": null,
  "metricsSpec": null,
  "transformSpec": null,
  "ioConfig": null,
  "taskContext": null
}

For any help thanks in advance !!

This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.

Hi All, is there anyway to ingest data in druid using gcloud dataproc job submit hadoop ?

This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.

Hi all, I’m new to druid and I want to configure the hadoop ingestion option. I know I need to provide hadoop xml files but not sure what are the mandatory parameters I should provide in order for it to work. IS there someone who did it and can share some knowledge of it with me?

Hi All, What config to tune to avoid this the s3 Timeout waiting for connection from pool

Caused by: java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Found error while reading input stream at 0
at org.apache.druid.java.util.common.Either.valueOrThrow(Either.java:95)
at org.apache.druid.frame.processor.FrameProcessorExecutor$1ExecutorRunnable.runProcessorNow(FrameProcessorExecutor.java:258)
... 8 more
Caused by: org.apache.druid.java.util.common.ISE: Found error while reading input stream at 0
at org.apache.druid.frame.channel.ReadableInputStreamFrameChannel.lambda$startReading$1(ReadableInputStreamFrameChannel.java:224)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
... 3 more
Caused by: <http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>: Unable to copy [msq/controller_query-73a122c7-7455-44f2-b2d8-b924bce030f5/stage_0/worker_7/taskId_query-73a122c7-7455-44f2-b2d8-b924bce030f5-worker7_0/part_41] to [/opt/druid/dc30ae2f-77ee-4c25-90bc-166c575e0fd4]
at org.apache.druid.storage.s3.output.S3StorageConnector$1.nextElement(S3StorageConnector.java:203)
at org.apache.druid.storage.s3.output.S3StorageConnector$1.nextElement(S3StorageConnector.java:129)
at java.base/java.io.SequenceInputStream.peekNextStream(Unknown Source)
at java.base/java.io.SequenceInputStream.nextStream(Unknown Source)
at java.base/java.io.SequenceInputStream.read(Unknown Source)
at java.base/java.io.InputStream.read(Unknown Source)
at org.apache.druid.frame.channel.ReadableInputStreamFrameChannel.lambda$startReading$1(ReadableInputStreamFrameChannel.java:192)
... 5 more
Caused by: java.io.IOException: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
at org.apache.druid.java.util.common.FileUtils.copyLarge(FileUtils.java:387)
at org.apache.druid.storage.s3.output.S3StorageConnector$1.nextElement(S3StorageConnector.java:166)
... 11 more

Hello… I am trying to add a column transformation, but when i try to use map or filter i get an error:

Failed to sample data: function 'map' is not defined

same error with filter, but on this page it has it listed as available: https://druid.apache.org/docs/latest/misc/math-expr.html#apply-functions

This message was deleted.

This message was deleted.

Hi. I saws a log while lookinig at the logs of task.

2023-06-22T00:49:43,369 WARN [[index_kafka_hi_878c4a365a16742_nlcjpmdf]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver - Slow segment handoff! Time taken for [2] segments is 19817040 ms

What could be the reason for this long taking handoff?

This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.