This message was deleted Apache Druid #general

Join Slack

This message was deleted.

# general

Slackbot

02/21/2023, 1:28 PM

This message was deleted.

🧐 1

Srikanth Kantamaneni

02/21/2023, 1:30 PM

Druid version 0.21.0 Tuning config in ingestion spec:

Copy code

"tuningConfig": {
  "type": "index_parallel",
  "appendableIndexSpec": {
    "type": "onheap"
  },
  "maxRowsInMemory": 100000,
  "maxBytesInMemory": -1,
  "maxTotalRows": null,
  "numShards": null,
  "splitHintSpec": null,
  "partitionsSpec": {
    "type": "hashed",
    "numShards": null,
    "partitionDimensions": [
      "id",
      "actual_event_time"
    ],
    "partitionFunction": "murmur3_32_abs",
    "targetRowsPerSegment": 3000000
  },
  "indexSpec": {
    "bitmap": {
      "type": "roaring",
      "compressRunOnSerialization": true
    },
    "dimensionCompression": "lz4",
    "metricCompression": "lz4",
    "longEncoding": "longs",
    "segmentLoader": null
  },
  "indexSpecForIntermediatePersists": {
    "bitmap": {
      "type": "roaring",
      "compressRunOnSerialization": true
    },
    "dimensionCompression": "lz4",
    "metricCompression": "lz4",
    "longEncoding": "longs",
    "segmentLoader": null
  },
  "maxPendingPersists": 0,
  "forceGuaranteedRollup": true,
  "reportParseExceptions": false,
  "pushTimeout": 0,
  "segmentWriteOutMediumFactory": null,
  "maxNumConcurrentSubTasks": 16,
  "maxRetry": 3,
  "taskStatusCheckPeriodMs": 1000,
  "chatHandlerTimeout": "PT10S",
  "chatHandlerNumRetries": 5,
  "maxNumSegmentsToMerge": 100,
  "totalNumMergeTasks": 10,
  "logParseExceptions": false,
  "maxParseExceptions": 2147483647,
  "maxSavedParseExceptions": 0,
  "maxColumnsToMerge": -1,
  "buildV9Directly": true,
  "partitionDimensions": [
    "id",
    "actual_event_time"
  ]
}

Tuning config from ingestion task:

Copy code

"tuningConfig" : {
  "type" : "index_parallel",
  "maxRowsPerSegment" : 3000000,
  "appendableIndexSpec" : {
    "type" : "onheap"
  },
  "maxRowsInMemory" : 100000,
  "maxBytesInMemory" : -1,
  "maxTotalRows" : null,
  "numShards" : null,
  "splitHintSpec" : null,
  "partitionsSpec" : {
    "type" : "hashed",
    "numShards" : null,
    "partitionDimensions" : [ "id", "actual_event_time" ],
    "partitionFunction" : "murmur3_32_abs",
    "maxRowsPerSegment" : 3000000
  },
  "indexSpec" : {
    "bitmap" : {
      "type" : "roaring",
      "compressRunOnSerialization" : true
    },
    "dimensionCompression" : "lz4",
    "metricCompression" : "lz4",
    "longEncoding" : "longs",
    "segmentLoader" : null
  },
  "indexSpecForIntermediatePersists" : {
    "bitmap" : {
      "type" : "roaring",
      "compressRunOnSerialization" : true
    },
    "dimensionCompression" : "lz4",
    "metricCompression" : "lz4",
    "longEncoding" : "longs",
    "segmentLoader" : null
  },
  "maxPendingPersists" : 0,
  "forceGuaranteedRollup" : true,
  "reportParseExceptions" : false,
  "pushTimeout" : 0,
  "segmentWriteOutMediumFactory" : null,
  "maxNumConcurrentSubTasks" : 16,
  "maxRetry" : 3,
  "taskStatusCheckPeriodMs" : 1000,
  "chatHandlerTimeout" : "PT10S",
  "chatHandlerNumRetries" : 5,
  "maxNumSegmentsToMerge" : 100,
  "totalNumMergeTasks" : 10,
  "logParseExceptions" : false,
  "maxParseExceptions" : 2147483647,
  "maxSavedParseExceptions" : 0,
  "maxColumnsToMerge" : -1,
  "buildV9Directly" : true,
  "partitionDimensions" : [ "id", "actual_event_time" ]
}

John Kowtko

02/21/2023, 1:41 PM

Hi Srikanth, I thought they were just different names (over time) for the same parameter ... am trying to confirm now ... Btw, how big is the resulting segment (in rows and MB), and what shard type/spec does it say you have?

Srikanth Kantamaneni

02/21/2023, 1:48 PM

In this case we have the segment size as 1.02 GB with 5,497,491 rows.

Srikanth Kantamaneni

02/21/2023, 1:49 PM

Partitioning is shown as

hashed

Srikanth Kantamaneni

02/21/2023, 1:51 PM

this is the shardSpec

Copy code

"shardSpec": {
  "type": "hashed",
  "partitionNum": 0,
  "partitions": 1,
  "bucketId": 0,
  "numBuckets": 1,
  "partitionDimensions": [
    "id",
    "actual_event_time"
  ],
  "partitionFunction": "murmur3_32_abs"
}

John Kowtko

02/21/2023, 1:52 PM

So it seems like it's doing the partitioning, just coming up with only one hash bucket due to your dimension values? I would guess that your two dimension fields have reasonable level of cardinality in them? Have you tried running this using numShards instead?

John Kowtko

02/21/2023, 1:55 PM

Do a select count(distinct id), count(distinct actual_event_time) from your datasource to make sure you don't accidentaly have static values in them.

Srikanth Kantamaneni

02/21/2023, 2:08 PM

We have

distinct

id

values and

distinct

actual_event_time

values in this time chunk.

Anil Gupta

02/21/2023, 2:55 PM

Incidentally, @Srikanth Kantamaneni, have you considered range partitioning? It's generally the best way to go. https://druid.apache.org/docs/latest/ingestion/native-batch.html#multi-dimension-range-partitioning

👀 1

Anil Gupta

02/21/2023, 2:57 PM

https://imply.io/blog/multi-dimensional-range-partitioning/

Srikanth Kantamaneni

02/21/2023, 3:19 PM

Thanks, I will try that. Is it okay for a datasource to have different partition types for different segments?

Anil Gupta

02/21/2023, 3:26 PM

Yes

Anil Gupta

02/21/2023, 3:27 PM

At the end of the day, though, you'll have the best query performance (and scalability) if the entire dataset is partitioned on a dimension that you are commonly filtering on.

Gian Merlino

02/22/2023, 12:51 PM

i'd also recommend doing it using SQL-based ingest. It's easier to get up and running with, & less stuff to tune in SQL, you get range partitioning using

CLUSTERED BY

, like

CLUSTERED BY id, actual_event_time

Open in Slack

Previous Next