This message was deleted.

I’m not sure this is related to Druid 25.0.0. But Kafka ingestion lag started to increase a lot more since I’ve upgraded

one change is that in 25.0 tasks are managed using http threads. What is druid.server.http.numThreads on the indexer and how many tasks are running?

Hello Vijay, thanks for having a look.

druid.server.http.numThreads=60

on indexers. And I have 140 running tasks at the moment with 20 indexers (7 slots each)

ok…take a look at https://druid.apache.org/docs/latest/design/indexer.html#server-http-threads

GCS

how big are these segments?

At ingestion time, segments size is 300MB max

is this failure always for one specific kafka topic or partition?

On different Kafka topics (4)

I see that the kafka task is pausing a lot. Have you set druid.indexer.tasklock.forceTimeChunkLock? Also are there many kafka tasks writing to the same time chunk with different priorities?

Yes I did set this param, I have index parallel tasks that run every hour to reprocess segments on now -3 hours

you set druid.indexer.tasklock.forceTimeChunkLock to true or false?

True

I can see

Adding lock on interval[2023-01-10T09:00:00.000Z/2023-01-10T10:00:00.000Z]

this means the timechunk is being locked.

Yep, here it is.

{
  "type": "kafka",
  "spec": {
    "dataSchema": {
      "dataSource": "beam-advanced",
      "timestampSpec": {
        "column": "timestamp",
        "format": "millis",
        "missingValue": null
      },
      "dimensionsSpec": {
        "dimensions":
      },
      "metricsSpec": [
      ],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "HOUR",
        "queryGranularity": "HOUR",
        "rollup": true,
        "intervals": []
      },
      "transformSpec": {
        "filter": null,
        "transforms": []
      }
    },
    "ioConfig": {
      "topic": "beam-druid-advanced",
      "inputFormat": {
        "type": "avro_stream",
        "avroBytesDecoder": {
          "type": "schema_registry",
          "url": "<http://schema-registry:80>",
          "capacity": 2147483647,
          "urls": null,
          "config": null,
          "headers": null
        },
        "binaryAsString": false,
        "extractUnionsByType": false
      },
      "replicas": 1,
      "taskCount": 20,
      "taskDuration": "PT1800S",
      "consumerProperties": {
        "bootstrap.servers": "10.132.0.164:9092,10.132.0.4:9092,10.132.0.94:9092"
      },
      "autoScalerConfig": {
        "autoScalerStrategy": "lagBased",
        "lagCollectionIntervalMillis": 30000,
        "lagCollectionRangeMillis": 600000,
        "scaleActionStartDelayMillis": 300000,
        "scaleActionPeriodMillis": 600000,
        "scaleOutThreshold": 1800000,
        "scaleInThreshold": 600000,
        "triggerScaleOutFractionThreshold": 0.3,
        "triggerScaleInFractionThreshold": 0.9,
        "taskCountMax": 60,
        "taskCountMin": 20,
        "scaleInStep": 3,
        "scaleOutStep": 10,
        "enableTaskAutoScaler": true,
        "minTriggerScaleActionFrequencyMillis": 600000
      },
      "pollTimeout": 100,
      "startDelay": "PT5S",
      "period": "PT30S",
      "useEarliestOffset": false,
      "completionTimeout": "PT3600S",
      "lateMessageRejectionPeriod": null,
      "earlyMessageRejectionPeriod": null,
      "lateMessageRejectionStartDateTime": null,
      "configOverrides": null,
      "idleConfig": null,
      "stream": "beam-druid-advanced",
      "useEarliestSequenceNumber": false,
      "type": "kafka"
    },
    "tuningConfig": {
      "type": "kafka",
      "appendableIndexSpec": {
        "type": "onheap",
        "preserveExistingMetrics": false
      },
      "maxRowsInMemory": 5000000,
      "maxBytesInMemory": 0,
      "skipBytesInMemoryOverheadCheck": false,
      "maxRowsPerSegment": 5000000,
      "maxTotalRows": null,
      "intermediatePersistPeriod": "PT10M",
      "maxPendingPersists": 0,
      "indexSpec": {
        "bitmap": {
          "type": "roaring",
          "compressRunOnSerialization": true
        },
        "dimensionCompression": "lz4",
        "stringDictionaryEncoding": {
          "type": "utf8"
        },
        "metricCompression": "lz4",
        "longEncoding": "longs"
      },
      "indexSpecForIntermediatePersists": {
        "bitmap": {
          "type": "roaring",
          "compressRunOnSerialization": true
        },
        "dimensionCompression": "lz4",
        "stringDictionaryEncoding": {
          "type": "utf8"
        },
        "metricCompression": "lz4",
        "longEncoding": "longs"
      },
      "reportParseExceptions": false,
      "handoffConditionTimeout": 0,
      "resetOffsetAutomatically": false,
      "segmentWriteOutMediumFactory": null,
      "workerThreads": null,
      "chatThreads": null,
      "chatRetries": 8,
      "httpTimeout": "PT10S",
      "shutdownTimeout": "PT80S",
      "offsetFetchPeriod": "PT30S",
      "intermediateHandoffPeriod": "P2147483647D",
      "logParseExceptions": false,
      "maxParseExceptions": 2147483647,
      "maxSavedParseExceptions": 0,
      "skipSequenceNumberAvailabilityCheck": false,
      "repartitionTransitionDuration": "PT120S"
    }
  },
  "context": null
}

can you check whether the segment beam-advanced_2023-01-10T090000.000Z_2023-01-10T100000.000Z_2023-01-10T090000.409Z_95 is there in deep storage?

beam-advanced_2023-01-10T09:00:00.000Z_2023-01-10T10:00:00.000Z_2023-01-10T09:00:00.409Z_95

is in deepstorage

I wanted to see if there are any metastore related errors in the log

I’m not seeing any on last 7 days log files

Try resetting the supervisor…that will most likely solve this problem

I’ve already tried to reset the supervisor yesterday

has there been any change to the topic name?

Nope