This message was deleted.

Could you also share the overlord log from around that time? Segment allocation is done by the overlord. So that might tell us something. Something I noticed is that the timestamp it is loading is from 20 days ago... that might or might not be relevant. The task failures in the middle manager logs (code 137) is out of memory.

i've fixed few things like segmentG, and resource because of which i was sseing 137 error MM. now most of them are in running state but can see error for what task error status

{
  "id": "index_kafka_eber_gateways_sensors_data_5a0ebe22f44a3f5_cepalohc",
  "groupId": "index_kafka_eber_gateways_sensors_data",
  "type": "index_kafka",
  "createdTime": "2023-06-28T06:53:01.032Z",
  "queueInsertionTime": "1970-01-01T00:00:00.000Z",
  "statusCode": "FAILED",
  "status": "FAILED",
  "runnerStatusCode": "WAITING",
  "duration": -1,
  "location": {
    "host": "10.101.60.160",
    "port": 8100,
    "tlsPort": -1
  },
  "dataSource": "eber_gateways_sensors_data",
  "errorMsg": "The worker that this task was assigned disappeared and did not report cleanup within timeout[PT15M]...."
}

and i can see error on my coordinator node as we

2023-06-28T04:05:45,454 ERROR [qtp1286172885-118] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not allocate pending segment for rowInterval[2023-06-27T11:15:20.599Z/2023-06-27T11:15:20.600Z], segmentInterval[2023-06-26T00:00:00.000Z/2023-07-03T00:00:00.000Z].
2023-06-28T04:48:58,796 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}

https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1687937064821759?thread_ts=1687788310.124809&cid=C0303FDCZEZ - Is this thread on #C0303FDCZEZ for a different issue?

its for the sam e

2023-06-28T044858,796 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}

There is a historical now so I don't think this should present anymore. Could you confirm if the historical recently restarted?

i've made few changes in resources that why it got restarted 25 h back

so i recently did the compaction where i put the above mention "skipOffsetFromLatest": "PT1H",

before it was for a week ive changed it to hour and below is the supervisor conf and there we have defined the granularity as MONTH are you sure this is because of the granularity as exact same thing is running in my different environment and it working fine that what possibly making the issue here ?

{
  "type": "kafka",
  "spec": {
    "dataSchema": {
      "dataSource": "eber_vehicle_components_status",
      "timestampSpec": {
        "column": "timestamp",
        "format": "millis",
        "missingValue": null
      },
      "dimensionsSpec": {
        "dimensions": [
          {
            "type": "string",
            "name": "gateway_id",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "string",
            "name": "value",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          },
          {
            "type": "float",
            "name": "id",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": false
          }
        ],
        "dimensionExclusions": [
          "__time",
          "timestamp"
        ],
        "includeAllDimensions": false
      },
      "metricsSpec": [],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "MONTH",
        "queryGranularity": {
          "type": "none"
        },
        "rollup": true,
        "intervals": []
      },
      "transformSpec": {
        "filter": null,
        "transforms": []
      }
    },
    "ioConfig": {
      "topic": "eber.vehicle.components.status.qc",
      "inputFormat": {
        "type": "json",
        "flattenSpec": {
          "useFieldDiscovery": true,
          "fields": []
        },
        "keepNullColumns": true,
        "assumeNewlineDelimited": false,
        "useJsonNodeReader": false
      },
      "replicas": 1,
      "taskCount": 1,
      "taskDuration": "PT86400S",
      "consumerProperties": {
        "bootstrap.servers": "qc-kafka-kafka-bootstrap.kafka.svc.cluster.local:9092,",
        "security.protocol": "SASL_PLAINTEXT",
        "sasl.mechanism": "SCRAM-SHA-512",
        "sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username='admin-etm-qc' password='xxx';",
        "auto.offset.reset": "earliest"
      },
      "autoScalerConfig": null,
      "pollTimeout": 100,
      "startDelay": "PT5S",
      "period": "PT30S",
      "useEarliestOffset": true,
      "completionTimeout": "PT1800S",
      "lateMessageRejectionPeriod": null,
      "earlyMessageRejectionPeriod": null,
      "lateMessageRejectionStartDateTime": null,
      "configOverrides": null,
      "idleConfig": null,
      "stream": "eber.vehicle.components.status.qc",
      "useEarliestSequenceNumber": true,
      "type": "kafka"
    },
    "tuningConfig": {
      "type": "kafka",
      "appendableIndexSpec": {
        "type": "onheap",
        "preserveExistingMetrics": false
      },
      "maxRowsInMemory": 1000000,
      "maxBytesInMemory": 0,
      "skipBytesInMemoryOverheadCheck": false,
      "maxRowsPerSegment": 5000000,
      "maxTotalRows": null,
      "intermediatePersistPeriod": "PT10M",
      "maxPendingPersists": 0,
      "indexSpec": {
        "bitmap": {
          "type": "roaring",
          "compressRunOnSerialization": true
        },
        "dimensionCompression": "lz4",
        "stringDictionaryEncoding": {
          "type": "utf8"
        },
        "metricCompression": "lz4",
        "longEncoding": "longs"
      },
      "indexSpecForIntermediatePersists": {
        "bitmap": {
          "type": "roaring",
          "compressRunOnSerialization": true
        },
        "dimensionCompression": "lz4",
        "stringDictionaryEncoding": {
          "type": "utf8"
        },
        "metricCompression": "lz4",
        "longEncoding": "longs"
      },
      "reportParseExceptions": false,
      "handoffConditionTimeout": 0,
      "resetOffsetAutomatically": false,
      "segmentWriteOutMediumFactory": null,
      "workerThreads": null,
      "chatThreads": null,
      "chatRetries": 8,
      "httpTimeout": "PT10S",
      "shutdownTimeout": "PT80S",
      "offsetFetchPeriod": "PT30S",
      "intermediateHandoffPeriod": "P2147483647D",
      "logParseExceptions": false,
      "maxParseExceptions": 2147483647,
      "maxSavedParseExceptions": 0,
      "skipSequenceNumberAvailabilityCheck": false,
      "repartitionTransitionDuration": "PT120S"
    }
  },
  "context": null
}

Could you share the compaction spec as well?

sure

Strange, my suspicion was that you were using a granularity of WEEK in the compaction spec. Sorry to repeat, but could you please confirm from the segments tab if there are segments with week granularity that already exist for 2023-06-26/2023-07-03?

@Amatya Avadhanula can you tell me how can do so! below is the ss for segments which might help you for your question

The first few (4) rows have a weekly granularity (refer to the start and end columns of segments). These belong to the datasource eber_v...

any suggestive ways to do so ?

and do think this is the only issue or it could be something else to ?

Do you observe any other issues with ingestion for datasources having similar volume + task count? If not, I think this could be the only issue

no i couldn’t

1. Suspend the supervisor for the datasource. (Three dots next to the magnifying glass icon > suspend) 2. Submit a reindexing task using the native batch ingestion wizard for the interval 2023-06-01/2023-08-01 for the datasource (eber_v...) and follow the wizard. Just change the segment granularity from week to month 3. Resume the supervisor for the datasource

let me try right now

The supervisor cannot be used to reindex

i have to add month here. ?

No, this is query granularity. Please don't modify it

ok i didn’t 😅is it the right place ?

Yes!

what should be the Partitioning type & Time intervals

Partitioning type -> dynamic (for now). intervals can be left blank

Max rows per segment Max total rows can i use the older one or we should use a new one

defaults should be fine

okay

Yes, the defaults would work since there isn't a lot of data yet

i think think this is the last one this should be default to

Yes

ive submitted it and its back to the same page with the same specs and still on suspend have we missed any thing ?

No, please check the task's status

its failed

Error?

in middlemanager logs

2023-06-28T18:01:36,648 ERROR [forking-task-runner-13] org.apache.druid.indexing.overlord.ForkingTaskRunner - Process exited with code[137] for task: index_parallel_eber_vehicle_components_status_beemdiaf_2023-06-28T18:01:17.205Z

but there are only 3 pods running and ive given maximumreplicas to 5 it doest seems to be an issue with resources

Code 137 is generally due to memory limits

let me check