This message was deleted Apache Druid #general

Join Slack

This message was deleted.

# general

Slackbot

06/26/2023, 2:05 PM

This message was deleted.

Hareesh Joshi

06/26/2023, 6:27 PM

What you have posted above is the supervisor status. Suggestions: Check the task status under the supervisor. Usually task status reveals more details on why the task is being marked as UNHEALTHY.

Anant Sharma

06/27/2023, 4:51 AM

@Hareesh Joshi this is my task status error it says could not allocate the segments

Copy code

{
  "id": "index_kafka_eber_vehicles_gps_1b7f49d6d2b1b16_gfpphkcj",
  "groupId": "index_kafka_eber_vehicles_gps",
  "type": "index_kafka",
  "createdTime": "2023-06-27T04:37:31.966Z",
  "queueInsertionTime": "1970-01-01T00:00:00.000Z",
  "statusCode": "FAILED",
  "status": "FAILED",
  "runnerStatusCode": "WAITING",
  "duration": 19170,
  "location": {
    "host": "10.101.56.213",
    "port": 8101,
    "tlsPort": -1
  },
  "dataSource": "eber_vehicles_gps",
  "errorMsg": "org.apache.druid.java.util.common.ISE: Could not allocate segment for row with timestamp[2023-06-26T..."

Anant Sharma

06/27/2023, 4:57 AM

{ "dataSource": "eber_vehicle_components_status", "stream": "eber.vehicle.components.status.qc", "partitions": 3, "replicas": 1, "durationSeconds": 604800, "activeTasks": [ { "id": "index_kafka_eber_vehicle_components_status_832fc79d45a7fc1_jobdjjoj", "startingOffsets": { "0": 128541, "1": 76702, "2": 54380 }, "startTime": "2023-06-26T221328.862Z", "remainingSeconds": 580620, "type": "ACTIVE", "currentOffsets": { "0": 129207, "1": 129047, "2": 129855 }, "lag": { "0": 0, "1": 0, "2": 0 } } ], "publishingTasks": [], "latestOffsets": { "0": 129207, "1": 129047, "2": 129855 }, "minimumLag": { "0": 0, "1": 0, "2": 0 }, "aggregateLag": 0, "offsetsLastUpdated": "2023-06-27T045602.258Z", "suspended": false, "healthy": false, "state": "UNHEALTHY_TASKS", "detailedState": "UNHEALTHY_TASKS", "recentErrors": [ { "timestamp": "2023-06-26T213259.042Z", "exceptionClass": "org.apache.druid.java.util.common.IAE", "message": "org.apache.druid.java.util.common.IAE: Received server error with status [400 Bad Request]; first 1KB of body: {\"error\":\"Can't find chatHandler for handler[index_kafka_eber_vehicle_components_status_832fc79d45a7fc1_cjiapapn]\"}", "streamException": false }, { "timestamp": "2023-06-26T214652.550Z", "exceptionClass": "org.apache.druid.java.util.common.IAE", "message": "org.apache.druid.java.util.common.IAE: Received server error with status [400 Bad Request]; first 1KB of body: {\"error\":\"Can't find chatHandler for handler[index_kafka_eber_vehicle_components_status_832fc79d45a7fc1_glbdmbel]\"}", "streamException": false } ] }

Amatya Avadhanula

06/27/2023, 7:19 AM

Could you check the overlord logs for errors related to lock acquisition and segment allocation in general?

Amatya Avadhanula

06/27/2023, 7:20 AM

Is kafka ingestion running on an interval in a datasource which already has segments with a different granularity than the segmentGranularity specified for the streaming ingestion job?

Anant Sharma

06/27/2023, 11:11 AM

below are the error im geeting 1- coordinator

Copy code

2023-06-27T11:05:55,032 ERROR [qtp1286172885-129] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not allocate pending segment for rowInterval[2023-06-27T08:33:04.624Z/2023-06-27T08:33:04.625Z], segmentInterval[2023-06-26T00:00:00.000Z/2023-07-03T00:00:00.000Z].
2023-06-27T11:06:23,854 ERROR [qtp1286172885-127] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not allocate pending segment for rowInterval[2023-06-27T08:33:04.624Z/2023-06-27T08:33:04.625Z], segmentInterval[2023-06-26T00:00:00.000Z/2023-07-03T00:00:00.000Z].
2023-06-27T11:06:43,555 ERROR [qtp1286172885-123] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not allocate pending segment for rowInterval[2023-06-27T08:33:04.624Z/2023-06-27T08:33:04.625Z], segmentInterval[2023-06-26T00:00:00.000Z/2023-07-03T00:00:00.000Z].
2023-06-27T11:07:21,343 ERROR [qtp1286172885-144] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not allocate pending segment for rowInterval[2023-06-27T08:33:04.624Z/2023-06-27T08:33:04.625Z], segmentInterval[2023-06-26T00:00:00.000Z/2023-07-03T00:00:00.000Z].
2023-06-27T11:07:50,156 ERROR [qtp1286172885-158] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not allocate pending segment for rowInterval[2023-06-27T08:33:04.624Z/2023-06-27T08:33:04.625Z], segmentInterval[2023-06-26T00:00:00.000Z/2023-07-03T00:00:00.000Z].
2023-06-27T11:08:19,541 ERROR [qtp1286172885-147] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not allocate pending segment for rowInterval[2023-06-27T08:33:04.624Z/2023-06-27T08:33:04.625Z], segmentInterval[2023-06-26T00:00:00.000Z/2023-07-03T00:00:00.000Z].
2023-06-27T11:08:48,454 ERROR [qtp1286172885-122] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not allocate pending segment for rowInterval[2023-06-27T08:33:04.624Z/2023-06-27T08:33:04.625Z], segmentInterval[2023-06-26T00:00:00.000Z/2023-07-03T00:00:00.000Z].
2023-06-27T11:09:16,756 ERROR [qtp1286172885-139] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not allocate pending segment for rowInterval[2023-06-27T08:33:04.624Z/2023-06-27T08:33:04.625Z], segmentInterval[2023-06-26T00:00:00.000Z/2023-07-03T00:00:00.000Z].
2023-06-27T11:09:36,043 ERROR [qtp1286172885-155] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not allocate pending segment for rowInterval[2023-06-27T08:33:04.624Z/2023-06-27T08:33:04.625Z], segmentInterval[2023-06-26T00:00:00.000Z/2023-07-03T00:00:00.000Z].

2- middle manager

Copy code

2023-06-27T11:06:44,471 ERROR [forking-task-runner-0] org.apache.druid.indexing.overlord.ForkingTaskRunner - Process exited with code[2] for task: index_kafka_eber_vehicle_components_status_38c7c2996ed5932_cmmlohne
[ec2-user@

Amatya Avadhanula

06/27/2023, 11:32 AM

Is kafka ingestion running on an interval in a datasource which already has segments with a different granularity than the segmentGranularity specified for the streaming ingestion job?

Could you confirm if this datasource has other segments for this interval?

Amatya Avadhanula

06/27/2023, 11:32 AM

Also I think you are using segmentGranularity of WEEK. Please avoid it

Hagen Rother

06/27/2023, 4:44 PM

You should never do more than 1h segments granularity it's the window that has to be replayed from kafka on failures. Run a compacting job to merge the segments later.

Anant Sharma

06/28/2023, 7:22 AM

i've fixed that now most of them are in running state but can see error for what task error status

Copy code

{
  "id": "index_kafka_eber_gateways_sensors_data_5a0ebe22f44a3f5_cepalohc",
  "groupId": "index_kafka_eber_gateways_sensors_data",
  "type": "index_kafka",
  "createdTime": "2023-06-28T06:53:01.032Z",
  "queueInsertionTime": "1970-01-01T00:00:00.000Z",
  "statusCode": "FAILED",
  "status": "FAILED",
  "runnerStatusCode": "WAITING",
  "duration": -1,
  "location": {
    "host": "10.101.60.160",
    "port": 8100,
    "tlsPort": -1
  },
  "dataSource": "eber_gateways_sensors_data",
  "errorMsg": "The worker that this task was assigned disappeared and did not report cleanup within timeout[PT15M]...."
}

and i can see error on my coordinator node as we

Copy code

2023-06-28T04:05:45,454 ERROR [qtp1286172885-118] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not allocate pending segment for rowInterval[2023-06-27T11:15:20.599Z/2023-06-27T11:15:20.600Z], segmentInterval[2023-06-26T00:00:00.000Z/2023-07-03T00:00:00.000Z].
2023-06-28T04:48:58,796 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}

Amatya Avadhanula

06/28/2023, 7:24 AM

Copy code

2023-06-28T04:05:45,454 ERROR [qtp1286172885-118] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not allocate pending segment for rowInterval[2023-06-27T11:15:20.599Z/2023-06-27T11:15:20.600Z], segmentInterval[2023-06-26T00:00:00.000Z/2023-07-03T00:00:00.000Z].

You are still using segment granularity of WEEK

Amatya Avadhanula

06/28/2023, 7:24 AM

I suspect you have segments with MONTH or some other granularity for the datasource+interval you are ingesting to

Amatya Avadhanula

06/28/2023, 7:27 AM

Copy code

2023-06-28T04:48:58,796 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}

Could you share screenshot of your servers tab from the druid console? Eg:

Anant Sharma

06/28/2023, 7:48 AM

here is the ss @Amatya Avadhanula

Open in Slack

Previous Next