This message was deleted Apache Druid #troubleshooting

Join Slack

This message was deleted.

# troubleshooting

Slackbot

01/10/2023, 10:44 AM

This message was deleted.

coordinator-overlord.log

Amatya Avadhanula

01/10/2023, 10:48 AM

Could you please hit this endpoint to see if there is no leader or if there is more than one leader?

GET /druid/coordinator/v1/leader

Julien Wilmet

01/10/2023, 10:49 AM

It gives me the leader IP with 200 status code

Julien Wilmet

01/10/2023, 10:49 AM

It also shows as Leader in the console

Julien Wilmet

01/10/2023, 10:50 AM

I’m looking into this issue, since I replaced every node maybe there is a lock conflict but I see no such error in the logs https://github.com/apache/druid/issues/12355

Amatya Avadhanula

01/10/2023, 10:51 AM

That particular issue has been fixed in 25.0.0

Amatya Avadhanula

01/10/2023, 10:52 AM

Your first screenshot indicates that the leader was down or unavailable

Julien Wilmet

01/10/2023, 10:52 AM

Yes but I still had locks in the table though no middleManager can run

Amatya Avadhanula

01/10/2023, 10:52 AM

Could you please try restarting it?

Amatya Avadhanula

01/10/2023, 10:52 AM

Also, could you please share the order in which the upgrade was performed

Julien Wilmet

01/10/2023, 10:52 AM

I did, several times. I also completely replaced my 3 overlords

Julien Wilmet

01/10/2023, 10:53 AM

I followed this guide : https://druid.apache.org/docs/latest/operations/rolling-updates.html 1- Historical - MiddleManager 2 - Broker - Router 3 - Coordinator - Overlord

✅ 1

Julien Wilmet

01/10/2023, 10:53 AM

Since then, I tried splitting Historical and MiddleManager

Julien Wilmet

01/10/2023, 10:56 AM

@Amatya Avadhanula I just saw this in my logs :

Copy code

2023-01-10T10:54:53,185 ERROR [HttpRemoteTaskRunner-worker-sync-4] org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner - Worker[ip] is not syncing properly. Current state is [{notSyncedForSecs=899, notSuccessfullySyncedFor=899, consecutiveFailedAttemptCount=0, syncScheduled=true}]. Resetting it.: {class=org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner}
2023-01-10T10:54:53,185 INFO [HttpRemoteTaskRunner-worker-sync-4] org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner - Kaboom! Worker[ip] removed!
2023-01-10T10:54:53,185 INFO [HttpRemoteTaskRunner-worker-sync-4] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - Stopping ChangeRequestHttpSyncer[<http://ip/_1673347193187>].
2023-01-10T10:54:53,185 INFO [HttpRemoteTaskRunner-worker-sync-4] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - Stopped ChangeRequestHttpSyncer[<http://ip/_1673347193187>].
2023-01-10T10:54:53,185 INFO [HttpRemoteTaskRunner-worker-sync-4] org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner - Worker[ip] reportin' for duty!
2023-01-10T10:54:53,185 INFO [HttpRemoteTaskRunner-worker-sync-4] org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner - Cancelling Worker[ip] scheduled task cleanup
2023-01-10T10:54:53,185 INFO [HttpRemoteTaskRunner-worker-sync-4] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - Starting ChangeRequestHttpSyncer[<http://ip/_1673348093185>].
2023-01-10T10:54:55,647 ERROR [HttpRemoteTaskRunner-worker-sync-4] org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner - Worker[ip] is not syncing properly. Current state is [{notSyncedForSecs=891, notSuccessfullySyncedFor=891, consecutiveFailedAttemptCount=0, syncScheduled=true}]. Resetting it.: {class=org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner}
2023-01-10T10:54:55,647 INFO [HttpRemoteTaskRunner-worker-sync-4] org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner - Kaboom! Worker[ip] removed!
2023-01-10T10:54:55,647 INFO [HttpRemoteTaskRunner-worker-sync-4] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - Stopping ChangeRequestHttpSyncer[<http://ip/_1673347195649>].
2023-01-10T10:54:55,647 INFO [HttpRemoteTaskRunner-worker-sync-4] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - Stopped ChangeRequestHttpSyncer[<http://ip/_1673347195649>].
2023-01-10T10:54:55,647 INFO [HttpRemoteTaskRunner-worker-sync-4] org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner - Worker[ip] reportin' for duty!
2023-01-10T10:54:55,647 INFO [HttpRemoteTaskRunner-worker-sync-4] org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner - Cancelling Worker[ip] scheduled task cleanup
2023-01-10T10:54:55,647 INFO [HttpRemoteTaskRunner-worker-sync-4] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - Starting ChangeRequestHttpSyncer[<http://ip/_1673348095647>].

Amatya Avadhanula

01/10/2023, 11:00 AM

I haven't seen this before but this seems like a different issue from https://github.com/apache/druid/issues/12355

✅ 1

Amatya Avadhanula

01/10/2023, 11:00 AM

Is ingestion working now?

Julien Wilmet

01/10/2023, 11:01 AM

Does not look like it, I am restarting the master nodes to make sure.

Julien Wilmet

01/10/2023, 11:03 AM

After a restart, it works for now

Julien Wilmet

01/10/2023, 11:05 AM

The only thing I did was remove all tasks in

druid_tasklocks

table + restart all master nodes, the second step I already did before. I don’t know what fixed it, but thank you for your help!

👍 1

Amatya Avadhanula

01/10/2023, 11:06 AM

Strange. It does seem like the issue was related to a mismatch in the task and lock priorities

8 Views

Open in Slack

Previous Next