This message was deleted.

yeah, after deleting the lookup table the problem is gone.

that doesn't seem so big if it's only 6M. Maybe some of the configuration, poll period, etc? Idk, what's the lookup configuration?

Hi @Ben Krug, I don’t know why adding the lookup would cause this problem. As a matter of fact, the coordinator crashed 218 times in these 2 days even I set the probe timeout to 30 seconds. In stead of error out from timeout, not the error becomes connection reset by peer:

Liveness probe failed: Get "<http://10.34.9.152:8081/status/health>": read tcp 10.34.7.138:55684->10.34.9.152:8081: read: connection reset by peer

Do you have any other lookups that work? Can you show the lookup's definition? (So we can see all the settings.)

@Ben Krug the small lookups that only has a few rows would work. The current one looks like this:

static map lookup, I'm surprised that's causing this issue. I'll do a little research...

Thank you @Ben Krug I can confirm that: 1. With a small lookup or no lookup, the coordinator is doing fine 2. With this lookup having 130k keys, then coordinator pod crashed & restarted 400+ times over the past weekend.

Can you also attach the coordinator runtime.properties? Are you also doing ingestions, and if so, how many running tasks do you have (you can look on the ingestion page of the console)?

Hi @Ben Krug, here is the runtime.properties of my coordinator. And cluster is not doing data ingestion at all. The cluster was 85% full. Before the crash, I also checked the disk space and CPU of the coordinator pod and there is nothing suspicious. Thanks!

Can you try without these settings and see if it makes a difference at all? druid.coordinator.startDelay=PT10S druid.coordinator.period=PT5S druid.indexer.queue.startDelay=PT5S

Hi @Ben Krug, sorry for the late response. I just realize that there is another config file for coordinator in /tmp/conf/druid/cluster/master/coordinator-overlord/ , and this file already has the above settings.

OK, I was saying take those settings out, and restart. Those don't look like good settings to me.