https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
a
Please try increasing replicationThrottleLimit and maxSegmentsInNodeLoadingQueue in coordinator dynamic config If they are already high and there's no progress, check if your segment loading queues are stuck in the servers page.
s
@Amatya Avadhanula Sure thanks
To how much i can increase replicationThrottleLimit and maxSegmentsInNodeLoadingQueue
a
replicationThrottleLimit -> 10000 maxSegmentsInNodeLoadingQueue -> 500
s
we have given replicationThrottleLimit only 10... should i increase it to 1000 ?
And maxSegmentsInNodeLoadingQueue was 100
a
Sure. Even 10000 should be fine unless it affects the coordinator cycle too much
s
after increasing it should i restart any coordinator pod ?
And can i increase maxSegmentsInNodeLoadingQueue from 100 to 500 ?
👍 1
a
These are dynamic configurations and the coordinator picks them up after each cycle
s
Actually we are doing these changes in prod cluster... so will there be any impact ?
a
You could leave maxSegmentsInNodeLoadingQueue at 100 but please increase replicationThrottleLimit
There won't be too much of an impact with 1000
s
ohh i increased maxSegmentsInNodeLoadingQueue to 500 and saved the changes
will it be a problem ?
a
It can make coordinator cycles a bit slow but you can decrease it after nodes become available. Was replicationThrottleLimit not increased? (10 -> 1000 / 5000 might help)
s
replicationThrottleLimit also i increased it to 1000
👍 1
can i decrease that maxSegmentsInNodeLoadingQueue from 500 to 100 right away ?
a
Yes
s
Okay thanks
@Amatya Avadhanula I was not seeing any improvement of datasource availability... will it take some time to recover ?
unavailable segments are still increasing
a
Could you please share your historical loading queues' snapshot from the servers tab?
s
services tab ?
a
unavailable segments are still increasing
Please try this query: 
select num_replicas, count(*) from sys.segments group by num_replicas
services tab ?
In the web console
s
Actually 3377 segments are unavailable it has replication factor 0
a
Hmm, your historicals have full disks
s
I am not sure how they went
ohh okay
No actually its not full
Copy code
# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         125G   11G  114G   9% /
tmpfs            64M     0   64M   0% /dev
tmpfs           120G     0  120G   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
/dev/nvme2n1    1.7T   73G  1.6T   5% /mnt/datadisk2
tmpfs           120G   16K  120G   1% /vault/secrets
/dev/root       125G   11G  114G   9% /etc/hosts
/dev/nvme0n1    1.7T   73G  1.6T   5% /mnt/datadisk1
/dev/nvme3n1    1.7T   73G  1.6T   5% /mnt/datadisk3
/dev/nvme1n1    1.7T   74G  1.6T   5% /mnt/datadisk0
tmpfs           120G   12K  120G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           120G     0  120G   0% /proc/acpi
tmpfs           120G     0  120G   0% /proc/scsi
tmpfs           120G     0  120G   0% /sys/firmware
#troubleshooting
@Amatya Avadhanula what is that current size and max size and where can we increase it... because we have disk space of 2TB on each node and we are using theree nodes for historical running as pods in kubernetes
a
You seem to have configured your historicals to use only 300GB though. Could you please try increasing it?
s
Where can i increase it... is it from druid console ?
a
p
Oh @Amatya Avadhanula you beat me to it 😄 😄
😄 1
a
This would require restarting the historicals. You could also add extra historicals
s
Okay
p
@sridhara there is some other interesting bits in the basic cluster tuning guide - here’s where it mentions the setting that @Amatya Avadhanula is pointing you to https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html#segment-cache-size
👍 2
s
This was the parameter is it
Copy code
druid.segmentCache.locations=[{"path": "/mnt/datadisk0", "maxSize": 300000000000}, {"path": "/mnt/datadisk1", "maxSize": 300000000000}]
a
Yes
s
@Amatya Avadhanula we have given druid.server.maxSize as 37.5GB... but why in console it is showing as 300GB
a
300000000000
-> 300GB
s
ohh sorry it was in bytes
👍 1
@Amatya Avadhanula After increasing the size it is recovering now... Thanks a lot
👍 1
Open in Slack
PreviousNext
Powered by