https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
b
Maybe 500M is causing a problem somehow? I'd try 1024K or something like that, see whether it works. (Just a guess.)
h
Hi @Ben Krug, thank you for getting back to me. I will definitely try it out. However, the coordinator right now is obsessed with this job. I restart the pod and call the /druid/indexer/v1/supervisor/terminateAll API, but nothing works … It just keeps flooding the logs with 
Broken pipe
error messages. Anyway to stop the Druid from doing this?
@Ben Krug so I change the znode size back to the default value and failed the task. So finally the craziness is stopped. I will try with the 1024k for the znode size. Thank you @Ben Krug
oh and I don’t think 1024k would work because my previous error message was 
too large[7,536,245 > 524,288]
so I need at least 8mb for the znode size … but how to reduce the data size?
b
let me check...
h
Thank you @Ben Krug As stated in https://druid.apache.org/docs/latest/ingestion/native-batch.html#segments-split-hint-spec, I might need to reduce 
maxNumFiles
But in my case, what would be a good number?
v
Which version of druid are you on? With later versions both task management and segment discovery use http instead of zk. This will get rid of this issue right away
h
Hi @Vijay Narayanan we are on Version 0.23.0
v
http task management is available from version 24.0.0 onwards https://druid.apache.org/docs/24.0.0/configuration/index.html#middlemanager-and-peons
can you upgrade?
k
Not sure an upgrade is necessary - 
httpRemote
druid.indexer.runner.type
has been around for a while https://druid.apache.org/docs/latest/configuration/index.html#overlord-operations
v
I checked and could not find it in the 0.23 docs
k
h
Hi @Ben Krug @Vijay Narayanan @Kyle Hoondert, thank you guys for looking into my issues. So after I changed the maxZnodeSize to 1mb and cut the maxFileNum in half (from 1000 to 500), I still got the similar error:
Copy code
org.apache.druid.java.util.common.IAE: Length of raw bytes for znode[/druid/indexer/tasks/10.34.35.19:8091/partial_index_generic_merge_mus_freeform_poc_v8_jcgbmmbl_2023-03-30T10:
37:29.444Z] too large[12,019,976 > 1,000,000]
So should I figure out how to reduce the length of raw bytes? Or should I see how to make Druid able to write large raw bytes? Thanks!
v
can you try setting druid.indexer.runner.type=httpRemote and try? this is in the middle manager run time properties
h
Hi @Vijay Narayanan, the Druid doc says that it is the default value: https://druid.apache.org/docs/latest/configuration/index.html#overlord-static-configuration From the logs, it looks like it was already the remoteTaskRunner:
Copy code
Exception while trying to assign task: {class=<http://org.apache.druid.in|org.apache.druid.in>
dexing.overlord.RemoteTaskRunner, exceptionType=class org.apache.druid.java.util.common.IAE, exceptionMessage=Length of raw bytes for znode[/druid/indexer/tasks/10.34.35.19:8091/
partial_index_generic_merge_mus_freeform_poc_v8_jcgbmmbl_2023-03-30T10:37:29.444Z] too large[12,019,976 > 1,000,000], taskId=partial_index_generic_merge_mus_freeform_poc_v8_jcgbm
mbl_2023-03-30T10:37:29.444Z}
I can try to explicitly configure it this way tho. What do you think?
Or is it like the indexer will always perform this kind of checks before writing the data? And is it normal to have such big (12mb) raw bytes to write? @Ben Krug @Vijay Narayanan @Kyle Hoondert
d
@Hao Wang is your ingestion spec gigantic? We had this problem before, we expanded the zk setting to: 
-Djute.maxbuffer=16777216
on both zk client and the zk server.
h
Hi @Didip Kerabat thank you for getting back to me. Ingestion spec is the Json object that we submit to kick off the data ingestion, right? If so, then no, it is only 8kb. It looks like Druid generated some big index data and tried to write them somewhere after enabling rollup. We didn’t have this problem when we disable rollup.
b
I think the 
Djute.maxbuffer=16777216
idea is a good call, I'd try it.
(Not sure if that's the appropriate number in your case, maybe @Didip Kerabat can say more about the setting. I found some cases where people set it to the equivalent of 1G, though.)
h
Hi @Ben Krug, fyi, after kicking off the data ingestion, I can observe the coordinator starts to crash periodically, just like when I adding the big lookups into Druid:
Copy code
NAME                                 READY   STATUS    RESTARTS      AGE
druid-coordinator-5cd78995cb-rckfc   1/1     Running   3 (23m ago)   134m
And the error message is still from Liveness/Readiness Prob for connection reset by peer:
Copy code
Warning  Unhealthy  26m                 kubelet  Readiness probe failed: Get "<http://10.34.3.93:8081/status/health>": read tcp 10.34.7.138:35060->10.34.3.93:8081: read: connection reset by peer
  Warning  Unhealthy  26m                 kubelet  Liveness probe failed: Get "<http://10.34.3.93:8081/status/health>": read tcp 10.34.7.138:35062->10.34.3.93:8081: read: connection reset by peer
b
It seems the djut.maxbuffer setting can affect coordinators too. Another option is to try http-based segment management, as Vijay suggested, to relieve pressure on zk.
h
Hi @Ben Krug, I have added druid.indexer.runner.type=httpRemote to my coordinator, but it still crashes like I showed above. Btw, is there any instruction for setting the 
jute.maxbuffer
for Druid cluster?
b
for instance, you might set -Djute.maxbuffer=104857600 in the jvm.config on zk, historicals, and coordinator.
(I've only seen it set on historicals and zk. I'm guessing about coordinator.)
h
Got it, if it crashes again, then I will try it out. So when we enable rollup, Druid writes more data to zk than no rollup?
b
That part I'll admit, I don't understand why that would be.
d
at ZK level, it’s all about segment file information. Rolling up will reduce number of rows, which will reduce number of segment files. So, number of records in ZK will reduce, but I don’t think the byte size of ZK record itself will be impacted.
v
@Hao Wang the httpRemote is to be added to middle manager run time properties not coordinator. Please try that. Btw remote is default but that just remote task through zoo keeper. httpRemote goes through http instead of zoo keeper
🙌 1
d
We have been using httpRemote since 4-5 versions ago. It’s stable.
h
@Vijay Narayanan @Didip Kerabat so enable httpRemote for my middle manager can be the solution? I will try it out. Thank you so much!
One more thing, the coordinator is really unstable after starting the data ingestion a few hours ago. It kept crashing (identified as unhealthy by Kubenetes and got restarted). The only thing I can see from the log is something like
Copy code
[LookupCoordinatorManager--3] org.apache.druid.server.lookup.cache.LookupCoordinatorManager - Failed to finish lookup management on node [http:10.34.71.105:8083]
: {class=org.apache.druid.server.lookup.cache.LookupCoordinatorManager, exceptionType=class java.util.concurrent.ExecutionException, exceptionMessage=org.jboss.netty.handler.timeout.ReadTimeo
utException}
But I don’t think it is the root cause. The coordinator itself just become super slow when the data ingestion is almost completed, but disk is not full and cpu is not highly utilized and I believe 2G memory should be enough. Any idea about what is going on?
d
I don’t think 2GB for coordinator is enough
the only tiny component we have is router. Everything else at least needs 12-20GB of RAM
👀 1
h
ok, like 4G is better?
oh, I will try with bigger settings
d
do you have budget for at least 12GB?
h
Not sure, but let me start with 8G
Thank you so much for the timely response @Didip Kerabat
Open in Slack
PreviousNext
Powered by