https://pinot.apache.org/ logo
Join Slack
Powered by
Hi Team Need help in debugging this issue. The cp...
# troubleshooting
m
Hi Team Need help in debugging this issue. The cpu for all servers in a tenant keeps spiking. There is no queries running, just kafka ingestion. Tried pausing ingestion for the main table then the cpu went down, But after resuming it spiked back again. Things I have tried. • Restarting servers • Reloading segments • Pausing ingestion None of my attempts worked for stabilising the cpu load. Can someone please help here?
@Mayank @Xiang Fu Need help please
x
how heavy is your kafka workload? what's the consumption rate for kafka and ingestion lag metrics?
m
The kafka throughput is 600/s for 3 brokers.
x
That seems not much, did you observe any ingestion lag?
m
Ingestion metrics were high then it was stuck
x
hmm
do you see any gc metrics
m
The spike happened yesterday also, the the ingestion lag kept on increasing , after today morning spike the servers were stuck
x
and how large is the table
is the table in heap mode or mmap mode?
m
I was also checking gc only. I am assuming the heap space is not enough the gc details from 1 server
Copy code
S0     S1     E      O      M     CCS    YGC     YGCT     FGC    FGCT     CGC    CGCT       GCT   
  0.00  75.33  84.11  96.94  99.30  97.03   8713    15.454  4986  3198.864    74     0.135  3214.453
  0.00  75.33  84.11  96.94  99.30  97.03   8713    15.454  4986  3198.864    74     0.135  3214.453
  0.00  75.33  84.11  96.94  99.30  97.03   8713    15.454  4986  3198.864    74     0.135  3214.453
  0.00  75.33  84.11  96.94  99.30  97.03   8713    15.454  4986  3198.864    74     0.135  3214.453
  0.00  75.33  84.11  96.94  99.30  97.03   8713    15.454  4986  3198.864    74     0.135  3214.453
  0.00  75.33  84.11  96.94  99.30  97.03   8713    15.454  4986  3198.864    74     0.135  3214.453
  0.00  75.33  84.11  96.94  99.30  97.03   8713    15.454  4986  3198.864    74     0.135  3214.453
  0.00  75.33  84.11  96.94  99.30  97.03   8713    15.454  4986  3198.864    74     0.135  3214.453
  0.00  75.33  84.11  96.94  99.30  97.03   8713    15.454  4986  3198.864    74     0.135  3214.453
  0.00  75.33  84.11  96.94  99.30  97.03   8713    15.454  4986  3198.864    74     0.135  3214.453
Copy code
"loadMode": "MMAP",
x
how many segments loaded there
m
4 servers around 1000 segments each, max size 200M
x
what's the jvmOpts like xmx xms and how much is the container memory you have
4,986 Full GCs, with total Full GC time of ~3,199 seconds — nearly 53 minutes spent in Full GC
try to configure jvm with Xms=Xmx= half or your container memory
say if you are using a 32G container then make Xms=16g and xmx=16g
m
Memory : requested=20g limit=32g jvm opts
Copy code
-Xms4G -Xmx4G -XX:MaxDirectMemorySize=8g
Shall I make this
Copy code
-Xms10G -Xmx10G
?
Should I increase MaxDirectMemorySize too ?
x
then do 16g
make your k8s request =limit to 32g or 30g
-Xms16G -Xmx16G -XX:MaxDirectMemorySize=16g
m
okay, trying this out
Open in Slack
PreviousNext
Powered by