Hi everyone I'm running into a memory issue with A...
# troubleshooting
h
Hi everyone I'm running into a memory issue with Apache Pinot 1.2.0 inside a Docker container. Despite allocating sufficient memory to the JVM, I’m getting the following error:
Copy code
[2161.229s][warning][os,thread] Attempt to protect stack guard pages failed (0x00007f73f9a6b000-0x00007f73f9a6f000).
#
# A fatal error has been detected by the Java Runtime Environment:
# Native memory allocation (mprotect) failed to protect 16384 bytes for memory to guard stack pages
# An error report file with more information is saved as:
# /opt/pinot/hs_err_pid1.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/corretto/corretto-17/issues/
Copy code
jvmOpts: "-Xms512M -Xmx10G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:MaxDirectMemorySize=16g .....

resources:
    requests:
      cpu: 500m
      memory: 28Gi
    limits:
      memory: 28Gi
m
Can you try Java-21 and the latest release 1.3?
If you still see the issue, that likely means your workload needs more memory than you are giving it.
x
also suggest to make xmx=xms=10g
can you also increase the ulimit for file open handlers?
suspect you might have many data files
h
Issue Description: We encountered a recurring issue where our Pinot server was being forcefully killed during operation due to resource constraints related to memory mapping. After extensive troubleshooting, we discovered that the root cause was related to the system parameter
vm.max_map_count
on our new Azure VM instance. Details: • Previous environment: • On our previous machine (Azure ds3_v2), the
vm.max_map_count
was set to 524288. This high limit allowed Apache Pinot to open a large number of memory-mapped files (mmap), which is critical for efficiently handling large volumes of data and for operations such as upserts. • Current environment: • On our new machine (Azure d8s_v3), we found that the
vm.max_map_count
was set to a much lower value of 65530. (default) This limit was reached quickly during the server’s operation, resulting in excessive mmap consumption and leading the operating system to kill the Pinot server process. Solution: To resolve the issue, we increased the
vm.max_map_count
parameter on the new machine to match the value of the previous, proven configuration.
x
nice!
I think we should add this into the documentation as well
@Mayank I’m seeing this
vm.max_map_count
more often blocked the cloud deployment
m
@Hassan Ait Brik how many segments do you have per server, and what's the total size? I am wondering if you have too many small segments, and perhaps increasing the segment size is a better solution here.
h
About 65k segments per server Pinot server memory mapped ~55GiB per server
m
What's the individual segment size and total data size of 65k segments?
h
The segment size depends on the table (realtime / offline / data source) --- 145 tables 65K segments per servers (replication factor 3) About 20 GB total size
z
Hi Mayank, I am also curious about the potential impact of too many small segments. What are the potential problems it would result in? What would be the best metrics to monitor?
Also found you mentioned in the other thread that for offline, the segment size can be up to 2GB uncompressed? Is that statement still accurate?