Hi Team, We are running fluentbit 3.1.9 as a conta...
# fluent-bit
s
Hi Team, We are running fluentbit 3.1.9 as a container on s390x machine with Ubuntu. Our machine was using Ubuntu 20.04 before and fluentbit was working fine. Recently we upgraded OS to 24.04 and since then we are getting the error
Too many open files
. We tried modifying the soft and hard limits but that didn;t help. One thing we were able to see is that issue is more likely to be happened on machines that have higher number of docker container running (200 to 300 in our cose). I saw an open issue #1777 on this topic. Right now we are attempting to see if changing the inotify limit could solve the issue. The current value is as below
Copy code
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 65536
We are planning to increase these numbers to as shown below
Copy code
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 8192
Is there any suggestions from anyone here on whats the right way to resolve this?
p
Ensure you are correctly setting it for the container runtime and the containers it launches. Check for other lockdown as well like selinux. Ultimately this is an OS change so will be down to configuration there
You can also launch the debug container to shell in and check
e
We have seen this issue in our infrastructure with log file monitoring agents although that was FileBeat and we have been increasing these values to resolve the issue. If I remember correctly main issue is too low value for >
Copy code
fs.inotify.max_user_instances = 128
Although we have increased second value too. >
- { param: fs.inotify.max_user_instances, val: 8192 }
>
- { param: fs.inotify.max_user_watches, val: 524288 }