<#C0CTQGHKJ|fluent-bit> I'm trying to set a storag...
# fluent-bit
g
#fluent-bit I'm trying to set a storage limit in the
[OUTPUT]
block to prevent it from exceeding the defined threshold. Although I'm using both memory and filesystem storage, it doesn't seem to be working as expected.
Copy code
/host/var/log/fluent-bit-storage # du -sh *
13.8M	emitter.10
4.0K	emitter.11
4.0K	emitter.12
12.0K	emitter.13
2.2G	emitter.14
247.9M	emitter.15
508.5M	emitter.2
25.1G	emitter.3
23.9M	emitter.4
3.0G	emitter.5
5.7M	emitter.6
212.0K	emitter.7
174.9M	emitter.8
11.6M	emitter.9
32.0G	tail.0
Copy code
fluent-bit.conf:
----
[SERVICE]
    Daemon Off
    Flush 1
    Log_Level info
    Parsers_File /fluent-bit/etc/parsers.conf
    Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
    HTTP_Server On
    HTTP_Listen 0.0.0.0
    HTTP_Port 2020
    Health_Check On
    storage.path    /var/log/fluent-bit-storage
    storage.metrics on
    storage.max_chunks_up 512
    storage.sync  normal
    storage.backlog.mem_limit 3G
    storage.total_limit_size 10G
@INCLUDE custom-*.conf

[INPUT]
    Name tail
    Path /var/log/containers/*
    Exclude_Path      /var/log/containers/fluent-bit*.log
    multiline.parser docker, cri
    Tag kube.*
    Mem_Buf_Limit 1GB
    storage.type   filesystem
    DB             /var/log/fluent-bit.db
[OUTPUT]
    Name              es
    Match             log.eod.hz.billing.sync.*
    Host              elasticsearch-elasticsearch
    Port              9200
    Suppress_Type_Name On
    Replace_Dots On
    Index fluent-bit-elastic-eod-hz-billing-sync
    Retry_Limit 20
    Buffer_Size false
    Generate_ID on
    Trace_Error on
    storage.total_limit_size 10G
#fluent-bit any help on this will be more helpfull.
@Pat any thought on this. would be more helpfull.
p
I'm not sure what the question is, what's not working? Can you reduce it to a simple reproducer? Are you on the latest version? You've also got a wildcard include in there so that could be affecting things.
g
i am limiting total_limit_size on the filesystem to 10G but still its exceeded the size
im on 2.1.9 (app-version)
p
are you sure you mean 2.1.9?
that's very old so won't be supported
g
fluent-bit logging 10 2025-04-09 165649.192836221 +0000 UTC deployed fluent-bit-0.38.0 2.1.9
p
try latest and see if that sorts it
the log output prints out the version at startup
g
ok let me upgrade
p
what do the logs show as well?
maybe your config is being ignored or there are warnings/etc. in there
g
Copy code
Fluent Bit v2.1.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2025/07/30 14:25:03] [error] [parser] parser named 'docker' already exists, skip.
[2025/07/30 14:25:03] [ info] [fluent bit] version=2.1.9, commit=c625ad7ea4, pid=1
[2025/07/30 14:25:04] [ info] [storage] ver=1.4.0, type=memory+filesystem, sync=normal, checksum=off, max_chunks_up=512