https://linen.dev logo
Join Slack
Powered by
Hi All, Has anyone faced this error in fluent-bit ...
# fluent-bit
p
Hi All, Has anyone faced this error in fluent-bit 
2026/07/21 12:33:56.499] [ warn] [output:s3:s3.1] Could not buffer chunk. Data order preservation will be compromised [2026/07/21 12:33:56.500] [ warn] [engine] failed to flush chunk '<tel:11784637203|1-1784637203>.399331802.flb', retry in 20 seconds: task_i
d=337, input=tail.2 > output=s3.1 (out_id=1) [2026/07/21 12:33:57.455] [error] [output:s3:s3.1] Buffer is full: current_buffer_size=18446744073522589696, new_data=1855
9, store_dir_limit_size=2000000000 bytes
current buffer seems to be a very big data but that much data is not tailed..
@eduardo @hhatake can you please help here .We are using fluentbit 5.7.0
e
It seems you have back pressure, not able to deliver data fast enough so your buffers are hitting the limit
p
Just added retry_limit 5 and retry_exhausted delete so that failed chnks are cleared from buffer path and moved to latest version as these variables are not available in 3.0.2 storage is filesystem i don't see the buffer path full too It is empty. Chunks are not cretaed in the buffer path
Copy code
[SERVICE]
    Flush                     120
    Log_Level                 info
    Daemon                    off
    Parsers_File              parsers.conf
    HTTP_Server               ${HTTP_SERVER}
    HTTP_Listen               0.0.0.0
    HTTP_Port                 ${HTTP_PORT}
    storage.path              /var/fluent-bit/state/flb-storage/
    storage.sync              normal
    storage.checksum          off
    storage.backlog.mem_limit 50M
    storage.max_chunks_up     512
[INPUT]
    Name                fluentbit_metrics
    Tag                 fluentbit_metrics
    scrape_interval     30
[INPUT]
    Name                tail
    Tag                 kubernetes.logs.*
    Path                /var/log/containers/*.log
    Docker_Mode         On
    Docker_Mode_Flush   5
    Docker_Mode_Parser  container_firstline
    Parser              docker
    DB                  /var/fluent-bit/state/flb_container.db
    Skip_Long_Lines     On
    Refresh_Interval    5
    Rotate_Wait         30
    storage.type        filesystem
    Read_from_Head      false
[FILTER]
    Name                kubernetes
    Match               kubernetes.logs.*
    tls.verify          Off
    Kube_URL            <https://kubernetes.default.svc.cluster.local:443>
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Tag_Prefix     kubernetes.logs.var.log.containers
    Merge_Log           On
    Labels              Off
    Annotations         Off
    Buffer_Size         0
    Use_Kubelet         true
    Kubelet_Port        10250
[FILTER]
    Name nest
    Match kubernetes.logs.*
    Operation lift
    Nested_under kubernetes
[FILTER]
    Name                modify
    Match               kubernetes.logs.*
    Add                 tag docker.all
[OUTPUT]
    Name                s3
    Match_regex         ^(kubernetes.logs.*)$
    Bucket              ${S3_BUCKET}
    Region              us-east-1
    endpoint            ${S3_ENDPOINT}
    compression         gzip
    s3_key_format       /logs/v1_%Y%m%d%H%M%S_$UUID.gz
    Upload_chunk_size   50M
    static_file_path    On
    use_put_object      Off
    store_dir           /var/log/fluentbit/log/s3/
    store_dir_limit_size   2GB
    retry_limit         5
    retry_exhausted_action  delete
[OUTPUT]
    Name            prometheus_exporter
    match           fluentbit_metrics
    host            ::
    port            2022


BinaryData
====

Events:  <none>
Is that something like S3 Counter Underflow @eduardo
e
Not sure what can be, I would suggest to open a GitHub issue and then we can ping the AWS maintainers
👍 1
p
okay
Opened a Github issue too. We are facing this issue in our high lab environment. Quick help is needed.
e
I am not able to help directly, it's either ask maintainers for help or in the community, if this is prod always is good to consider a commercial engagement with services providers
p
Thank you
h
I investigated that this buffer overflow issue is caused by overflow issue on used buffer size. This can be addressed in github.com/fluent/fluent-bit/pull/12266.
p
thanks. when can we expect in release
Open in Slack
PreviousNext
Powered by