Hi Team, I have an interesting behaviour to report...
# fluent-bit
u
Hi Team, I have an interesting behaviour to report for Fluentbit 4.0.8. Based on the Metrics I feel that Fluentbit is not honoring the Memory Buffer Overlimit for the Tail Plugin. I am using conf based configuration. As per my understanding, If I have set the Memory Buffer Overlimit to 512 MB, the Memory Utilization cannot exceed this range + some buffer. However, I am seeing a constant rise in memory of the Pod upto 1.4G (Which is the limit and the Pod crashes). I have never seen this behaviour with Fluentbit 3.0.7 with the same configuration. Attaching some screenshots here. Also attaching my configuration in this thread. Has anyone seen this or any pointers on what to do? I think this is a bug
FLB Confiration
Copy code
[SERVICE]
    Flush         1
    Log_Level     warn
    Daemon        off
    Parsers_File  parsers.conf
    HTTP_Server   On
    HTTP_Listen   0.0.0.0
    HTTP_Port     2020
    storage.path      /var/tmp/o11y/buffer
    storage.max_chunks_up 500
    storage.delete_irrecoverable_chunks On
    storage.pause_on_chunks_overlimit Off
    storage.backlog.mem_limit 500M
@INCLUDE input-kubernetes.conf
@INCLUDE output-s3.conf

[INPUT]
    Name             fluentbit_metrics
    Tag              internal_metrics
    Scrape_Interval  2
[INPUT]
    Name              tail
    Tag               app.*
    Path              /var/log/pods/*/*/*.log
    Parser            containerd
    DB                /var/tmp/o11y/app.db
    Mem_Buf_Limit     512MB
    Read_from_Head    True
    Refresh_Interval  2
    Buffer_Chunk_Size 18k
    Buffer_Max_Size   64k
    DB.locking        True
    DB.sync           Normal
    Rotate_Wait       60
    Skip_Empty_Lines  On
    File_Cache_Advise Off
[INPUT]
    Name              tail
    Tag               com.iks-compliance.audit
    Path              /var/log/audit/audit.log
    DB                /var/tmp/o11y/sys_audit.db
    DB.Sync           Normal
    Mem_Buf_Limit     16MB
    Read_from_Head    True
    Buffer_Chunk_Size 4k
    Buffer_Max_Size   64k
    Skip_Long_Lines   On
    File_Cache_Advise Off
[INPUT]
    Name              tail
    Tag               com.iks-compliance.secure
    Path              /var/log/secure
    DB                /var/tmp/o11y/var_secure.db
    DB.Sync           Normal
    Mem_Buf_Limit     16MB
    Read_from_Head    True
    Buffer_Chunk_Size 4k
    Buffer_Max_Size   64k
    Skip_Long_Lines   On
    File_Cache_Advise Off
[INPUT]
    Name              tail
    Tag               system.*
    Path              /var/log/kubelet.log,/var/log/containerd.log,/var/log/messages,/var/log/cron,/var/log/aws-routed-eni/*.log,/var/log/iptables-diff.log
    DB                /var/tmp/o11y/kubelet.db
    DB.Sync           Normal
    Mem_Buf_Limit     32MB
    Read_from_Head    True
    Buffer_Chunk_Size 4k
    Buffer_Max_Size   64k
    Skip_Long_Lines   On
    File_Cache_Advise Off

[OUTPUT]
    Name                         s3
    Match                        app.*
    bucket                       ${APP_BUCKET_NAME}
    region                       ${BUCKET_REGION}
    total_file_size              60M
    retry_limit                  10
    upload_timeout               20s
    log_key                      msg
    log_level                    warn
    net.keepalive_idle_timeout   60
    tls.verify                   False
    compression                  gzip
    use_put_object               True
    store_dir                    /var/tmp/o11y/s3
    store_dir_limit_size         5G
    role_arn                     ${ROLE}
    s3_key_format                /${HEC_ENDPOINT}/iks/raw/iks2/v2/${CLUSTER_NAME}/${AZ}/${K8S_NODE}/$TAG[4]/$TAG[5]/stdout/unknown/%H%M%S-$UUID.gz
    workers                      1
[OUTPUT]
    Name                         s3
    Match                        com.iks-compliance.audit
    bucket                       ${SYS_BUCKET_NAME}
    region                       ${BUCKET_REGION}
    total_file_size              60M
    retry_limit                  10
    upload_timeout               60s
    log_key                      log
    log_level                    warn
    tls.verify                   False
    net.keepalive_idle_timeout   60
    compression                  gzip
    use_put_object               True
    store_dir                    /var/tmp/o11y/s3
    store_dir_limit_size         1G
    role_arn                     ${ROLE}
    s3_key_format                /${HEC_ENDPOINT}/iks/raw/generic/$TAG[1]/$TAG[2]/${K8S_NODE}/iks-sys_${CLUSTER_NAME}_compliance.var.log.audit.audit.log/%H%M%S-$UUID.gz
[OUTPUT]
    Name                         s3
    Match                        com.iks-compliance.secure
    bucket                       ${SYS_BUCKET_NAME}
    region                       ${BUCKET_REGION}
    total_file_size              60M
    retry_limit                  10
    upload_timeout               60s
    log_key                      log
    log_level                    warn 
    tls.verify                   False
    net.keepalive_idle_timeout   60
    compression                  gzip
    use_put_object               True
    store_dir                    /var/tmp/o11y/s3
    store_dir_limit_size         1G
    role_arn                     ${ROLE}
    s3_key_format                /${HEC_ENDPOINT}/iks/raw/generic/$TAG[1]/$TAG[2]/${K8S_NODE}/iks-sys_${CLUSTER_NAME}_compliance.var.log.secure/%H%M%S-$UUID.gz
[OUTPUT]
    Name                         s3
    Match                        system.*
    bucket                       ${SYS_BUCKET_NAME}
    region                       ${BUCKET_REGION}
    total_file_size              60M
    log_level                    warn
    retry_limit                  10
    upload_timeout               60s
    log_key                      log
    tls.verify                   False
    net.keepalive_idle_timeout   60
    compression                  gzip
    use_put_object               True
    store_dir                    /var/tmp/o11y/s3
    store_dir_limit_size         1G
    role_arn                     ${ROLE}
    s3_key_format                /${HEC_ENDPOINT}/iks/raw/generic/iks/fluent/${K8S_NODE}/iks-sys_${CLUSTER_NAME}_$TAG/%H%M%S-$UUID.gz
[OUTPUT]
    Name                         prometheus_exporter
    Match                        internal_metrics
    Host                         0.0.0.0
    Port                         2021

[PARSER]
    Name              containerd
    Format            regex
    Regex             ^\d{4}-(0[1-9]|1[0-2])-([0-2][0-9]|3[01])T([0-1][0-9]|2[0-4])(:[0-5][0-9]){2}(?:.\d{1,10})?Z (?<stream>stdout|stderr) [F|P] (?<msg>.*)$
    Time_Key          nil
    Skip_Empty_Values Off
Pod Limits :- 1.5G Memory and 2 Core CPU. Load Tested - 1. 2000 char log lines : 7.5k TPS 2. 20,000 char lines : 2.5k TPS 10k TPS total with varies log lines
l
Hello. Can you share logs from problematic run? Are you seeing any backpressure? How the output looks compared to the input rate?
u
Hi @lecaros You can ignore this thread, I had unknowingly given 75k TPS instead of 7.5k. There definitely was back pressure due to which it crashed