Hi Team,Fluent bit on high incoming load we see lo...
# fluent-bit
p
Hi Team,Fluent bit on high incoming load we see logs missing We have file system as storage.During high load fluentbit_storage_fs_chunks_up goes to max limit configured 512 and the chunks goes to down state. fluent_bit_dropped_records_total does not show any dropped records but still we see the data loss. How can we tune fluent-bit to the high load of 800K /min. fluent-bit version used is 4.0.3
fluent-bit is running as dameonset
Copy code
[SERVICE]
        Flush                     120
        Log_Level                 debug
        Daemon                    off
        Parsers_File              parsers.conf
        HTTP_Server               ${HTTP_SERVER}
        HTTP_Listen               0.0.0.0
        HTTP_Port                 ${HTTP_PORT}
        storage.path              /var/fluent-bit/state/flb-storage/
        storage.sync              normal
        storage.checksum          off
        storage.backlog.mem_limit 50M
        storage.max_chunks_up     512
    [INPUT]
        Name                fluentbit_metrics
        Tag                 fluentbit_metrics
        scrape_interval     30
    [INPUT]
        Name                tail
        Tag                 kubernetes.logs.*
        Path                /var/log/containers/*.log
        Docker_Mode         On
        Docker_Mode_Flush   5
        Docker_Mode_Parser  container_firstline
        Parser              docker
        DB                  /var/fluent-bit/state/flb_container.db
        Skip_Long_Lines     On
        Refresh_Interval    5
        Rotate_Wait         30
        storage.type        filesystem
        Read_from_Head      false
        Threaded            true
        Buffer_Chunk_Size   10M
        Buffer_Max_Size     15M
        File_Cache_Advise   Off
    [FILTER]
        Name                record_modifier
        Match               *
        uuid_key _hash
        Record              site_name ${SITE_NAME}
        Record              node_name ${NODE_NAME}
        Record              cluster_name ${CLUSTER_NAME}
        Record              k8s_cluster ${k8s_cluster}
    [FILTER]
        Name                kubernetes
        Match               kubernetes.logs.*
        tls.verify          Off
        Kube_URL            <https://kubernetes.default.svc.cluster.local:443>
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kubernetes.logs.var.log.containers
        Merge_Log           On
        Labels              Off
        Annotations         Off
        Buffer_Size         0
        Use_Kubelet         true
        Kubelet_Port        10250
    [FILTER]
        Name                lua
        Match               *
        script              append_time.lua
        call                append
    [FILTER]
        Name nest
        Match kubernetes.logs.*
        Operation lift
        Nested_under kubernetes
    [FILTER]
        Name modify
        Match *
        Copy container_name application_name
        Copy namespace_name namespace
    [FILTER]
        Name                modify
        Match               kubernetes.logs.*
        Add                 tag docker.all
    [OUTPUT]
        Name                s3
        Match_regex         ^(kubernetes.logs.*)$
        Bucket              ${S3_BUCKET}
        Region              us-east-1
        endpoint            ${S3_ENDPOINT}
        compression         gzip
        s3_key_format       /logs/site_name=${SITE_NAME}/cluster_name=${CLUSTER_NAME}/year=%Y/month=%m/day=%d/hour=%H/minute=%M/v1_%Y%m%d%H%M%S_$UUID.gz
        Upload_chunk_size   50M
        static_file_path    On
        use_put_object      Off
        storage.total_limit_size 20G
        store_dir           /var/log/fluentbit/log/s3/
        store_dir_limit_size   2GB
    [OUTPUT]
        Name            prometheus_exporter
        match           fluentbit_metrics
        host            0.0.0.0
        port            2022
u
Is there backpressure from the s3 plugin? I am able to ship 30k per second without any log losses. It also depends on the size of the log lines too
a
Where are you seeing the log loss?
p
We are seeing log loss in s3 when pulling the data
Backpressure from s3 plugin is there a way to identify any metrics available.
n
To confirm, the input is file system and output is S3? Do you see any error counts in the any of the filters? If possible, can you run without any filters? Also, curious to know how did you determine the log loss?
a
We are seeing log loss in s3 when pulling the data
Is the data missing in S3?
p
Data is missing in s3.. input is filesystem and outpue is s3. we get the logs from s3 and when we validate there is no logs. there is no way to identify from metrics as the output dropped and errors total does not indicate there is loss
n
when we validate there is no logs
so the logs are completely missing?
p
It is completely missing when high income of load appearing over (for 15-20 minutes)post that we could see data is available when load spike is back to normal
n
This is weird, I would expect some data to be outputted. Not a regular user of AWS but do you think this could be due to AWS S3 api throttling?
p
We could see fluentbit_input_storage_chunks_busy reduced over the time when input storage chunks are high and fschunks went to down state
Apart from that s3 throttling anything not seen
@naresh amrutham anything to check over on backpressure
and performance tuning
n
Are there any errors related to S3 or error metrics for the Output stage S3 in Fluent Bit? I see you have already turned-on
debug
, I assume there should be something in the logs.
p
There is no error in the logs
n
unfortunately, I don't have aws account to repro this. But I highly suspect this is due to the S3 throttling.