Hello Team, I see lot of log line drops not seeing error or errors from upstream loki not sure if something is wrong with config. Can anyone help me on this ? New to fluentbit, evaluating few other options if this doesn't work for us. Initially it started working but later input bytes started reducing even though apps are writing logs.

apiVersion: v1
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    debug
        Parsers_File /fluent-bit/etc/parsers.conf
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020
        Storage.path              /fluent-bit/logs
        Storage.sync              normal
        Storage.checksum          off
        Storage.metrics           on
        Storage.backlog.mem_limit 128MB
        Storage.max_chunks_up     2048

    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Tag               kube.*
        Parser            cri
        Parser_Fallback   docker_json
        Refresh_Interval  5
        Buffer_Max_Size   64MB
        Buffer_Chunk_Size 2MB
        Mem_Buf_Limit     1GB
        Skip_Long_Lines   On
        DB                /var/log/flb_kube.db
        DB.Sync           full
        Storage.type      filesystem

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            <https://kubernetes.default.svc:443>
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           Off
        Keep_Log            On
        Annotations         Off
        Labels              On

    [FILTER]
        Name  modify
        Match *
        Add   project qa
        Add   cluster qa

    [OUTPUT]
        Name   stdout
        Match  * 
        Format json_lines

    [OUTPUT]
        Name        loki
        Match       kube.*
        Host        test.com
        Port        443
        Tenant_ID   qa
        tls         On
        tls.verify  Off
        Uri         /loki/api/v1/push
        Compress    gzip
        Workers     32
        Retry_Limit 5
        Labels      job=fluent-bit,project=$project,cluster=$cluster,namespace=$kubernetes['namespace_name'],pod=$kubernetes['pod_name'],container=$kubernetes['container_name'],app=$kubernetes['labels']['app.kubernetes.io/name'],tier=$kubernetes['labels']['tier'],release=$kubernetes['labels']['release'],host=$kubernetes['host']
        storage.total_limit_size   8G
        Auto_Kubernetes_Labels On
        net.keepalive             On
        net.connect_timeout       10s
        net.keepalive_idle_timeout 30s

  parsers.conf: |
    # Docker JSON (fallback)
    [PARSER]
        Name        docker_json
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%LZ
        Time_Keep   On

    # CRI / containerd (primary)
    [PARSER]
        Name        cri
        Format      regex
        Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<flags>[^ ]*) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
        Decode_Field_As  escaped_utf8  log  do_next
        Decode_Field_As  json          log

What do the logs for Fluent Bit say? What do the metrics say, is it getting output errors? If so then that'll be down to Loki

Loki is not down, I am able to ingest from other agents without any back pressure

Sure but what are the answers to the above? Is Fluent Bit reporting errors and what are they? Whilst it may not be down it may still be rejecting or not handling the messages for some reason. Basically trying to narrow down what the issue is - are there any other problems reported in the logs, e.g. backpressure? https://docs.fluentbit.io/manual/administration/backpressure

Yes I saw this error

[warn] [input] {input name or alias} paused (mem buf overlimit)
[info] [input] {input name or alias} resume (mem buf overlimit)

later I configured to use disk fill / storage configuration since then I don't see any error on back pressure but writes to loki is bit slow so most of the logs are getting queued in storage.

why are you setting buffer sizes and mem limits? what are you expecting from such configs? how did you come to such numbers?

I read this document and tuned https://docs.fluentbit.io/manual/administration/backpressure

Increasing buffer sizes will not help with mem buf overlimit. It can contribute to actually use more memory and cause delays in flushing data

@lecaros, I'm facing same issue do you recommend any tuning in config for this

hi there. I can't make any recommendations without knowing your context. You can start a new thread and provide your config, logs, version, etc.

sure, but what do you recommend for this thread for the config what Nitesh has given

we still don't have answers to all the questions, but I'd start by removing the buffer settings