Hi Team, Am running fluent bit in kubernetes cluster as daemonset to get logs of each pods in different namespace almost many logs get kubernetes metadata appended to the logs like kubernetes.pod_name, kubernetes.namespace_name, but there are few logs which do not have field entries like kubernetes.pod_name, kubernetes.namespace_name even though they are from the same namespace, here is my fluent bit yaml config pasted How to fix this issue i want the kubernetes metadata appended to all logs.

custom_parsers.conf: >
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name         json-custom
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L%z
        Time_Keep    On
        Decode_Field_As   escaped    message
    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L  
    [PARSER]
        Name          docker-multiline
        Format        regex
        flush_timeout 1000
        Regex         /(?<log>^{"log":"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z.*)/
    [PARSER]
        Name          istio-envoy-proxy
        Format        regex
        Regex         ^\[(?<start_time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)? (?<protocol>\S+)" (?<response_code>[^ ]*) (?<response_flags>[^ ]*) (?<response_code_details>[^ ]*) (?<connection_termination_details>[^ ]*) (?<upstream_transport_failure_reason>[^ ]*) (?<bytes_received>[^ ]*) (?<bytes_sent>[^ ]*) (?<duration>[^ ]*) (?<x_envoy_upstream_service_time>[^ ]*) "(?<x_forwarded_for>[^ ]*)" "(?<user_agent>[^\"]*)" "(?<x_request_id>[^\"]*)" "(?<authority>[^ ]*)" "(?<upstream_host>[^ ]*)" (?<upstream_cluster>[^ ]*) (?<upstream_local_address>[^ ]*) (?<downstream_local_address>[^ ]*) (?<downstream_remote_address>[^ ]*) (?<requested_server_name>[^ ]*) (?<route_name>[^  ]*)
        Time_Format   %Y-%m-%dT%H:%M:%S.%L%z
        Time_Keep     On
        Time_Key      start_time
    [PARSER]	
        Name        cri	
        Format      regex	
        Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
        Time_Key    time	
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
  fluent-bit.conf: >
    [SERVICE]
        Flush         1
        Log_Level     error
        Daemon        off
        Parsers_File  custom_parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    [INPUT]
        Name              tail
        Read_from_Head    True
        Tag               kube.*
        Path              /var/log/containers/*.log
        # Exclude_Path      /var/log/containers/*_lamm_*.log
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     50MB
        Parser            cri
        Refresh_Interval  10

    [FILTER]    
        Name       kubernetes   
        Match      kube.* 
        Keep_Log   Off   
        Merge_Log  On
    
    [FILTER]
        Name record_modifier
        Match *
        Record applicationenv ${APPLICATIONENV}
        Record businessentity ${BUSINESSENTITY}
        Record businessunit ${BUSINESSUNIT}
        Record cluster_name ${CLUSTER_NAME}

    [OUTPUT]
        Name           kafka
        Match          *
        Brokers        "<broker>"
        Topics         <topic>
        # Logstash_Format On
        Timestamp_Key  @timestamp
        Retry_Limit    false
        rdkafka.enable.ssl.certificate.verification false
        rdkafka.request.required.acks 1
        rdkafka.log.connection.close false
        rdkafka.compression.codec snappy
        rdkafka.security.protocol    ssl
        #rdkafka.ssl.ca.location /etc/ssl/certs/ca.crt
        rdkafka.ssl.certificate.location  /etc/ssl/certs/tls.crt
        rdkafka.ssl.key.location  /etc/ssl/certs/tls.key

That metadata usually comes from the filename for the pod log and then extra metadata comes from the K8S API so my guess is that it's failing for some reason to get that information. It may be overloaded or being passed incorrect information, the logs should tell you

hey @Sunil Sylvester I don't know if could help but I am collecting all console logs of all pods of all our Kube clusters too .. and below an example of how I am injecting some data on every logs records the fluentbit config (processors on input tail)

name: tail
          path: /var/log/containers/*.log
          processors:
            logs:
              - buffer: "On"
                multiline.key_content: log
                multiline.parser: java, go, python
                name: multiline
              - action: insert
                key: kubernetes_cluster
                name: content_modifier
                value: ${KUBERNETES_CLUSTER}
              - action: insert
                key: kubernetes_host
                name: content_modifier
                value: ${KUBERNETES_NODE_NAME}
....

we add a lot of additional data and some of them are from the

env

(added to the values of the helm chart) below some examples:

env:
    - name: KUBERNETES_NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: KUBERNETES_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: KUBERNETES_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: KUBERNETES_NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: KUBERNETES_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: KUBERNETES_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
...

Hi, Thanks for all your suggestion, So what i noticed was that kubernetes filter has the buffer of 32kb by default, so when i had set the logging level to info i was getting many warning regarding [ warn] [http_client] cannot increase buffer: current=32000 requested=63768 max=32000. The API response for some pods is ~64KB (large because of many labels, annotations, long container names like the ES ones), Fluent Bit's HTTP client buffer is capped at 32KB The response gets cut in half — the JSON is incomplete, JSON parse fails silently — the kubernetes block is never populated. The record flows through with zero metadata

Yup so sounds like you need to increase the buffer which I think you can do