Hi Team, We have a deployment where a single pod i...
# fluent-bit
r
Hi Team, We have a deployment where a single pod includes 5 sidecar containers, and we're using Fluent Bit to collect logs. However, we’ve observed that when the pod has 5 sidecars, the logs do not appear with Kubernetes metadata such as pod name, container name, etc. Interestingly, when we reduce the number of sidecars to 3, the logs are correctly ingested and visible in OpenSearch (Kibana) with all expected metadata. Could you please help us identify what might be going wrong with the configuration? Do we need to apply any OS-level tuning or adjustments to Fluent Bit to support more sidecars? Fluent bit version "3.0.6"
Copy code
[INPUT]
         Name tail
         Path /var/log/containers/v2-xxxxxxx*.log
         Tag v2-xxxxxxx.*
         Parser cri-new
         DB /var/log/flb_kube_v2int.db
         Mem_Buf_Limit 50MB
         Skip_Long_Lines On
         Refresh_Interval 10
[FILTER]
         Name kubernetes
         Match v2-xxxxxxx.*
         Kube_URL https://kubernetes.default.svc:443
         Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
         Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
         Kube_Tag_Prefix v2-xxxxxxx.var.log.containers.
         Merge_Log On
         Merge_Log_Key log_processed
         Keep_Log Off
         K8S-Logging.Parser On
         K8S-Logging.Exclude Off
         Labels On
[OUTPUT]
         Name es
         Match v2-xxxxxxx.*
         Logstash_Prefix v2-xxxxxxx
         Host opensearch-cluster-xxxxxx.yyyyyyy.svc.cluster.local
         Port 9200
         HTTP_User ${OPENSEARCH_USER}
         HTTP_Passwd ${OPENSEARCH_PASSWORD}
         tls On
         tls.verify Off
         Suppress_Type_Name On
         Logstash_Format On
         Trace_Error On
p
The kubernetes filter is intended to work with kubelet logs, the log filename must include the relevant information to look up the metadata from the API server
I did some talks and blog posts on it with examples but me guess is the info is not right for the lookup and so no metadata is returned
Debug logs will show you the failure and why
r
@Pat Yes... but don't know why it is working for 3 sidecars not for 5 sidecars. Any idea on this?
p
Check the logs for those 5, I bet the API server is not giving you the info for them for some reason
Debug logs should show you the query and the response
You're also on an old unsupported version so I'd try the latest to confirm it's not a resolved issue
c
yeah what fluent bit is doing basically request metadata information from kubernetes api-server and use response to enrich logs, I don't think it's related to sidecar count or what sidecar is doing. Please just check filter doc and be sure while adding new sidecar you don't also change RBAC or mounting volumes filter references etc. Like Pat suggested first thing yo do run in debug log level and see the error
r
I didn't see any errors in the logs, but I was able to retrieve metadata using the
Use_Tag_For_Meta
filter. However, it's still unclear why metadata is not being collected when the pod has 5 sidecars. When limited to 3 sidecars, everything works as expected. But with 5, no metadata is collected for any container—including the pod itself. For now i am going with
Use_Tag_For_Meta
this filter to debug app services logs. Thanks for help. @Celalettin @Pat
p
Without RCA I personally would be concerned there's some underlying intermittent issue in your infra causing the problem that may occur again. Are your pod definitions fixed or are you adding dynamic sidecars? The filter caches responses to prevent API limiting or throttling issues so potentially if you add sidecars after the initial query they won't be known about until the next time it refreshed.
r
Our pod definitions are fixed here. But if i checked for this demo deployment the logs are getting with k8s metadata,
Copy code
apiVersion: apps/v1
kind: Deployment
metadata:
  name: v2-rohit-sidecar
  labels:
    app: v2-rohit-sidecar
spec:
  replicas: 1
  selector:
    matchLabels:
      app: v2-rohit-sidecar
  template:
    metadata:
      labels:
        app: v2-rohit-sidecar
    spec:
      containers:
        - name: v2-rohit-primary
          image: busybox
          command: ["/bin/sh", "-c"]
          args: ['while true; do echo "{\"level\":\"info\",\"msg\":\"Main container running\"}"; sleep 5; done']

        - name: v2-rohit-sidecar1
          image: busybox
          command: ["/bin/sh", "-c"]
          args: ['while true; do echo "{\"level\":\"info\",\"msg\":\"v2-rohit-sidecar 1 active\"}"; sleep 5; done']

        - name: v2-rohit-sidecar2
          image: busybox
          command: ["/bin/sh", "-c"]
          args: ['while true; do echo "{\"level\":\"info\",\"msg\":\"v2-rohit-sidecar 2 active\"}"; sleep 5; done']

        - name: v2-rohit-sidecar3
          image: busybox
          command: ["/bin/sh", "-c"]
          args: ['while true; do echo "{\"level\":\"info\",\"msg\":\"v2-rohit-sidecar 3 active\"}"; sleep 5; done']

        - name: v2-rohit-sidecar4
          image: busybox
          command: ["/bin/sh", "-c"]
          args: ['while true; do echo "{\"level\":\"info\",\"msg\":\"v2-rohit-sidecar 4 active\"}"; sleep 5; done']

        - name: v2-rohit-sidecar5
          image: busybox
          command: ["/bin/sh", "-c"]
          args: ['while true; do echo "{\"level\":\"info\",\"msg\":\"v2-rohit-sidecar 5 active\"}"; sleep 5; done']

        - name: v2-rohit-sidecar6
          image: busybox
          command: ["/bin/sh", "-c"]
          args: ['while true; do echo "{\"level\":\"info\",\"msg\":\"v2-rohit-sidecar 6 active\"}"; sleep 5; done']
Also, if we add 5 sidecars, no metadata is retrieved for any of the containers or the pod. However, when we use only 3 sidecars, it works correctly.
c
I don't under to be honest above 5 busybox sidecars are reproducer of with 5 sidecar kubernetes filter doesn't work or opposite works from your messages. I can think of tens of edge cases like let's say if you're adding proxy sidecar to you deployment fluent bit call through k8s api-server can be blocked. We don't know your deployment we don't know your sub resources or extra sidecars you're adding so if you have basic reproducer that fluent bit maintainers can reproduce quickly recommend you to create an issue with reproduction steps and other details like logs etc. Anyway it looks like you found a workaround.
what fluent bit kubernetes filter is doing just straightforward. It sends requests to k8s api-server. Requirements for this are clear. you should just check yourself why it doesn't access
r
Sure. Thanks @Celalettin