*`Issue: Include Pod Name in Fluent Bit Logs`* `...
# fluent-bit
a
Issue:   Include Pod Name in Fluent Bit Logs
Hi Team
, We're currently forwarding NGINX ingress controller
logs to Elasticsearch via Fluent Bit. However, the logs are missing the *pod name* of the container emitting the logs, which is important for debugging and filtering in Kibana.
Current Setup Highlights:
•
Input tag: ingress
•
Logs parsed via json_parser
•
Output to Elasticsearch (ingress_staging-* index)
Action Taken:
I'm planning to add the following filter to enrich logs with Kubernetes metadata:
Copy code
ini
Copy code
[FILTER]
    Name kubernetes
    Match ingress
    Kube_Tag_Prefix ingress.var.log.containers.
    Merge_Log On
    Keep_Log Off
G*`oal:`*
Add pod name (kubernetes.pod_name) to each log record.
Please let me know if:
•
You're already handling this differently
•
There's a cleaner or more efficient way
•
You see any compatibility issues with our current Fluent Bit chart
our configuration :
inputs: |
[INPUT] Name tail Tag ingress Path /var/log/containers/nginx-ingress-nginx-controller*.log Mem_Buf_Limit 50MB Parser json_parser [FILTER] Name parser Match ingress Parser json_parser Key_Name log [FILTER] Name type_converter Match ingress str_key status status int str_key bytes_sent bytes_sent int str_key body_bytes_sent body_bytes_sent uint str_key request_time request_time float str_key connection connection int str_key connection_requests connection_requests int str_key upstream_connect_time upstream_connect_time float str_key upstream_header_time upstream_header_time float str_key upstream_response_length upstream_response_length uint str_key upstream_response_time upstream_response_time float str_key upstream_status upstream_status int str_key request_length request_length int customParsers: | [PARSER] Name json_parser Format json Decode_Field_As escaped log ## https://docs.fluentbit.io/manual/pipeline/outputs outputs: | [OUTPUT] Name es Match ingress Host xxxxxx Port 80 http_user ddddd http_passwd ddddd Retry_Limit False Index ingress_staging Type _doc Include_Tag_Key ON Logstash_Format true Logstash_Prefix ingress_staging Logstash_Dateformat %Y.%m.%d Tag_Key tag Buffer_Size 500M
p
I think you've messed up your formatting a bit
However, for this the main thing to look at is
log_level debug
to see the HTTP requests and responses to the K8S API server which almost always tells you the problem.
And you have a mismatched config for
kube_tag_prefix
- it does not match your actual tag which is just
ingress
and it is missing the actual log file name which is required by the
kubernetes
filter to make a query to the K8S API. I guarantee that
log_level debug
would show you that the query request is using nonsense values for pod name, etc.
If you're using kubelet logs which I'm guessing you are I think you also need to handle that format in your tail input - it's not JSON formatted files on disk usually.
Do something like this and check the output contains what you want?
Copy code
[INPUT]
  Name tail
  tag kube.*
  path /var/log/containers/nginx-ingress-nginx-controller*.log
  multiline.parser docker,cri

[FILTER]
  Name kubernetes
  match kube.*
  Merge_log on

[OUTPUT]
  name stdout
  match *
Pod name, namespace and container ID come from the log filename on disk which you're not preserving in the tag as you do not have a wildcard
I'm writing another article on this but I also did a webinar a while back about how to use the K8S filter properly.
https://www.fluent.do/blog/kubernetes-logging may explain it a bit better, just updated that today