Padma
09/19/2025, 12:41 PMPadma
09/19/2025, 12:42 PMPadma
09/19/2025, 12:42 PM[SERVICE]
Flush 120
Log_Level debug
Daemon off
Parsers_File parsers.conf
HTTP_Server ${HTTP_SERVER}
HTTP_Listen 0.0.0.0
HTTP_Port ${HTTP_PORT}
storage.path /var/fluent-bit/state/flb-storage/
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 50M
storage.max_chunks_up 512
[INPUT]
Name fluentbit_metrics
Tag fluentbit_metrics
scrape_interval 30
[INPUT]
Name tail
Tag kubernetes.logs.*
Path /var/log/containers/*.log
Docker_Mode On
Docker_Mode_Flush 5
Docker_Mode_Parser container_firstline
Parser docker
DB /var/fluent-bit/state/flb_container.db
Skip_Long_Lines On
Refresh_Interval 5
Rotate_Wait 30
storage.type filesystem
Read_from_Head false
Threaded true
Buffer_Chunk_Size 10M
Buffer_Max_Size 15M
File_Cache_Advise Off
[FILTER]
Name record_modifier
Match *
uuid_key _hash
Record site_name ${SITE_NAME}
Record node_name ${NODE_NAME}
Record cluster_name ${CLUSTER_NAME}
Record k8s_cluster ${k8s_cluster}
[FILTER]
Name kubernetes
Match kubernetes.logs.*
tls.verify Off
Kube_URL <https://kubernetes.default.svc.cluster.local:443>
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kubernetes.logs.var.log.containers
Merge_Log On
Labels Off
Annotations Off
Buffer_Size 0
Use_Kubelet true
Kubelet_Port 10250
[FILTER]
Name lua
Match *
script append_time.lua
call append
[FILTER]
Name nest
Match kubernetes.logs.*
Operation lift
Nested_under kubernetes
[FILTER]
Name modify
Match *
Copy container_name application_name
Copy namespace_name namespace
[FILTER]
Name modify
Match kubernetes.logs.*
Add tag docker.all
[OUTPUT]
Name s3
Match_regex ^(kubernetes.logs.*)$
Bucket ${S3_BUCKET}
Region us-east-1
endpoint ${S3_ENDPOINT}
compression gzip
s3_key_format /logs/site_name=${SITE_NAME}/cluster_name=${CLUSTER_NAME}/year=%Y/month=%m/day=%d/hour=%H/minute=%M/v1_%Y%m%d%H%M%S_$UUID.gz
Upload_chunk_size 50M
static_file_path On
use_put_object Off
storage.total_limit_size 20G
store_dir /var/log/fluentbit/log/s3/
store_dir_limit_size 2GB
[OUTPUT]
Name prometheus_exporter
match fluentbit_metrics
host 0.0.0.0
port 2022Ujjwal Sharma
09/19/2025, 1:04 PMAnurag Gupta
09/22/2025, 3:39 AMPadma
09/22/2025, 5:04 AMPadma
09/22/2025, 5:05 AMnaresh amrutham
09/22/2025, 10:07 PMAnurag Gupta
09/23/2025, 9:34 PMWe are seeing log loss in s3 when pulling the dataIs the data missing in S3?
Padma
09/24/2025, 6:06 AMnaresh amrutham
09/24/2025, 7:51 AMwhen we validate there is no logsso the logs are completely missing?
Padma
09/24/2025, 8:11 AMnaresh amrutham
09/24/2025, 6:27 PMPadma
09/26/2025, 2:54 PMPadma
09/26/2025, 2:54 PMPadma
09/26/2025, 6:18 PMPadma
09/26/2025, 6:19 PMnaresh amrutham
09/27/2025, 7:06 PMdebug, I assume there should be something in the logs.Padma
09/29/2025, 6:54 AMnaresh amrutham
09/30/2025, 7:06 PM