hey I was updating fluentbit config by adding storagetype fi Fluent Community #fluent-bit

hey, I was updating fluentbit config by adding sto...

Sujay

11/05/2025, 3:07 PM

hey, I was updating fluentbit config by adding storagetype: filesystem

Copy code

storage.max_chunks_up: 1000
  storage.backlog.mem_limit: 500M
  storage.total_limit_size: "10G"

    inputTail:
      Buffer_Chunk_Size: 300k
      Buffer_Max_Size: 500MB
      Ignore_Older: 10m
      Mem_Buf_Limit: 500MB
      storage.type: filesystem
      Rotate_Wait: "30"
      Refresh_Interval: "1"

But started getting timeouts after this we it was uable to connect to fluentd and we saw increase in fluentbit retry metrics and also fluentbit input metrics below is the error can someone help me out

Copy code

24:02] [error] [upstream] connection #1771 to <tcp://172.20.193.46:24240> timed out after 10 seconds (connection timeout) [2025/11/05 13:24:02] [error] [upstream] connection #1773 to <tcp://172.20.193.46:24240> timed out after 10 seconds (connection timeout) [2025/11/05 13:24:02] [error] [upstream]

Pat

11/05/2025, 3:09 PM

I doubt changing the buffering to include filesystem is affecting actually sending data

➕ 1

Sujay

11/06/2025, 8:15 AM

but when i revert the change its working properly but when i switch to filesystem buffering again i face upstream connection error

Sujay

11/06/2025, 8:15 AM

scaling fluentd replicas also didnt help in minimizing [error] [upstream] connection

Sujay

11/06/2025, 9:03 AM

is it possible that im pushing huge chunks and with current replicas of fluentd its causing the problem

Sujay

11/06/2025, 12:26 PM

to debug I checked network connections in fluentd it had increased to max that might be causing this issue is there any way to set max connection in fluentd ? netstat -an 2>/dev/null | grep :24240 | grep ESTABLISHED | wc -l" || echo "unreachable"

Open in Slack

Previous Next