hi folks, we are trying to track down a "memory leak." We had hoped that #11846 would resolve the problem, but unfortunately not.
We queried 6 days of Thanos history for four Fluent Bit pods under normal cluster log volume. All four grew linearly and never recovered:
t92nz: 728.7 MiB to 1602.0 MiB — 139.6 MiB/day, R² 0.999
xl5r8: 603.0 MiB to 1256.6 MiB — 105.3 MiB/day, R² 0.999
hk58z: 522.1 MiB to 957.2 MiB — 69.3 MiB/day, R² 0.998
72jnz: 500.1 MiB to 929.0 MiB — 68.1 MiB/day, R² 1.000
R² of 0.998–1.000 over six days is effectively a perfect straight line with no plateau — the leak signature from #11846, at the ~100 MB/day rate reporters describe.
All four pods were running 5.1.1, confirmed from the image label on the historical series. This matches the dissenting reports on #11846 (users ivan-bushmarinov, TyranVdmKingMakers) rather than the "fixed in 5.1.0" consensus.
Please let me know if there is any information that would be useful for tracking this down.