Hey fellow Fluent Bit users! I just ran into the f...
# fluent-bit
j
Hey fellow Fluent Bit users! I just ran into the following problem today, and I'd love to hear your thoughts and insights on what could be happening: For context, I've installed Fluent Bit in most of my Google Cloud virtual machine (VM) instances for collecting & sending logs & metrics to other VMs, Grafana being one of them. When I looked at my Grafana dashboards earlier today, I saw that some of them said No data. As the first step of my troubleshooting process, I SSH-ed into the first VM whose corresponding Grafana dashboard said No data and examined Fluent Bit's state by running the following command:
systemctl status fluent-bit.service
. And I got the following output in the terminal:
Copy code
root@dev-server:~# systemctl status fluent-bit.service 
ā—‹ fluent-bit.service - Fluent Bit
     Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; preset: enabled)
     Active: inactive (dead)
       Docs: <https://docs.fluentbit.io/manual/>
root@kilogram-dev-1:~#
As soon as I started Fluent Bit by running
systemctl start fluent-bit.service
, I could start seeing metrics coming into my Grafana dashboard. This was really surprising to me, as I never stopped Fluent Bit in those VMs. So here's my question: What are some possible reasons why Fluent Bit would stop on their own?
p
I think this is going to be hard to solve without more information. The logs should show you some more details, e.g. if it crashed or there was some kind of issue. Did you check if it was stopped? It looks like it died though to me. Did the VM get restarted or similar? It looks like the service is bit enabled so won't restart by default. Is it the latest version? If not then upgrade as well.
šŸ’” 1
j
Yeah Pat, I agree that I need to provide more information. I did check out Fluent Bit's own logs, but because I set the log level to
error
, the most recent log I'm seeing is from 8/6/25, which is way before the death of the Fluent Bit agent. Here's a snippet of my `/etc/fluent-bit/fluent-bit.conf`:
Copy code
[SERVICE]
    Flush           1
    Log_Level       error
    Log_File        /var/log/fluent-bit/fluent-bit.log
If you were me, in order to catch future deaths like this, what would you set the log level to instead? Fluent bit wasn't stopped by anyone, since I've been the only person working on it. My teammate said the VMs didn't get restarted, but I'm going to review the Google Cloud audit logs to see what those VMs went through exactly. It looks like the Fluent Bit version is
v4.0.4
in one of those VMs, but the latest version is
v4.0.7
. I'll go ahead and update it right now. Is there a way for me to sign up/subscribe for future updates so I can keep my Fluent Bit version up-to-date as always? And I've subscribed to future releases by watching the Fluent Bit GitHub repo.
p
Monitor metrics to catch failures would be my suggestion.
šŸ‘ 1
I also would not trust what people say or think happened šŸ™‚ And I would enable the service to start on boot
j
Valid. Yeah, I double checked whether the VM was restarted as soon as I talked to my teammate, just to be sure.