Sagar Nikam
09/04/2025, 4:19 PMPat
09/04/2025, 4:56 PMSagar Nikam
09/05/2025, 1:20 AMPat
09/05/2025, 9:40 AMpipeline:
inputs:
- name: tail
alias: tail_kube_critical
path: '/var/log/containers/*.log
processors:
logs:
- name: kubernetes
labels: on
annotations: on
merge_log: on
outputs:
- name: forward_service_1
match: '*'
processors:
logs:
- name: grep
logical_op: or
exclude:
- service: ^(SERVICE_1_NAME)$
We should look at this as an issue in your test pipeline as well: changes that break monitoring ideally should never be deployed to production and caught earlier. You can probably do something right now to catch those, e.g. monitoring the output levels to ensure they are consistent in testing before being accepted - this can be done even at the service level potentially to ensure changes do not substantially change the metrics being reported there but certainly at a dev/test/qa cluster level.
Otherwise you can use LUA to sample or the logs-to-metrics filter to extract metrics to monitor using some common field (e.g. service name) to provide the metrics. Check the docs for these.
Commercially we also provide various options to help here too, including de-duplication and log sampling: https://docs.fluent.do/