noisy neighbours problem. - We have multiple micro...
# fluent-bit
s
noisy neighbours problem. • We have multiple micro services running on EKS cluster, each team will have multiple assigned micro services to run and change the code independently. • A Fluent-bit running as Daemonset in EKS cluster which scrapes logs and send them to Loki • Problem - ◦ A few micro services generate logs in high volume and high velocity randomly, no control here. ◦ Sometimes a Developer switch on Debug mode and forgets to off it, which generates 100 GBs of logs in a day. ◦ How can we monitor this situation? How to identify which micro service is making noise by generating huge logs lines in less than a threshold time? ◦ Don't find any metrics to measure, on Fluent-bit-prometheus side, which will measure the file or any label specific activity?
p
Normally I segregate tail inputs by namespace or similar then you can get metrics that way per input plus the inputs can be isolated more from each other. Otherwise you could maybe use filters or outputs that pull specific things you care about and monitor that way. Even a logs to metric filter might work but may be a lot of overhead.
s
• segregate tail inputs by namespace - I have multiple services running in single namespace - will you please guide here? • using filters or outputs that pull specific things - how? as I have a single Daemonset, which will pull all the .log files present on each node of a EKS cluster. When a Developer introduced new service, my Daemonset must scrape it's logs.
p
You're in a pickle really: tail is limited by the kubelet file format which only includes namespace, pod and container name. Can you use any of those? If you cannot segregate the tail inputs then you will always be subject to some form of noisy neighbour issue, e.g. it may add latency to file ingestion if a less critical service logs a lot and delays reading other service logs that way. This is why I tend to do it so I can also configure other useful options for more critical vs less-critical logs (e.g. you may want different buffering or similar). You can potentially segregate on output either with a rewrite_tag or using a grep processor to drop all data apart from that expected for your specific output - use the service name, annotation, whatever you have in the data. That way you can have an output per service, namespace, whatever distinction you choose and then monitor that output, e.g. something like:
Copy code
pipeline:
          inputs:
            - name: tail
              alias: tail_kube_critical
              path: '/var/log/containers/*.log
              processors:
                logs:
                   - name: kubernetes
                     labels: on
                     annotations: on
                     merge_log: on
          outputs:
            - name: forward_service_1
              match: '*'
              processors:
                logs:
                  - name: grep
                    logical_op: or
                    exclude: 
                      - service: ^(SERVICE_1_NAME)$
We should look at this as an issue in your test pipeline as well: changes that break monitoring ideally should never be deployed to production and caught earlier. You can probably do something right now to catch those, e.g. monitoring the output levels to ensure they are consistent in testing before being accepted - this can be done even at the service level potentially to ensure changes do not substantially change the metrics being reported there but certainly at a dev/test/qa cluster level. Otherwise you can use LUA to sample or the logs-to-metrics filter to extract metrics to monitor using some common field (e.g. service name) to provide the metrics. Check the docs for these. Commercially we also provide various options to help here too, including de-duplication and log sampling: https://docs.fluent.do/