Hello Team, I was exploring the option to drop log...
# fluent-bit
s
Hello Team, I was exploring the option to drop log levels (e.g.,
info
,
warn
,
error
,
debug
) from fluentbit using some filters. Can fluentbit filter help me here or is there anyother way ?
p
Yes, there's a few ways to do it. My advice is to standardise your levels so they ended up converted to a fixed set of values with some filters first then use a grep to drop them. You could also do it all in one lua filter too.
I keep meaning to update with some more recent examples on fluent.do but I'm on holiday at the moment 🙂
s
Hey Pat, Thanks for quick reply. But as of now it is challenge to standardise. So do we have any work around without standardise 🤔
p
My point is you can standardise with filters yeah, a lua one can help. My blog post covers the couchbase logs which at the time were wildly different even just for levels. I did it with a chained series of modify filters but lua would be better I think.
E.g. uppercase to start then handle different strings, etc It's also useful to convert to an integer value for alerting support.
Otherwise you can drop with a grep filter that handles all the different options but standard levels are useful not just for exclusion but downstream too
Look at the docs for grep to see examples of excluding
s
Got it. Let me check the blog and get back to you. And once your back from vacation can you share some more examples
p
My blog links to the actual config as well we used
I took my solution from other advice here a few years ago 😂
A LUA filter though would let you cope with a lot of variants, set defaults when missing, add integer value, etc all in one filter.
s
ohh, Not much familiar with lua at the moment. Let with check that toooo 🧐
p
Yeah it's up to you really what works better
s
Thanks for guidance pat
@Pat i was able to standardise the logs but drop is not working using grep, Do we have any other method to drop logs based on level
p
Please provide more information, grep works fine so how is it not working, What's wrong?
What's your actual config and what data is "not working"?
s
yeah grep function which i tried:
Copy code
filters:
  - grep:
      exclude: "log_processed.level DEBUG|log_processed.level debug|level DEBUG|level debug"

- grep:
    exclude: "(?i)\"level\": \"debug\""
log in rawstring:
Copy code
{
  "tag": "ccsportal.var.log.containers.structured-log-generator-6c56f8956b-7w8cf_demo_logger-ffb6731dbc72e73b02aca6cab7b929a66f04847e8856c91663a05e086d39addc.log",
  "kubernetes": {
    "docker_id": "ffb6731dbc72e73b02aca6cab7b929a66f04847e8856c91663a05e086d39addc",
    "namespace_name": "demo",
    "pod_ip": "demo",
    "master_url": "demo",
    "host": "ip-10-150-30-203.us-west-2.compute.internal",
    "labels": {
      "name": "structured-log-generator",
      "app": "structured-log-generator",
      "clusterName": "ccsdr02",
      "pod-template-hash": "6c56f8956b",
      "version": "demo"
    },
    "container_name": "logger",
    "namespace_id": "demo",
    "container_image_id": "demo",
    "container_hash": "docker.io/library/python@sha256:a40cf9eba2c3ed9226afa9ace504f07ad30fe831343bb1c69f7a6707aadb7c21",
    "pod_name": "structured-log-generator-6c56f8956b-7w8cf",
    "container_image": "docker.io/library/python:3.9-slim",
    "namespace_labels": {
      "kubernetes_io/metadata_name": "demo",
      "app_kubernetes_io/name": "demo",
      "kubernetes_io/managed-by": "demo",
      "name": "demo",
      "istio-injection": "demo"
    },
    "pod_id": "2d7159d8-0a57-4e10-9f97-3fd047dc230f"
  },
  "time": "2025-06-18T11:04:31.064832Z",
  "logtag": "F",
  "level": "debug",
  "log": "{\"time\": \"2025-06-18T11:04:31.064832Z\", \"level\": \"debug\", \"msg\": \"Generated log at level: debug\"}",
  "stream": "stdout",
  "@timestamp": "2025-06-18T11:04:31.064Z",
  "log_processed": {
    "level": "debug",
    "msg": "Generated log at level: debug",
    "time": "2025-06-18T11:04:31.064832Z"
  },
  "docker": {
    "container_id": "ffb6731dbc72e73b02aca6cab7b929a66f04847e8856c91663a05e086d39addc"
  },
  "msg": "Generated log at level: debug",
  "log_collector": "FluentBit"
}
p
What's your full input config and what is the actual data after parsing before the filter? You can use stdout filter to print it out before the grep
s
yes tried to did that
Copy code
[4] ccsportal.var.log.containers.structured-log-generator-6c56f8956b-tpmt4_demo_logger-550678cc53d50e8b2549a87c0fa9dd4ca614b316452f6dd344a165b02ea2eacf.log: [[1750320617.694199000, {}], {"time"=>"2025-06-19T08:10:17.694199Z", "logtag"=>"F", "level"=>"FATAL", "log"=>"{"time": "2025-06-19T08:10:17.694199Z", "level": "fatal", "msg": "Generated log at level: fatal"}", "log_processed"=>{"time"=>"2025-06-19T08:10:17.694199Z", "level"=>"FATAL", "msg"=>"Generated log at level: fatal"}, "@timestamp"=>"2025-06-19T08:10:17.694Z", "msg"=>"Generated log at level: fatal", "docker"=>{"container_id"=>"550678cc53d50e8b2549a87c0fa9dd4ca614b316452f6dd344a165b02ea2eacf"}, "log_collector"=>"FluentBit", "stream"=>"stdout", "kubernetes"=>{"container_name"=>"logger", "namespace_name"=>"demo", "container_hash"=>"docker.io/library/python@sha256:a40cf9eba2c3ed9226afa9ace504f07ad30fe831343bb1c69f7a6707aadb7c21", "pod_name"=>"structured-log-generator-6c56f8956b-tpmt4", "labels"=>{"pod-template-hash"=>"6c56f8956b", "version"=>"demo", "clusterName"=>"ccsdr02", "name"=>"structured-log-generator", "app"=>"structured-log-generator"}, "pod_id"=>"3cdb2812-c2da-4485-8b1c-5aed8acdf7e9", "container_image"=>"docker.io/library/python:3.9-slim", "container_image_id"=>"demo", "namespace_id"=>"demo", "master_url"=>"demo", "pod_ip"=>"demo", "namespace_labels"=>{"kubernetes_io/managed-by"=>"demo", "kubernetes_io/metadata_name"=>"demo", "istio-injection"=>"demo", "name"=>"demo", "app_kubernetes_io/name"=>"demo"}, "host"=>"ip-10-150-9-1.us-west-2.compute.internal", "docker_id"=>"550678cc53d50e8b2549a87c0fa9dd4ca614b316452f6dd344a165b02ea2eacf"}}]
but this doesn't help much
Found a way to drop this using parser script: but i feel this is not the best method. And when i tried to change drop of logs to INFO it takes time to reflect
Copy code
function normalize_all_log_levels(tag, timestamp, record)
        local function canonicalize(val)
            if not val then return nil end
            local lvl = tostring(val):lower()
            if lvl == "error" then return "ERROR"
            elseif lvl == "warn" or lvl == "warning" then return "WARN"
            elseif lvl == "info" then return "INFO"
            elseif lvl == "debug" then return "DEBUG"
            elseif lvl == "fatal" then return "FATAL"
            else return lvl:upper()
            end
        end

        local top_keys = {"logLevel", "level"}
        for _, key in ipairs(top_keys) do
            if record[key] then
                record[key] = canonicalize(record[key])
            end
        end

        if record["log_processed"] then
            for _, key in ipairs(top_keys) do
                if record["log_processed"][key] then
                    record["log_processed"][key] = canonicalize(record["log_processed"][key])
                end
            end
        end

        -- Final log level
        local log_level = record["logLevel"] or record["level"]
        if not log_level and record["log_processed"] then
            log_level = record["log_processed"]["logLevel"] or record["log_processed"]["level"]
        end

        -- Ensure top-level field for grep
        if log_level then
            record["level"] = log_level
        end

        if log_level == "WARN" or log_level == "ERROR" then
            return -1  -- drop
        end

        return 2, timestamp, record
    end