Hi team, I'm running Fluent Bit via the Fluent Ope...
# fluent-bit
s
Hi team, I'm running Fluent Bit via the Fluent Operator and using a Lua filter to drop logs based on log level. Here's a snippet of my filter config:
Copy code
- lua:
    call: normalize_product_logs
    script:
      name: fluent-bit-modify-k8s-fields
      key: custom.lua
The
custom.lua
script uses environment variables like
drop_error_product
,
drop_debug_product
, etc., to control which logs should be dropped dynamically. For example:
Copy code
if log_level == "ERROR" and drop_error_product == "true" then
  return -1  -- drop log
end
These variables are passed via environment variables in the Fluent Bit container or set via config maps. Problem When I update the values of these variables (e.g., setting
drop_error_product
from
false
to
true
) and confirm that the
ConfigMap
is updated and pods are restarted manually, the updated logic does not take effect. However, the moment I: • remove the entire Lua block from the filter config and deploy • then add the same Lua block back and deploy again ...the changes start reflecting and logs get dropped correctly. Is Fluent Bit or the Lua plugin caching the script or its environment somewhere internally? Is there a supported or recommended way to force Fluent Bit to reload updated Lua script logic and/or environment config? Thanks a lot — any insights or best practices would be appreciated!
p
Changing environment variables always requires you to recreate pods, that's a K8S thing. Fluent Bit does not cache LUA, my suspicion would be that the mounted LUA is not being changed by the helm chart. Did you confirm that? The config uses the helm trick of an annotation to roll the pods if it changes but wondering if something funky is happening with the separate mount for the LUA scripts.
Ah this is operator, sorry I thought you were using helm
s
i tried to recreate the pod but that doesn't help. But when i try to edit the Clusterfilter by remove my lua block and adding the same block again things start to work
p
I would look at the actual config map being used to see what is going on
My point was if you change environment variables then you have to recreate the pod, there's nothing that Fluent Bit can do - K8S sets the env vars up.
s
here is config map in my helm
Copy code
function normalize_product_logs(tag, timestamp, record)
        local function canonicalize(val)
            if not val then return nil end
            local lvl = tostring(val):lower()
            if lvl == "error" then return "ERROR"
            elseif lvl == "warn" or lvl == "warning" then return "WARN"
            elseif lvl == "info" then return "INFO"
            elseif lvl == "debug" then return "DEBUG"
            elseif lvl == "fatal" then return "FATAL"
            else return lvl:upper()
            end
        end

        -- Canonical drop list
        local drop_levels = {}
        {{- if .Values.log_filters.drop_debug_product }} drop_levels["DEBUG"] = true {{- end }}
        {{- if .Values.log_filters.drop_info_product }} drop_levels["INFO"] = true {{- end }}
        {{- if .Values.log_filters.drop_warn_product }} drop_levels["WARN"] = true {{- end }}
        {{- if .Values.log_filters.drop_error_product }} drop_levels["ERROR"] = true {{- end }}
        {{- if .Values.log_filters.drop_fatal_product }} drop_levels["FATAL"] = true {{- end }}

        local log_level = nil
        local top_keys = {"logLevel", "level"}

        for _, key in ipairs(top_keys) do
            local val = record[key]
            if val then
                local canon = canonicalize(val)
                if drop_levels[canon] then
                    record[key] = canon
                    log_level = canon
                end
            end
        end

        if record["log_processed"] then
            for _, key in ipairs(top_keys) do
                local val = record["log_processed"][key]
                if val then
                    local canon = canonicalize(val)
                    if drop_levels[canon] then
                        record["log_processed"][key] = canon
                        log_level = log_level or canon
                    end
                end
            end
        end

        if not log_level and record["log_processed"] then
            log_level = canonicalize(record["log_processed"]["logLevel"] or record["log_processed"]["level"])
        end

        if not log_level then
            log_level = canonicalize(record["logLevel"] or record["level"])
        end

        if log_level and drop_levels[log_level] then
            return -1
        end

        return 2, timestamp, record
    end
    {{- end }}
and deployment looks like this
Copy code
function normalize_product_logs(tag, timestamp, record)
    local function canonicalize(val)
        if not val then return nil end
        local lvl = tostring(val):lower()
        if lvl == "error" then return "ERROR"
        elseif lvl == "warn" or lvl == "warning" then return "WARN"
        elseif lvl == "info" then return "INFO"
        elseif lvl == "debug" then return "DEBUG"
        elseif lvl == "fatal" then return "FATAL"
        else return lvl:upper()
        end
    end

    -- Canonical drop list
    local drop_levels = {} drop_levels["INFO"] = true drop_levels["ERROR"] = true

    local log_level = nil
    local top_keys = {"logLevel", "level"}

    for _, key in ipairs(top_keys) do
        local val = record[key]
        if val then
            local canon = canonicalize(val)
            if drop_levels[canon] then
                record[key] = canon
                log_level = canon
            end
        end
    end

    if record["log_processed"] then
        for _, key in ipairs(top_keys) do
            local val = record["log_processed"][key]
            if val then
                local canon = canonicalize(val)
                if drop_levels[canon] then
                    record["log_processed"][key] = canon
                    log_level = log_level or canon
                end
            end
        end
    end

    if not log_level and record["log_processed"] then
        log_level = canonicalize(record["log_processed"]["logLevel"] or record["log_processed"]["level"])
    end

    if not log_level then
        log_level = canonicalize(record["logLevel"] or record["level"])
    end

    if log_level and drop_levels[log_level] then
        return -1
    end

    return 2, timestamp, record
end
So when i try to change
Copy code
-- Canonical drop list
    local drop_levels = {} drop_levels["INFO"] = true drop_levels["ERROR"] = true
to
Copy code
-- Canonical drop list
    local drop_levels = {} drop_levels["FATAL"] = true drop_levels["INFO"] = true
Cm gets updated but it doesn't reflect the same and when i remove lua block in filter and add again it works
p
It's really hard to follow what you're meaning sorry so let me try to clarify. You change the LUA script and the config map is updated right? You then change the environment variable value and restart your pods (after the config map is updated) and they have the new environment variable set now? Do you actually mean an environment variable as it seems like you are changing LUA code, not environment variables? What is actually mounted in the pods? Is it correct? You can run the debug image and shell into the pods to check what the actual scripts and variables look like there. That would be my recommendation to narrow down the issue.
s
Yes, let me clarify the exact flow and setup: • I have a Lua script (
custom.lua
)
stored in a
ConfigMap
, which is mounted into the Fluent Bit container. • The Lua script contains logic that checks environment variables (set in the container) like
drop_error_product
,
drop_debug_product
, etc. Example in
custom.lua
• When I want to change behavior (e.g., start dropping ERROR logs), I do not edit the Lua script itself — I only update the environment variable value in the pod spec (e.g.,
drop_error_product=true
). • I confirm that: The
ConfigMap
is correctly mounted The pod has the updated environment variable. However, after restarting the Fluent Bit pods, the new environment variable values do not seem to affect the Lua behavior — ERROR logs are still not dropped. The only thing that works is: • Remove the Lua filter block from the config • Then re-add the same Lua filter block I hope this clear at bit 😅.
p
Have you shelled into the container though to check exactly what is mounted and set?
s
No as my container don't allow to shell, I am using kubesphere image.
p
The debug image has a shell, just add
-debug
to the tag name
Check how old the kubesphere one is, no idea when it was built or how