Hello, We are modernizing our logging infrastructu...
# fluent-bit
a
Hello, We are modernizing our logging infrastructure, and since Fluent Bit is the recommended tool for efficient log collection and aggregation, we aim to use it as both the collector and aggregator in our logging stack. Currently, we’re using Fluentd as an aggregator, but I’ve encountered a problem while translating its configuration to Fluent Bit. Specifically, I’m unable to directly translate the following snippet, which adds a
userEmail
field to a log record by extracting it from the
headers.Cookie
field:
Copy code
<filter {kube.var.log.containers.traefik-**}>
      @type record_modifier
      <record>
        userEmail ${record["headers.Cookie"]&.match(/[^|]+@[^|]+/)}
      </record>
    </filter>
The Fluent Bit
record_modifier
plugin doesn't support the Ruby code that Fluentd uses. If I understand correctly, I should use a similar Lua script to handle this case :
Copy code
function extract_email(tag, timestamp, record)
    local cookie = record["headers.Cookie"]
    if cookie ~= nil then
        local email = string.match(cookie, "[^|]+@[^|]+")
        if email ~= nil then
            record["userEmail"] = email
        end
    end
    return 1, timestamp, record
end
This leads me to two questions: • Am I right in thinking that Lua scripting is the correct solution here, or have I missed an appropriate configuration available in existing plugins? • Is Lua scripting the general fallback for cases not supported by plugins, or should it be avoided (e.g., because it's not the intended way to use the tool or could cause performance issues)?
a
Lua is the right use case here
Yea lua generally is the catch all similar to enable ruby. I will also say lua is quite performant
The major perf implications are actually the decode / encode portion and lua allows you to perform this within a single decode / encode operation
We (Chronosphere) use lua as our main method for transforms in our product and have seen it scale effectively
a
Very clear, thanks !