I'm trying to work out how best to rename a nested...
# fluent-bit
a
I'm trying to work out how best to rename a nested element in a log line. e.g. given this:
Copy code
{"msg"=>"doing something", "cluster"=>"my-awesome-cluster", "log"=>"{"colour":"blue"}"}
What is the best way to modify log->colour if I want to rename that to "foo"? I have a solution using lua like this:
Copy code
function rename_field(tag, timestamp, record)
        if record["log"] then
          record["log"] = string.gsub(record["log"], '"colour"', '"foo"')
        end
        return 1, timestamp, record
      end
Tried using cjson but looks like that it's not available in the default Docker image
p
I think modify or record modifier can also potentially do it
a
I've tried that and it won't unfortunately. Those will only work on top level labels
p
Lua might be easier to maintain and let's you cope with various options
I thought it might let you do it with record accessor syntax for nested
a
Record accessor syntax does work for
Condition
but not actually changing values 🙂 (been digging into this a lot today 😅 )
p
Yeah to be honest, LUA is probably the easiest of the lot 🙂
I guarantee you'll start with this then keep adding more custom rules
with YAML format you can also keep all the LUA inline in the config rather than as separate files
a
BTW do you know a way to convert a log string (?) e.g.
Copy code
"log"=>"{"colour":"blue", "foo":"bar"}
to a map
Copy code
"log"=>"{"colour"=>"blue", "foo"=>"bar"}
One strange thing I don't quite understand, is that the fields in
log
do eventually become fields in the log entry in Victoria Logs. It's not clear to me as to how/when that happens in the pipeline 😅
p
so split an existing key into a nested object basically - you can do it with LUA again or a parser
I suspect VictoriaLogs just parse it on ingestion
you can use a
parser
filter though to parse individual keys: https://docs.fluentbit.io/manual/pipeline/filters/parser Typically you do this anyway ingesting logs from kubelet format first and then into app-specific formats
a
That worked thanks ❤️