Hi, i'm wondering if anyone managed to use fluent-...
# fluent-bit
j
Hi, i'm wondering if anyone managed to use fluent-bit to send k8s logs to an otel collector with some kind of transformation in between? I have an app in k8s outputting structured json logs like
{"time":"2025-06-27T183417.891659338+00:00","level":"INFO","msg":"everything ready","request_id":"0197b2ab-4397-7958-ac3d-d307b6001ace","operation":"is-ready","method":"GET","path":"/v0/ready","trace_id":"1bf8bd7b774805451aa97975094510f6"}
I'd like to lift the
msg
field to the otel
Body
field and
level
to the
SeverityText
and have the rest fields as
LogAttributes
, the metadata from
kubernetes
filter would be
ResourceAttributes
. I tried to use the
opentelemetry
and
content_modifier
processor with the new yaml config, but it looks like the
upsert
action can only take static values? For example my config below wouldn't take the value of field
path
in the
LogAttributes
or setting
service.name
correctly in the resource attributes
Copy code
processors:
  logs:
    - name: kubernetes
      kube_tag_prefix: kube.
      regex_parser: kube-tag
      merge_log: On
      keep_log: Off
      buffer_size: 68k
    - name: opentelemetry_envlope
    - name: content_modifier
      context: attributes
      action: upsert
      key: path
      value: $path
    - name: content_modifier
      context: otel_resource_attributes
      action: upsert
      key: service.name
      value: ${kubernetes.labels['app.kubernetes.io/name']}
Curious to know how you are configuring fluent-bit for otel?
p
I'll try to take a look at this next week, I wrote up a post on using otel so be good to add
j
hey Pat! thanks for the info. I've already used the
opentelemetry_envlope
processor as you can see in my config, but I struggle to combine it with the
kubernetes
filter where i'm expecting some k8s metadata to be in
LogAttributes
and others
ResourceAttributes
p
I'll have to look at the code to be honest but I presume the docs don't cover it? Does $path work then or not? If it does then so should the other... Ah, I see - you're using curly braces which is variable syntax not record accessor one
${VAR}
will expect a variable to be set but you're trying to reference another field which should just be
$field
I think what you want is this: https://docs.fluentbit.io/manual/1.5/administration/configuring-fluent-bit/record-accessor
Copy code
$kubernetes['labels']['app.kubernetes.io/name']
However, I'd probably just set up a new key with another filter first to simplify that and then reference a single key - if you do it with a LUA script then you can also cope with defaulting or missing values
j
i do have
$path
there as well just to test both syntax. so are you saying the
value
field of a filter/processor supports record accessor syntax too? i thought it was only the
key
p
It depends is the answer to record accessor syntax - it depends on whether the specific plugin supports it for the specific config option you are referencing
There is not a global "it is supported everywhere for X"
But I would expect value to be more useful - keys tend to be fixed whereas looking up values makes more sense to me
@hhatake do you happen to know if the value parameter can take just a key or use RA syntax?
h
I didn't implement RA syntax support on processor_content_modifier. So, we didn't support RA syntax on content modifier processor yet.
j
thank you both for the info! without RA i think content_modifier wouldn't be of much help to us right now because of the transformations we wanna implement. I guess I can only forward everything to the otel receiver and do it on the other end.
sorry to ping the old thread again, but is there any plan to support RA in
processor_content_modifier
? I think it only makes sense when RA is supported there coz we don't want to hard code values in the config
p
Is there a feature request for it? I'd raise one if not so we can track it properly. I'd say it would be good to do but requires someone to do it 🙂
j
sure! do I just create an issue with this label?
p
There is an issue template for feature requests so follow that
But do a check for an existing one first, duplicates are annoying
j
https://github.com/fluent/fluent-bit/issues/4651 oh i see there's an issue about
modify
filter