https://linen.dev logo
Join Slack
Powered by
Hi team. I've been having an issue trying to set ...
# fluent-bit
b
Hi team. I've been having an issue trying to set a service.name field. I've tried everything and would welcome any help!
logging: driver: fluentd options: fluentd-address: 127.0.0.1:24224 fluentd-async: "true" fluentd-retry-wait: "1s" fluentd-max-retries: "30" tag: "xxxx-ui"
i was going down this rout e
processors: logs: - name: lua script: /fluent-bit/scripts/inject_tag.lua call: inject_tag - name: opentelemetry_envelope - name: content_modifier context: otel_resource_attributes action: upsert key: service.name value: ${record["fluent_tag"]}
.
function inject_tag(tag, timestamp, record) record["fluent_tag"] = tag return 1, timestamp, record end
p
What's your actual input date and what are you getting at the moment for output?
Just use stdout output to test it and you can iterate quite easily locally - use a dummy input to replace the input from docker for example: https://www.fluent.do/blog/debugging-tips
b
I think i need to set this service.name {"date":1761221225.0,"__internal__":{"group_attributes":{"resource":{"attributes":{"service.name":""}},"scope":{}},"log
so yes theres a stdout but its always blank
p
What's your actual Fluent Bit config?
b
pipeline: inputs: - name: forward listen: 0.0.0.0 port: 24224 processors: logs: - name: lua script: /fluent-bit/scripts/inject_tag.lua call: inject_tag - name: opentelemetry_envelope - name: content_modifier context: otel_resource_attributes action: upsert key: service.name value: ${record["fluent_tag"]} filters: [] outputs: - name: stdout match: '*' format: json - name: opentelemetry match: '*' host: XXXX port: 4318 header: Authorization ${HYPERDX_API_KEY} tls: false logs_uri: /v1/logs
p
I used content_modifier to set it in some examples a while back: https://www.fluent.do/blog/otel-collector-integration#opentelemetry-envelope-processor
I'm not sure if it supports record-accessor syntax like you want to use, also I don't think that's the syntax to get the tag
RA syntax is for getting fields in the record, the tag is the routing for the whole record and not part of it usually.
Are you saying you get no output for stdout? You're matching everything that comes in so if nothing is coming out then it means something else is wrong
b
Yes stdout, the service.name is empty
If I use a static value
image.png
my observability tool is happy
image.png
The problem is how to read this tag which im set in my docker compose
p
yeah so it does not support RA syntax, I think you want LUA like the example here: https://docs.fluentbit.io/manual/data-pipeline/filters/lua#opentelemetry-example
b
Ive tried 20 different ways but no way to get it
would rather avoid lua if i can
p
why?
b
ok i can use if ti works
p
The metadata aware function gives you the tag and access to the metadata
So just inject the tag into the metadata that way, the example shows updating the metadata and copying the service name
b
confused sorry!
What is the standard way when I have docker compose, say 10 services. One fluentd, and I want to distinguish between where the logs are coming from?
p
Something like:
Copy code
function cb_groups_and_metadata(tag, timestamp, group, metadata, record)
    group['resource']['attributes']['service.name'] = tag
    return 1, timestamp, metadata, record
end
Completely untested, you may need to ensure the groups are created first if empty and other things. I would use a dummy input with opentelemetry_envelope processor first then stdout output afterwards so you can see it all.
b
maybe this tag was a bad idea
p
Have a look at the example in the docs, it shows OTEL input as well: https://docs.fluentbit.io/manual/data-pipeline/filters/lua#opentelemetry-example
Ah, there is a bit at the bottom which may be a blocker:
Copy code
Group metadata is read-only and should not be modified. If you don't need group or metadata support, you can continue using the three-argument prototype.
You may have to inject it into the metadata like the example does for OTLP
Or just add another field in your record and use that
b
ok thankyou let me look into all of this
Open in Slack
PreviousNext
Powered by