hi there- I'm trying to build an log obs pipeline ...
# fluent-bit
p
hi there- I'm trying to build an log obs pipeline that I thought would be pretty common but not having much success ๐Ÿ˜ž log4j/logback application docker container => fluent-bit => otel-collector I'm thinking of
fluent-bit
as a normalization step, to take structured logging data (json) and transforming it into an OTLP compliant log structure - is this a reasonable expectation? details in the ๐Ÿงต
the generated log in the application looks like this:
Copy code
{
  "@timestamp": "2025-06-12T23:34:14.977849719Z",
  "@version": "1",
  "message": "my log message here",
  "logger": "io.ys.pws4s.persist.influxdb.InfluxDBClient$",
  "thread": "io-compute-2",
  "severity": "DEBUG",
  "level_value": 10000,
  "service.name": "pws4s",
  "service.version": "0.1.0",
  "service.namespace": "homelab"
}
one approach was something like this:
Copy code
pipeline:
  inputs:
    - name: forward
      listen: 0.0.0.0
      port: 24224

  filters:
    - name: parser
      match: '*'
      parser: log4j_json
      key_name: log
      reserve_data: 'true' # keep the docker log driver fields like 'container_id', 'container_name'
    - name: modify
      match: '*'
      Remove: source # .. but remove 'source', it's always 'stdout'

  outputs:
    - name: opentelemetry
      match: '*'
      host: otel-collector
      port: 443
      tls: 'On'
      log_response_payload: true
      logs_body_key: message
      logs_severity_text_message_key: severity
      logs_severity_number_metadata_key: level_value
      logs_body_key_attributes: true
      logs_timestamp_metadata_key: '@timestamp'
      logs_observed_timestamp_metadata_key: '@timestamp'
at the otel-collector side I see something like this:
Copy code
otel-collector  | Resource SchemaURL: 
otel-collector  | ScopeLogs #0
otel-collector  | ScopeLogs SchemaURL: 
otel-collector  | InstrumentationScope  
otel-collector  | LogRecord #0
otel-collector  | ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
otel-collector  | Timestamp: 2025-06-12 23:47:15.064183195 +0000 UTC
otel-collector  | SeverityText: DEBUG
otel-collector  | SeverityNumber: Unspecified(0)
otel-collector  | Body: Str(my log message here)
otel-collector  | Attributes:
otel-collector  |      -> @timestamp: Str(2025-06-12T23:47:15.064183195Z)
otel-collector  |      -> @version: Str(1)
otel-collector  |      -> logger: Str(io.ys.pws4s.ingest.ecowitt.EcoWittRoutes)
otel-collector  |      -> thread: Str(io-compute-3)
otel-collector  |      -> severity: Str(DEBUG)
otel-collector  |      -> level_value: Int(10000)
otel-collector  |      -> service.name: Str(pws4s)
otel-collector  |      -> service.version: Str(0.1.0)
otel-collector  |      -> service.namespace: Str(homelab)
otel-collector  |      -> container_id: Str(c8e570802fbaa1b5fac7cd516eec916dbda55d2aafc78e37fe081958aab04f4f)
otel-collector  |      -> container_name: Str(/pws4s-pws4s-1)
otel-collector  | Trace ID: 
otel-collector  | Span ID: 
otel-collector  | Flags: 0
a few small problems like
ObservedTimestamp
and
SeverityNumber
being wrong, the main problem is trying to elevate the
service.*
attributes to Resource Attributes - I tried doing that on the Otel Collector side too without success .. ideally I would want to just output this correctly from fluent-bit I also tried the
opentelemetry-envelope
processor (unclear if this is needed when using the
opentelemetry
output - seems like it is from the docs) along with
content_modifier
to upsert the
service.name
for example but it only supports static values, not ideal, it's already provided in the log metadata
c
Henrik has couple of examples 1. https://github.com/isItObservable/fluentbit-vs-collector/blob/master/fluentbit/pipeline/step3-logs-otlp-prometheus/fluentbit.yaml 2. https://github.com/isItObservable/fluentbit-vs-collector/blob/master/fluentbit/pipeline/step2-logs-otlp/fluentbit.yaml maybe you can give it a try, also not sure why you need otel collector in this architecture, to test how it is going to be seen on otlp endpoint?
and also I don't see your fluent bit version so if it's outdated please use latest as always ๐Ÿ™‚
ahh re: small problems like
ObservedTimestamp
and
SeverityNumber
being wrong, You're using metadata key instead of message_key so if you replace
Copy code
logs_timestamp_metadata_key: '@timestamp'
logs_observed_timestamp_metadata_key: '@timestamp'
with
Copy code
logs_observed_timestamp_message_key: '@timestamp'
logs_severity_number_message_key: level_value
it should work The issue lies in the distinction between
*_metadata_key
and
*_message_key
configuration options in fluent-bit's OpenTelemetry output plugin: โ€ข *`*_metadata_key`*: Looks for the key in the record's metadata (not the log message body) โ€ข *`*_message_key`*: Looks for the key in the log message body/record content
re: opentelemetry-envelope it converts your otlp non-compliant schema to otlp schema (before output converts it) so you can do manipulation on data like elevate service.* atributes as they are metadata level attributes E.g.
Copy code
pipeline:
  inputs:
      # Your inputs

      processors:
        logs:
          - name: opentelemetry_envelope

          - name: content_modifier
            context: otel_resource_attributes
            action: upsert
            key: service.name
            # Please replace YOUR_SERVICE_NAME with the name of your service
            value: YOUR_SERVICE_NAME
and you're right I think content_modifier doesn't support record accessor @Pat I remember you raised that before right?
b
Feels weird to suggest in the Fluent Bit Slack, but would all of this be far simpler going straight to an OTel Collector? The OTTL needed to do the transformations you want should be quite simple. Fluent Bit feels like an unnecessary middleman here.
c
Yeah I don't think weird IMHO whatever best and lean ๐Ÿ˜„
p
Thank you so much @Celalettin! I totally missed the distinction between message_key and metadata_key @Braydon Kains fair point and I did get that working last night, using the otel log appender to publish directly to the otel collector. In my mind it is a little more resilient to have fluent bit in the middle in case the collector is down ... I'd rather externalize the back pressure away from my services and onto a local fluent bit instance (in this case running directly on the host and not in a container) ... maybe this is wishful thinking, I'll ponder this some more ... reducing complexity is always welcome
b
I can understand the thought process if the collector(s) are some external gateway type deployment and you want a local aggregator of some kind. I see it as a bit of a who-watches-the-watchmen thing; if you really are having a problem with an external destination being down the hammer comes down somewhere even if there's a local aggregator. Personally I would think the surface area for failures would be a bit too wide; given that the collectors are likely sending to another destination which can also be down, that means the pipeline has 3 parts (aggregator, gateway, end destination) that all need to be functioning for the whole pipeline to flow, and any one part could cause data loss. This sort of deployment design isn't really my area of expertise anyway, you likely know more than I do. I was only making my suggestion with the consideration that knowing the destination of your data is a Collector anyway, you might have a simpler mental model for the data transformation if you just use a Collector. Log4j -> OTLP as opposed to Log4j -> Fluent -> Transform Fluent particular ways for the OTLP result -> OTLP.
p
@eduardo thanks again for the scripting demo today! I'll open something more coherent on GitHub Issues to hopefully illustrate my requirements or figure out where I went wrong