Peter VR
06/12/2025, 11:38 PMfluent-bit as a normalization step, to take structured logging data (json) and transforming it into an OTLP compliant log structure - is this a reasonable expectation? details in the ๐งตPeter VR
06/12/2025, 11:40 PM{
"@timestamp": "2025-06-12T23:34:14.977849719Z",
"@version": "1",
"message": "my log message here",
"logger": "io.ys.pws4s.persist.influxdb.InfluxDBClient$",
"thread": "io-compute-2",
"severity": "DEBUG",
"level_value": 10000,
"service.name": "pws4s",
"service.version": "0.1.0",
"service.namespace": "homelab"
}Peter VR
06/12/2025, 11:48 PMpipeline:
inputs:
- name: forward
listen: 0.0.0.0
port: 24224
filters:
- name: parser
match: '*'
parser: log4j_json
key_name: log
reserve_data: 'true' # keep the docker log driver fields like 'container_id', 'container_name'
- name: modify
match: '*'
Remove: source # .. but remove 'source', it's always 'stdout'
outputs:
- name: opentelemetry
match: '*'
host: otel-collector
port: 443
tls: 'On'
log_response_payload: true
logs_body_key: message
logs_severity_text_message_key: severity
logs_severity_number_metadata_key: level_value
logs_body_key_attributes: true
logs_timestamp_metadata_key: '@timestamp'
logs_observed_timestamp_metadata_key: '@timestamp'Peter VR
06/12/2025, 11:52 PMotel-collector | Resource SchemaURL:
otel-collector | ScopeLogs #0
otel-collector | ScopeLogs SchemaURL:
otel-collector | InstrumentationScope
otel-collector | LogRecord #0
otel-collector | ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
otel-collector | Timestamp: 2025-06-12 23:47:15.064183195 +0000 UTC
otel-collector | SeverityText: DEBUG
otel-collector | SeverityNumber: Unspecified(0)
otel-collector | Body: Str(my log message here)
otel-collector | Attributes:
otel-collector | -> @timestamp: Str(2025-06-12T23:47:15.064183195Z)
otel-collector | -> @version: Str(1)
otel-collector | -> logger: Str(io.ys.pws4s.ingest.ecowitt.EcoWittRoutes)
otel-collector | -> thread: Str(io-compute-3)
otel-collector | -> severity: Str(DEBUG)
otel-collector | -> level_value: Int(10000)
otel-collector | -> service.name: Str(pws4s)
otel-collector | -> service.version: Str(0.1.0)
otel-collector | -> service.namespace: Str(homelab)
otel-collector | -> container_id: Str(c8e570802fbaa1b5fac7cd516eec916dbda55d2aafc78e37fe081958aab04f4f)
otel-collector | -> container_name: Str(/pws4s-pws4s-1)
otel-collector | Trace ID:
otel-collector | Span ID:
otel-collector | Flags: 0Peter VR
06/12/2025, 11:57 PMObservedTimestamp and SeverityNumber being wrong, the main problem is trying to elevate the service.* attributes to Resource Attributes - I tried doing that on the Otel Collector side too without success .. ideally I would want to just output this correctly from fluent-bit
I also tried the opentelemetry-envelope processor (unclear if this is needed when using the opentelemetry output - seems like it is from the docs) along with content_modifier to upsert the service.name for example but it only supports static values, not ideal, it's already provided in the log metadataCelalettin
06/13/2025, 8:34 AMCelalettin
06/13/2025, 8:38 AMCelalettin
06/13/2025, 8:49 AMObservedTimestamp and SeverityNumber being wrong,
You're using metadata key instead of message_key
so if you replace
logs_timestamp_metadata_key: '@timestamp'
logs_observed_timestamp_metadata_key: '@timestamp'
with
logs_observed_timestamp_message_key: '@timestamp'
logs_severity_number_message_key: level_value
it should work
The issue lies in the distinction between *_metadata_key and *_message_key configuration options in fluent-bit's OpenTelemetry output plugin:
โข *`*_metadata_key`*: Looks for the key in the record's metadata (not the log message body)
โข *`*_message_key`*: Looks for the key in the log message body/record contentCelalettin
06/13/2025, 8:58 AMpipeline:
inputs:
# Your inputs
processors:
logs:
- name: opentelemetry_envelope
- name: content_modifier
context: otel_resource_attributes
action: upsert
key: service.name
# Please replace YOUR_SERVICE_NAME with the name of your service
value: YOUR_SERVICE_NAMECelalettin
06/13/2025, 9:06 AMBraydon Kains
06/13/2025, 2:05 PMCelalettin
06/13/2025, 2:39 PMPeter VR
06/13/2025, 2:57 PMBraydon Kains
06/13/2025, 3:40 PMPeter VR
06/13/2025, 7:55 PM