https://linen.dev logo
Join Slack
Powered by
Hi I am facing with this error ```[2026/09/01 06:1...
# fluent-bit
h
Hi I am facing with this error
Copy code
[2026/09/01 06:18:00.088] [error] % Failed to produce to topic pod-logs: Broker: Message size too large
Obviously the log lines are larger than the Redpanda topic’s 
message.max.bytes
But I don't have the control on the log size and would not want to change the Redpanda settings as it is a central multi-tenant cluster. What I could do here on the FluentBit's configuration?
my current output
Copy code
- name: kafka
          queue_full_retries: 0
          Brokers: redpanda:9094
          rdkafka.ssl.certificate.location: /ssl/rp-user-fluentbit-prod/tls.crt
          Match: kube.var.logs.*
          rdkafka.ssl.key.location: /ssl/rp-user-fluentbit-prod/tls.key
          dynamic_topic: "On"
          rdkafka.compression.codec: snappy
          storage.total_limit_size: 1G
          rdkafka.request.required.acks: 1
          rdkafka.log.connection.close: false
          <http://rdkafka.message.timeout.ms|rdkafka.message.timeout.ms>: 3600000
          rdkafka.batch.size: 900000
          <http://rdkafka.queue.buffering.max.ms|rdkafka.queue.buffering.max.ms>: 10
          rdkafka.queue.buffering.max.kbytes: 1048576
          topic_key: _output_topic_
          alias: out.k8s-pod-logs.redpanda.k8s-pod-logs.k8s-other-pod-logs
          rdkafka.queue.buffering.max.messages: 10000
          Topics: k8s-other-pod-logs
          rdkafka.ssl.ca.location: /ssl/redpanda-root-cacrt/redpanda-root-cacrt.crt
          rdkafka.security.protocol: ssl
          workers: 2
and the service section
Copy code
service:
      flush: 1
      grace: 5
      daemon: "off"
      dns.mode: UDP
      log_level: warn
      http_server: "on"
      http_listen: 0.0.0.0
      http_port: 2020
      coro_stack_size: 24576
      scheduler.cap: 2000
      scheduler.base: 5
      json.convert_nan_to_null: false
      sp.convert_from_str_to_num: true
      Health_Check: "on"
      storage.path: /var/log/logs-storage
      storage.sync: full
      storage.backlog.mem_limit: 1G
      storage.max_chunks_up: 200
      Hot_Reload: "on"
      storage.backlog.flush_on_shutdown: "on"
      storage.metrics: "on"
      storage.type: filesystem
p
You can probably use LUA to split those longer records up into separate ones. There's an example of splitting records in the docs: docs.fluentbit.io/manual/data-pipeline/filters/lua#… Note that you can use this as a processor just on your Kafka output to ensure it is not split for others.
12 Views
Open in Slack
PreviousNext
Powered by