Good morning, I have issues with GELF output, and for some messages I get that short_message is missing, or

Unrecognized character escape 'C' (code 67)

and I have no idea .. which messages, because Graylog dropping them. So I need a way to get the exact output, which the Graylog gets. I tried TCPdump, but because the host gets so many messages, that I can't see, which one is the problematic one.

Sadly, but I think they will direct you to raise an issue in GitHub just like they advised me in my topic above (link here). So quick clone of the suggestion of @Eric D. Schabell : Issues can be raised here: https://github.com/fluent/fluent-bit/issues (green New Issue button at the top). Good luck!

Hi found a way .. netcat:

#!/bin/bash
while [ true ];
do
    nc -lp 12201 -w 0
    echo ------------------- 
done

and then configure a 2nd GELF output with 127.0.0.1 address and start the script and restart fluent-bit

you can get the contents of the records from Fluent Bit with a stdout output

Sure ..

service:
  # -----------------------------------------------------------
  # Grundkonfiguration
  # -----------------------------------------------------------
  daemon:        Off
  log_level:     error
  http_server:   On
  http_listen:   0.0.0.0
  http_port:     2020

  # -----------------------------------------------------------
  # Konfigurationsquellen
  # -----------------------------------------------------------
  parsers_file:  parsers.conf
  plugins_file:  plugins.conf
  hot_reload:    On

  # -----------------------------------------------------------
  # Speicher und Stabilität
  # -----------------------------------------------------------
  storage.path:               /var/log/flb-storage/
  storage.sync:               normal
  storage.checksum:           off
  storage.max_chunks_up:      512
  storage.backlog.mem_limit:  256M

  # -----------------------------------------------------------
  # Performance-Tuning
  # -----------------------------------------------------------
  flush:          1.0
  grace:          5
  coro_stack_size: 24576
  workers:        4
...
#######################################################################
# INPUTS
#######################################################################
pipeline:
  inputs:
    - name: syslog
      listen: 127.0.0.1
      port: 5140
      parser: syslog_rfc3339
      tag: syslog
      mode: udp
      buffer_chunk_size: 32000
      buffer_max_size: 64000
      receive_buffer_size: 512000

    - name: forward
      port: 24224
      tag: container
      listen: 127.0.0.1

...

    - name: gelf
      match: ns.*
      host: <http://fc-r01-graylog.inatec.com|fc-r01-graylog.inatec.com>
      port: 12201
      mode: tcp
      tls: false
      Gelf_Short_Message_Key: log
      Gelf_Timestamp_Key: timestamp
      Gelf_Host_Key: host
      Gelf_Level_Key: level

I've added for debug:

- name: gelf
      match: ns.*
      host: 127.0.0.1
      port: 12201
      mode: tcp
      tls: false
      Gelf_Short_Message_Key: log
      Gelf_Timestamp_Key: timestamp
      Gelf_Host_Key: host
      Gelf_Level_Key: level

and had my netcat listen on localhost:12201 and try to find the messages which have broken JSON or missing fields.

broken how, what does the input look like?

Input is from Nomad (Docker -> forward via 24224) , not Syslog.

Right, and again what's the actual input?

hmm, you mean splitted via TCP in two half JSON parts?

Or split at the source

but then I would also see issues on Victoria-Logs ... or VL is discarding messages and I don't see any logs

My gist is to simplify things: you have a pipeline with various components and stacks involved, trying to debug where or what is going wrong by looking at the end is very hard plus almost impossible for anyone remote to help with. You can use stdout to see what the output is at in each stage of the pipeline to confirm your assumptions and ideally identify where it is going wrong. You can easily add stdout without impacting the current setup, just as another filter or output to see.

ok, will do it 🙂

I mean if you know what to look for you can just use stdout with a grep processor:

outputs:
  - name: stdout
    match: '*'
    processors:
      logs:
        - name: grep
          regex: XXX

https://docs.fluentbit.io/manual/data-pipeline/processors/filters

uhh, that is cool

you can either include or exclude via regex so pick something where it flags lines that are broken

right now, I'm using file and have

tail -f debug.log | jq

.. so if I have a broken json, jq will tell me