Hi, a small question: I have only one input (docke...
# fluent-bit
d
Hi, a small question: I have only one input (docker fluentd plugin with remote):
Copy code
pipeline:
  inputs:
    - name: forward
      port: 24224
      tag: nomad
      listen: 127.0.0.1
but I need to check the content, to route to different outputs: for example: JSON field com.hashicorp.nomad.namespace contains "backoffice", then I want, that the output goes to a different endpoint. I thought, that I can use TAG on filters, but I guess, it needs to be done in a different way.
p
rewrite_tag
filter or use a
grep
processor on output
d
Uff, that was fast :-)
p
routing is done by tags so yeah either you make the tags you want match the outputs, or you drop the data you don't want at the output 🙂
Note that you are overriding the tag in that forward input, tags are part of the forward protocol so you can use the original ones if that helps
an example for grep similar to something I'm doing at the moment:
Copy code
pipeline:
          outputs:
            - name: forward
              match: '*'
              processors:
                logs:
                  - name: grep
                    # Only forward records that do have a critical priority set.
                    regex: 'fluentdo_priority ^critical.*$'
Adjust as required but notice I match all records then only allow through ones that have a specific field set to a specific value - look at the grep docs for more details.
d
That logs also very good. My usecase is, to send logs to different VictoriaLogs endpoints, depending on the Containername (or Nomad Namespace).
VictoriaLogs is very nice, but isn't userfriendly, if it comes to user management 🙂 So I need workarounds.
p
I'm not sure about Nomad but K8S log filenames include that information already so when you use a wildcard in tail, e.g.
tag: kube.*
it will be expanded to something like
kube.var.logs.containers.podname_namespace_containername
and so you can use it for matching, e.g.
match_regex: *_namespace1_*
But yeah, it's probably a choice of re-tagging after input or filtering on output.
d
Another sleepless night 🙂 without understand how the rewrite_tag works. I have the following:
Copy code
pipeline:
  inputs:
    - name: forward
      port: 24224
      tag: container
      listen: 127.0.0.1

  filters:
    - name: rewrite_tag
      match: container
      rule: $com.hashicorp.nomad.namespace ^(backoffice-test)$ ns.backoffice false
      emitter_name: re_emitted
With stdout I see:
Copy code
Jul 22 05:53:47 fra-test-nomad-01 fluent-bit[766365]: [0] container: [[1753156426.000000000, {}], {"source"=>"stdout", "com.hashicorp.nomad.namespace"=>"backoffice-test", "com.hashicorp.nomad.node_id"=>"b9e216fb-d242-76e1-641e-ff3efb8f6760", "com.hashicorp.nomad.job_id"=>"backoffice_gunicorn", "log"=>"127.0.0.1 - - [22/Jul/2025:03:53:46 +0000] "GET /adminlogin/ HTTP/2.0" 200 4466 "-" "curl/7.88.1" "-"", "com.hashicorp.nomad.job_name"=>"backoffice_gunicorn", "com.hashicorp.nomad.node_name"=>"fra-test-nomad-01", "com.hashicorp.nomad.task_name"=>"backoffice_gunicorn", "container_id"=>"770f67d86cc22aeedc65fb6801ebdf7fff4fa122947903a8772f6a6fa1522683", "container_name"=>"/backoffice_gunicorn-7d213d9b-a23e-eaf9-124a-5c5d41915de8", "com.hashicorp.nomad.task_group_name"=>"backoffice_gunicorn", "com.hashicorp.nomad.alloc_id"=>"7d213d9b-a23e-eaf9-124a-5c5d41915de8"}]
So from my understanding the docs, it says: I should use "com.hashicorp.nomad.namespace" as key and then check the content, for "backoffice-test". If it matches, than retag "container" to ns.backoffice. Just as background: Nomad uses docker with fluentd plugin (so "forward" is used). So, something still is wrong 🙂 btw: big thanks for helping Pat :-)
AHH found it ! https://github.com/fluent/fluent-bit/issues/7556 just removing the
$
from key and it started to work !