Hello I’m curious if anyone knows if there’s a way...
# fluent-bit
s
Hello I’m curious if anyone knows if there’s a way to enable the yaml format with the helm chart. When I try to template the latest helm chart, I’m still getting classic files. And if I try to set the file content, I get an error during startup. Any help would be great! [error] [config] error in /fluent-bit/etc/fluent-bit.conf1 undefined value - check config is in valid classic format which is not, as I’m trying to change it to the yaml @Pat any hints from your side?
p
I'm actually writing an article with this at the moment but yeah it is, easiest way is to use extra files
I've got open issues to switch to the yaml format by default on both the main and charts repo but been distracted by life I'm afraid to finish them
The gist is you need to ensure the file ends in .yaml otherwise it'll assume it's classic format and that means you also need to override the default launch parameters to reference it
s
do I also need to add the config.extraFiles to the data: in the configMap or will this happen with the helm chart automatically? I need definitely to switch from conf to yaml.
p
No, just define it in the helm chart values like my example.
Basically you end up with an the default config but it's ignored and just uses the extra file
Copy code
config:
  extraFiles:
    fluent-bit.yaml: |
      service:
        # Required for health checks
        http_server: on
      pipeline:
        inputs:
          - name: tail
            tag: kube.*
            path: /var/log/containers/*.log
            multiline.parser: docker, cri
            Skip_Long_Lines: On
            processors:
              logs:
                - name: kubernetes
                  kube_tag_prefix: kube.var.log.containers.
                  merge_log: on

        outputs:
          - name: stdout
            match: "*"

args:
  - --workdir=/fluent-bit/etc
  - --config=/fluent-bit/etc/conf/fluent-bit.yaml
If you're using your own config map, i.e. not generated by helm, then you can just add the yaml file and all you need is the
args
to specify it to
-c filename
Does that make sense? Essentially you need to override the default config file specified and it must have a
.yaml
suffix
s
I think, I got the point, but unfortunately, this gives much more to change, then I was hoping.
p
Yeah the problem is that default config file and needing it to end in .yaml
I want to change the default or make this a bit better upstream but that's not in place yet
s
I will start fresh, with the latest helm chart and adopt file by file to the new format, as for example in the daemonset the file is as .conf defined and the switch would break the existing setup.
p
I think you can include legacy format into the top level yaml as well
s
we using includes in the fluent-bit.yaml file and I guess, this include also needs to be .yaml ?
p
No, I don't think it needs to be but should be easy to test. I think it handles each file individually, the actual config is in memory after
s
ok
p
Let me know as that's a good thing to add to my article 😆
s
that would be great to add as info in your article
some issues with the helm chart and the yaml config http_server true, does restart the pod as it seems nothing to do as the config, which are included are not present to the system [2025/06/20 115235] [error] [config] YAML error found in file “/fluent-bit/etc/conf/fluent-bit.yaml”, line 12, column 4: unexpected event ‘sequence-start-event’ (7) in state ‘section-value’ (5). [2025/06/20 115235] [error] yaml error [2025/06/20 115235] [error] configuration file contains errors, aborting. apiVersion: v1 kind: ConfigMap metadata: labels: app.kubernetes.io/instance: fluent-bit app.kubernetes.io/name: fluent-bit name: fluent-bit data: fluent-bit.yaml: | service: daemon: Off flush: 5 log_level: debug parsers_file: parsers.conf parsers_file: extra-config/custom_parsers.yaml http_server: false http_listen: 0.0.0.0 http_port: 2020 includes: - extra-config/out-splunk*.yaml - extra-config/out-splunk_oauth-apiserver.yaml - extra-config/out-splunk_openshift-ingress_router.yaml
p
It looks like it doesn't like your first include
My guess is yaml formatting or the wrong list Vs section
I think @Eric D. Schabell has been doing a load of yaml docs updates
You can also do a dry run of your yaml config in a local container
Oh it may need quoting for wildcards in yaml
I'd quote the includes anyway as a general practice
e
just looking at this, I’m thinking you need to quote the lines with wildcards, I often use single quotes ‘something’.
and I am not sure you can use two of the same keys, like parsers_file, might need to do this:
parsers_file:
- parsers.conf
- extra-config/custom_parsers.yaml
I’m also assuming you are not using / importing parser files with both classic and YAML configurations?
p
Yeah any usage of a wildcard needs to be quoted I think, it's a special character for yaml or something
s
have single quoted the includes and removed the wildcard one, same issue
the parsers.conf is from your setup and the custom_parsers.yaml is our own, so I can’t modify the included parsers.conf.
changed parsers_file as mentioned above, which makes sense.
found the issue, the includes is not part of the service, so the indentation was wrong
🎯 1
p
Ah well done mate, those errors are so annoying. Are the docs not right? We should correct them if so and add an example so people can copy paste
🙌 1
s
a sample would be great, in the docs the includes are a page by its own. but to be clear, I made a mistake from the old include to the new one by moving it too much to the right side.
p
Feel free to send a docs PR for it
e
@srutz raise an issue here and I’ll take it on: https://github.com/fluent/fluent-bit-docs/issues/new/choose
s
@Eric D. Schabell by the way, the parsers_file do not accept an arary (-) but it can be included, same as the yaml file
😒 1
p
@srutz can I check if the include files worked then using legacy format in a YAML file?
s
@Pat do you mean, I should test a legacy formated file with the include statement from an yaml service?
p
yeah I was wondering if you included a .conf in your top level yaml
s
created a simple shell script to re-format the include pipelines to the new format, but I’ll test if a classic style file will work as well.