https://linen.dev logo
Join Slack
Powered by
# fluent-bit
  • z
    Okay. I'll give it a try. I used fluentd with a community developed ULS plugin but need to replace it with fluentbit, and lacking the feature. Thanks.
  • e
    <!here> Fluent Bit 
    v4.0.8
    is out!     • release notes: https://fluentbit.io/announcements/v4.0.8/
    important note: 
    v4.0.7
    have a regression in the JSON packer state, users must upgrade
  • s
    Hi everyone, as a user in an enterprise environment I noticed that the official Helm chart for fluent-bit is still marked as beta and does not appear to be hardened. For example: • 
    securityContext
    is empty • the container runs as 
    root
    by default • the embedded HTTP server is enabled by default, and turning it off seems to break liveness/readiness probes I’m curious: what are your experiences with this in enterprise use cases? • Do you use the official chart despite these issues? • Have you created your own hardened Helm charts or rely on alternatives? • How do you handle the beta status and missing security hardening? Would love to hear how others are approaching this.
  • s
    👋 Hello, team!
  • s
    @admin waiting for parquet support on fluentbit https://github.com/fluent/fluent-bit/pull/10691
    • 1
    • 3
  • p
    Hi team, I’m embedding Fluent Bit inside my application using the C API. I tried using flb_lib_config_file() to load a configuration file. The function returns success, and the context is created properly, but the plug-ins from the config file don’t seem to be applied. Platform: Windows Am I missing some additional step after calling flb_lib_config_file(), or is there any limitation on how this API works? Thanks in advance!
  • e
    @PraveenJeffri does the same code works in linux ?
    p
    • 2
    • 2
  • s
    Hey! I'm using fb tail input to read data from a file and using a custom output to upload the data to Azure blob storage (I don't want to use the built-in Azure blob storage output). I'm using these fb configurations:
    Copy code
    [SERVICE]
    Flush                 60
    HTTP_Server           Off
    Daemon                Off
    Log_Level             warn
    Log_File              <log file path>
    storage.path          <storage path>
    storage.type          filesystem
    storage.checksum      On
    storage.max_chunks_up 60

[INPUT]
    Name tail
    Tag <tag>
    Path <path>
    DB <db>
    Key Data
    Path_Key filepath
    Skip_Long_Lines On
    Read_from_Head True
    storage.type filesystem

[OUTPUT]
    Name            myoutput
    Retry_Limit     false
    Match           <tag>
    I have few questions that I couldn't answer from fb docs and I would love to get your help: 1. What controls the scheduler? i.e. when does the engine decides to flush the data to output (in my case "myoutput") 2. What controls how much data is being flushed each time? 3. My goal is to have a flush one the first of these two conditions hold: 1) There's ~250Kib of records ready to be flushed. 2) The 1 minute timeout reached. Is it possible to do with FB configuration?
    e
    • 2
    • 2
  • s
    Heya, according to both the v3.2.7, v4.0.0 zstd compression is now supported in Fluent Bit (for ~6 months now) According to the v4.0.0 docs , I couldn't find any output which supports zstd What am I missing?
    👀 1
    k
    u
    • 3
    • 5
  • s
    Why is 
    FLB_VALGRIND
    flag treated with OR (ref):
    Copy code
    # Valgrind support
check_c_source_compiles("
  #include <valgrind/valgrind.h>
  int main() {
     return 0;
  }" FLB_HAVE_VALGRIND)

if(FLB_VALGRIND OR FLB_HAVE_VALGRIND)
  FLB_DEFINITION(FLB_HAVE_VALGRIND)
endif()
    This means if I have set 
    FLB_VALGRIND
    explicitly Off (or unset) then the presence of valgrind headers is enough to flip 
    FLB_HAVE_VALGRIND
    to On?
  • a
    does anyone know why i am getting this error [error] [inputtailtail.0] cannot execute insert file /var/log/containers/fluent-bit-dev-abcde_<Namespace> ?
  • r
    I'm on version 4.0.3. I want to try yaml based configuration. How can I set the default config to yaml file /etc/fluent-bit.yaml instead of /etc/fluent-bit.conf
    l
    p
    • 3
    • 9
  • u
    Hi Team, I have an interesting behaviour to report for Fluentbit 4.0.8. Based on the Metrics I feel that Fluentbit is not honoring the Memory Buffer Overlimit for the Tail Plugin. I am using conf based configuration. As per my understanding, If I have set the Memory Buffer Overlimit to 512 MB, the Memory Utilization cannot exceed this range + some buffer. However, I am seeing a constant rise in memory of the Pod upto 1.4G (Which is the limit and the Pod crashes). I have never seen this behaviour with Fluentbit 3.0.7 with the same configuration. Attaching some screenshots here. Also attaching my configuration in this thread. Has anyone seen this or any pointers on what to do? I think this is a bug
    l
    • 2
    • 3
  • e
    Worked through fixing broken analytics tracking code for docs, PR ready for review (forgot to squash commits in the PR, please do on merge): https://github.com/fluent/fluent-bit-docs/pull/2053
  • p
    @Pierluigi Lenoci has left the channel
  • p
    Hi all, I’m trying to take values from an incoming JSON payload and insert them into the OTEL log record attributes dynamically. For example, if my incoming JSON has:
    Copy code
    {
  "user": "postgres",
  "env": "production",
  "event": "connection_start"
}
    I want to transform it into a valid OTEL log format like this:
    Copy code
    {
  "resourceLogs": [
    {
      "scopeLogs": [
        {
          "logRecords": [
            {
              "timeUnixNano": "1666799656123456789",
              "body": {
                "stringValue": "connection_start"
              },
              "attributes": [
                {
                  "key": "user",
                  "value": {
                    "stringValue": "postgres"
                  }
                },
                {
                  "key": "env",
                  "value": {
                    "stringValue": "production"
                  }
                },
                {
                  "key": "event",
                  "value": {
                    "stringValue": "connection_start"
                  }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}
    Basically, I need to parse the original JSON and map those key/value pairs into the attributes array of the OTEL log record. Is this possible using Content Modifier? Can we solve this? How can we dynamically add the data into Content Modifier, since it only accepts static data?
  • s
    @Ujjwal Sharma [ZSTD]
    u
    • 2
    • 10
  • a
    Hey all, I wanted to clarify few things regarding the 
    kubernetes_events
    input plugin. I have noticed at some points the plugin does not collect the events as it supposed to do, and when I turn on the debug logs of the input plugin I noticed the the 
    [debug] [input coro] destroy coro_id=xxx
    being logged continuously (Refer to the attached flb-k8s-events-input-debug.log). Has anyone else experienced the same or does anyone have any idea of why this occurs and how to solve the issue? Also I noticed in the debug logs that 
    [debug] [input:kubernetes_events:kubernetes_events.0] could not process payload, incomplete or bad formed JSON:
    lines are getting continuously logged as well, before the burst of destry coro_id logs. I have looked at the source code and realised it is due to the chunks streamed to the input plugin might not always be in complete JSON format. What I wanted to clarify on this is, are we buffering the incomplete chunks so that those events won't be dropped from the events pipeline? If so, would you be able to explain or point me out to the code? Thanks in advance.
    flb-k8s-events-input-debug.log
    p
    • 2
    • 7
  • u
    Hi Fluentbit Team, I have a PR raised which just got approved thanks to @hhatake. However the Build keeps on failing due to 
    run-macos-unit-tests
    with the error 
    make: *** [all] Error 2
    . I dont see any explicit errors in the build logs. Does anyone have any idea about what is going wrong? I have not made any changes related to build pipeline PR - https://github.com/fluent/fluent-bit/pull/10794
    h
    p
    • 3
    • 4
  • e
    Hello Fluent, I hope this is the right forum for this question. I notice the documentation does not mention Windows 11 or Windows Server 22. Is this because these are not supported?
    p
    • 2
    • 4
  • m
    I am writing my own plugin since Fluentbit http out plugin does nto support a secondary destination for failed log events I am following this https://docs.fluentbit.io/manual/fluent-bit-for-developers/golang-output-plugins This requires recompiling fluent-bit with golang support. Questions: Is there an official iimage with golang support? Can I instead use Rust to build my plugin and use it directly with Fluent-bit without requiring recompilation I am using fluent-bit 4.0.8
    p
    h
    • 3
    • 2
  • s
    noisy neighbours problem. • We have multiple micro services running on EKS cluster, each team will have multiple assigned micro services to run and change the code independently. • A Fluent-bit running as Daemonset in EKS cluster which scrapes logs and send them to Loki • Problem - ◦ A few micro services generate logs in high volume and high velocity randomly, no control here. ◦ Sometimes a Developer switch on Debug mode and forgets to off it, which generates 100 GBs of logs in a day. ◦ How can we monitor this situation? How to identify which micro service is making noise by generating huge logs lines in less than a threshold time? ◦ Don't find any metrics to measure, on Fluent-bit-prometheus side, which will measure the file or any label specific activity?
    p
    • 2
    • 3
  • d
    hi folks. My colleague submitted this. Idk if it is the right place. It is really just a question at this point: is grpc expected to work? https://github.com/fluent/fluent-bit/issues/10821
    p
    • 2
    • 2
  • e
    @Ethan Yates has left the channel
  • j
    Hello, I’m having hard time understanding why my basic filter grep doesn’t work 😕 Pipeline: INPUT forward (fluentd) -> fluentbit fluentbit config:
    Copy code
    custom_parsers.conf: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
  fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    debug
        Config_Watch On
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020
        Parsers_File /opt/bitnami/fluent-bit/conf/parsers.conf
        Parsers_File /opt/bitnami/fluent-bit/conf/custom_parsers.conf

    [INPUT]
        Name              forward
        Listen            0.0.0.0
        Port              24224
        Buffer_Chunk_Size 1M
        Buffer_Max_Size   6M


    [FILTER]
        Name   modify
        Match  *
        Condition Key_exists message
        Add    has_message true

    [OUTPUT]
        Name   stdout
        Match  *
        Format json_lines
    fluetnbit logs:
    Copy code
    Fluent Bit v4.0.5
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* <https://fluentbit.io>

______ _                  _    ______ _ _             ___  _____ 
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/ 


[2025/09/05 09:30:38] [ info] Configuration:
[2025/09/05 09:30:38] [ info]  flush time     | 1.000000 seconds
[2025/09/05 09:30:38] [ info]  grace          | 5 seconds
[2025/09/05 09:30:38] [ info]  daemon         | 0
[2025/09/05 09:30:38] [ info] ___________
[2025/09/05 09:30:38] [ info]  inputs:
[2025/09/05 09:30:38] [ info]      forward
[2025/09/05 09:30:38] [ info] ___________
[2025/09/05 09:30:38] [ info]  filters:
[2025/09/05 09:30:38] [ info]      modify.0
[2025/09/05 09:30:38] [ info] ___________
[2025/09/05 09:30:38] [ info]  outputs:
[2025/09/05 09:30:38] [ info]      stdout.0
[2025/09/05 09:30:38] [ info] ___________
[2025/09/05 09:30:38] [ info]  collectors:
[2025/09/05 09:30:38] [ info] [fluent bit] version=4.0.5, commit=d9be919eab, pid=1
[2025/09/05 09:30:38] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2025/09/05 09:30:38] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/09/05 09:30:38] [ info] [simd    ] disabled
[2025/09/05 09:30:38] [ info] [cmetrics] version=1.0.4
[2025/09/05 09:30:38] [ info] [ctraces ] version=0.6.6
[2025/09/05 09:30:38] [ info] [input:forward:forward.0] initializing
[2025/09/05 09:30:38] [ info] [input:forward:forward.0] storage_strategy='memory' (memory only)
[2025/09/05 09:30:38] [debug] [forward:forward.0] created event channels: read=25 write=26
[2025/09/05 09:30:38] [debug] [in_fw] Listen='0.0.0.0' TCP_Port=24224
[2025/09/05 09:30:38] [debug] [downstream] listening on 0.0.0.0:24224
[2025/09/05 09:30:38] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
[2025/09/05 09:30:38] [debug] [filter:modify:modify.0] Initialized modify filter with 1 conditions and 1 rules
[2025/09/05 09:30:38] [debug] [stdout:stdout.0] created event channels: read=28 write=29
[2025/09/05 09:30:38] [ info] [output:stdout:stdout.0] worker #0 started
[2025/09/05 09:30:38] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2025/09/05 09:30:38] [ info] [sp] stream processor started
[2025/09/05 09:30:38] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [task] created task=0x7f002d4296e0 id=0 OK
[2025/09/05 09:30:57] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
{"date":1757064652.918883,"time":"2025-09-05T09:30:52.918883142+00:00","stream":"stdout","log":"HTTP server listening on :8080","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"M2M3OGY1ZDYtY2E2MC00ZjBjLWE3NTEtOThiMDQxYzQ3OWJj"}
{"date":1757064652.919008,"time":"2025-09-05T09:30:52.919007627+00:00","stream":"stdout","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"@timestamp":"2025-09-05T09:30:52.918899636Z","log_level":"INFO","application":"some-namespace","thread":"restartedMain","logger":"org.springframework.boot.devtools.env.DevToolsPropertyDefaultsPostProcessor","message":"Devtools property defaults active! Set 'spring.devtools.add-properties' to 'false' to disable","es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"ZDcxZjJjMDQtYmUxMC00Y2ZhLThkZTMtY2UxNDNmY2JmMmQ3"}
{"date":1757064652.919029,"time":"2025-09-05T09:30:52.919028978+00:00","stream":"stdout","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"@timestamp":"2025-09-05T09:30:52.919006474Z","log_level":"INFO","application":"some-namespace","thread":"restartedMain","logger":"redacted","message":"xxx","es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"MzUwYTJjYTUtNjE2ZS00NWVkLWI5ZDgtNGJmZWJiODljZjIx"}
{"date":1757064652.919029,"time":"2025-09-05T09:30:52.919028978+00:00","stream":"stdout","log":"Finished printing sample Spring Boot log lines.","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"ZjgyYTEwZDMtMzZmNC00MWQyLThkMGQtY2ZmNGIyZTRiNDU2"}
[2025/09/05 09:30:57] [debug] [out flush] cb_destroy coro_id=0
[2025/09/05 09:30:57] [debug] [task] destroy task=0x7f002d4296e0 (task_id=0)
    --- For some reason FILTER says that key “*message*” does not exist but if you check the ouput you can clearly see that key “*message*” is there:
    Copy code
    [2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
...
{"date":1757064652.919029,"time":"2025-09-05T09:30:52.919028978+00:00","stream":"stdout","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"@timestamp":"2025-09-05T09:30:52.919006474Z","log_level":"INFO","application":"some-namespace","thread":"restartedMain","logger":"redacted","message":"xxx","es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"MzUwYTJjYTUtNjE2ZS00NWVkLWI5ZDgtNGJmZWJiODljZjIx"}
    I’m definitely missing something crucial. I have checked docs, github issues and also tried three different AI tools but all of them saying that FILTER should work 😄…. I would greatly appreciate if someone could point me in the direction how I could debug it! Thanks.
    p
    • 2
    • 17
  • u
    Hi Team, can I get a Github Action retry for this PR https://github.com/fluent/fluent-bit/pull/10825? The earlier test failed because of 
    504
    from chocolatey which I feel is transient
    p
    s
    • 3
    • 4
  • u
    Hi Fluentbit Team, can I get a review for this PR if possible? https://github.com/fluent/fluent-bit/pull/10825
  • v
    Need help I'm using server_create helper for beats input plugin for achieving the shared port for multiworkers and using parser.feed(data) method to parse the data but getting the duplicates as of now I enabled only one worker
  • n
    Hello, I am reaching out from the observability team at GitHub. I wanted to ask what is the best way to go about getting feedback for patches to fluent-bit. We have two PRs that have gone stale that we would love to get some eyes on. https://github.com/fluent/fluent-bit/pull/9567 https://github.com/fluent/fluent-bit/pull/9588 Thank you 😁 🙏
    e
    • 2
    • 1
  • y
    Hi Team, I wanted to ask Fluent bit compatibility with Kubernetes Version. I couldn't find any official doc mentioned about it. So I posted a question to github. Could please someone help answer it? https://github.com/fluent/fluent-bit/discussions/10860, thanks🙏
    p
    • 2
    • 11