I've completed writing my plugin and now would like to make sure I wait until all internal queues are drained so that I can call

flb_stop

and

flb_destroy

. I think

flb_loop

is that function but I don't find any documentation. Looking at the implementation, I see:

while (ctx->status == FLB_LIB_OK) {
        sleep(1);
    }

The loop logic is there, but from a distance I do not understand whether the semantics of

FLB_LIB_OK

are "I have nothing to do".

@eduardo would you accept a patch that enhances

in_lib

and

flb_lib_push

to accept MsgPack in addition to JSON? It's already possible to consume MsgPack with

out_lib

, but not yet

in_lib

. This would make the library API symmetric.

Hi all, I am a bit beginner in using fluentbit. but I was wondering if there is a way to use this plugin https://github.com/majst01/fluent-bit-go-redis-output when running fluentbit on kubernetes?

We are using fluent bit 2.0.6 version. for me exec plugin is not working. Is it something with the version only or in my configuration?

please define "not working"

Hey everyone, I am working on creating a fluent-bit input plugin using https://github.com/calyptia/plugin. It looks like that cmetrics is an indirect dependency and I need to install the cmetrics lib to be able to build my input plugin. After installing cmetrics 0.6.0 (and headers) on amd64 machine I am successfully able to compile the go binary. However I also need to cross compile to arm64. Currently I use the following command for this:

CGO_ENABLED=1 CC=aarch64-linux-gnu-gcc GOOS=linux GOARCH=arm64 go build -ldflags "-X 'main.revision=' -X 'main.builddate='" -buildmode=c-shared -o output.so .

However using this my cross compilation fails with

#17 39.58 # <http://github.com/calyptia/cmetrics-go|github.com/calyptia/cmetrics-go>

#17 39.58 /usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/bin/ld: skipping incompatible /usr/local/lib/libcmetrics.a when searching for -lcmetrics

#17 39.58 /usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/bin/ld: skipping incompatible /usr/local/lib/libcmetrics.a when searching for -lcmetrics

#17 39.58 /usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/bin/ld: cannot find -lcmetrics

#17 39.58 /usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/bin/ld: skipping incompatible /lib/../lib/libcfl.a when searching for -lcfl

#17 39.58 /usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/bin/ld: skipping incompatible /lib/libcfl.a when searching for -lcfl

#17 39.58 /usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/bin/ld: cannot find -lcfl

#17 39.58 /usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/bin/ld: skipping incompatible /usr/local/lib/libmpack.a when searching for -lmpack

#17 39.58 /usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/bin/ld: skipping incompatible /usr/local/lib/libmpack.a when searching for -lmpack

#17 39.58 /usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/bin/ld: cannot find -lmpack

#17 39.58 /usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/bin/ld: skipping incompatible /usr/local/lib/libxxhash.a when searching for -lxxhash

#17 39.58 /usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/bin/ld: skipping incompatible /usr/local/lib/libxxhash.a when searching for -lxxhash

#17 39.58 /usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/bin/ld: cannot find -lxxhash

which looks to be a case of not able to find the arm64 cmetric libraries. Does anyone know how I can avoid this while still cross compiling?

I think I found a logic bug that creates a segfault. My setup is: 1. Create context:

flb_create

2.

lib

input 3. invalid output (user error, e.g.,

xxx

) 4.

flb_destroy

👇

239           if (ctx->config->is_running == FLB_TRUE) {
-> 240               flb_engine_shutdown(ctx->config);
   241           }

And then in `flb_output.c`:

479       params = FLB_TLS_GET(out_flush_params);
   480       if (params) {
-> 481           flb_free(params);   <---------- segfault 💥
   482       }
   483   }

This appears to reference global TLS state, so I dug deeper and think I found a bug: The

do_start

function is the place where this state gets initialized in

flb_output_prepare

, via:

pthread_once(&flb_lib_once, flb_init_env);

However, the output got never correctly initialized and the the user hasn't invoked

flb_start

, this state is never initialized. So I'm freeing a point that is never allocated, which is exactly the output:

tenzir(63774,0x16c21b000) malloc: *** error for object 0x16c21b000: pointer being freed was not allocated

Hi Team; Any plan to officially support RabbitMQ as an output? There is project igd-geo/fluent-bit-rabbitmq-output that is not actively maintained. I have forked that and updated the dependencies and Go version, but still would like to see fluentbit offecially support amqp output. Thanks

I need to implement a fluent-bit output plugin. I want to feed to the output to cohere. I'm looking for a step-by-step tutorial on implementing an output plugin. I have looked at the fluent-bit-go examples, but I'm looking for something more in-depth. Is there any such example?

There's a full example here: https://github.com/calyptia/plugin

Having poked into the innards of in_node_exporter_metrics to (successfully!) disable the nvme check noise, I figured I'd like to work on the gap between our current basic collectd on each node set of metrics and what we can get with node_exporter. One of which is the count of logged in users; Looking at https://github.com/collectd/collectd/blob/main/src/users.c it's kist munging /var/run/utmp through standard calls. Is it better to try and roll this into in_node_exporter_metrics or create a new input metric?

Quick Question: In what directory do individuals normally place their custom plugins?

I'm attempting to install a custom output plug_in and I'm installing fluent-bit using helm. During the installation I receive an error that my plugin_file couldn't be found. How do I get my plugin-file installed when installing fluent-bit with helm? Where do I specify the location of my plugin conf file so that the installation can pick it up?

@Aimee Ukasick has left the channel

Hey everyone, I'm a developer at Scaleway, a European cloud provider. Many of our users use Fluent Bit, and we would like to offer a plugin so they can easily connect our products to our observability tools. Would it be simpler to create it using the Go plugin interface, or should we build it directly in C so that our plugin can be integrated directly? Thanks!

hi @jonathan R. first of all welcome 🙂 if you create the plugin in go, you will need to take care of packaging, which long term strategy needs time. If is written in C and available in upstream, the whole world of Fluent Bit users will get access to it

I usually suggest doing it in C but sometimes Go is fine depending of the team you have

Hello 👋 a bit of precision, our product at the moment are using loki, and we would like to have something mostly based on the loki plugin but with native support for Scaleway configuration (auth, endpoints, special headers, ...). Do you think that this contribution will be merged? What would be the next steps? @eduardo

Hi everyone! I see plugins in C are generally published in this directory: https://github.com/fluent/fluent-bit/tree/master/plugins. Is there a similar location for plugins written in Go?

We don’t really have a repo of go plugins, though that would be an interesting idea. @eduardo @Pat what are your thoughts on making a fluent-bit-go-example repo

Out of curiousity @Raahi Chada which plugin are you looking for?

we have the calyptia-plugin one and the dummy one we use for Calyptia Core: • https://github.com/chronosphereio/calyptia-core-fluent-bit-dummy • https://github.com/chronosphereio/calyptia-plugin It would be worth a fully worked example kept up-to-date I agree so maybe extend calyptia-plugin for this? We can auto-insert it into docs as well if needs be

making a simple base repo for all plugins is easy enough too, the main thing is maintaining it really - a lot of the code gets out of date quickly but potentially if they were all public or moved to the fluent org we can auto-build/test with each Fluent Bit release?

@Rémy Léone Hello, I am currently working on an out_scaleway plugin. Since Scaleway uses Loki, a large part of the code will be duplicated from the Loki plugin. However, wouldn't it be simpler if I could reuse the functions I need? For this to be possible, they would need to not be declared as

static

and to declare them in the header. Is this something you could accommodate?

How can we access default Fluent-bit output fields? I can use the function output.FLBPluginConfigKey to get the value of new fields that I've created in the output plugin, but it doesn't seem to work with fields like Match. I'm assuming because this is a default field so I was wondering if there was another function I could use? I get a null/empty value for Match even when it is provided and the instance is properly created.

Is there a developer's guide for (unit) testing for Go plugins?

@eduardo @fujimotos @Masoud Koleini @Pat, Hello Guys, I have a PR for enhancing features and fixing few issues in Azure Kusto output plugin https://github.com/fluent/fluent-bit/pull/8430. Requesting for your help in reviewing the same.

Hi, how do we log errors in go plugins? I know there are various functions in C - but can these be used for Go plugins? Does the log_level and log_file field still work for go plugins? Any advice or insight would be greatly appreciated.

While building plug-ins, do you all use devcontainers mostly? I am unable to build fluent-bit on my Mac successfully without resorting to devcontainers despite installing all the necessary dependencies. Even inside devcontainers running

cmake -DFLB_ALL=On ..

throws errors although it completes successfully (logs in thread). Running make has similar error dump with build stopping at 12% even though the binary is generated at the end. Is this all expected?

archived the channel