I am trying to use AWS MSK (KAFKA) , I was wonderi...
# fluent-bit
m
I am trying to use AWS MSK (KAFKA) , I was wondering if the storage.type parameter is supported for kafka plugin? Which plugins support storage.type or does all plugin buffer in memory?
p
I believe they all support it, have you tried it?
m
I tried it the with filesystem type but it says doesn’t support storage.type for Kafka plugin. Let me try again and copy and paste the error
p
Yes, provide the full config and logs. Also check you're on the latest version
Use the OSS version rather than an AWS image as well because the AWS image is on a much older version (1.9 instead of 4.0)
m
fluent-bit-simple.yaml
I am using helm chart version 0.49.1 which is application version 4.0.3 the latest. Here is the error and my config
Copy code
[2025/06/24 20:55:01] [error] [config] kafka: unknown configuration property 'storage.type'. The following properties are allowed: topic_key, dynamic_topic, format, message_key, message_key_field, timestamp_key, timestamp_format, queue_full_retries, gelf_timestamp_key, gelf_host_key, gelf_short_message_key, gelf_full_message_key, gelf_level_key, topics, brokers, client_id, group_id, rdkafka., and raw_log_key.
[2025/06/24 20:55:01] [ help] try the command: /fluent-bit/bin/fluent-bit -o kafka -h
@Pat I've also tried, different versions as well. like 4.0.0 and tried with the s3 plugin as well
@Pat any thoughts on this?
p
Not had a chance to look, do the docs say storage type is per plugin?
m
It does not. I thought it would be the same way the networking configuration isn’t on the output plugins docs but we are able to use them there. Looks like storage.type isn’t supported in the Kafka or s3 ouput plugin
p
The s3 plugin has it's own approach I believe to filesystem buffering
m
Why doesn’t Kafka support it?