Hey, I'm trying to setup a fluentbit http output to forward logs from a pfsense to a siem (Sekoia.io) but i keep having buffer errors like that :

[ warn] [http_client] cannot increase buffer: current=4096 requested=36864 max=4096
[ warn] [engine] failed to flush chunk '1-1775221390.165888982.flb', retry in 10 seconds: task_id=1, input=tcp.0 > output=http.1 (out_id=1)
[error] [output:http:http.1] could not flush records to <http://intake.sekoia.io:443|intake.sekoia.io:443> (http_do=-1)

Most of the incomming traffic is forwarded but i'm afraid i'm loosing some logs when the error occurs. There are not a lot of information about buffers in the official documentation. I tried putting "http_buffer_size: 65536" in the fluent-bit.yml but it doesn't seem to work. I also tried buffer_size and chunk_size in the input but it doesn't to help either. My fluent-bit.yml :

service:
  flush: 1
  log_level: info
  storage.path: /var/log/flb_storage
  storage.max_chunks_up: 128
  http_buffer_size: 65536

includes:
  - /fluent-bit/etc/fb_conf.yaml

My fb_conf.yaml:

pipeline:
  inputs:
    - name: tcp
      tag: pfsense
      port: 5036
      listen: 0.0.0.0
      format: none
      storage.type: filesystem
      #      buffer_size: 2M
      #      chunk_size: 4k
      mem_buf_limit: 8M

  outputs:
    - name: stdout
      match: pfsense_*
      format: json_lines

    - name: http
      match: pfsense_all
      host: <http://intake.sekoia.io|intake.sekoia.io>
      URI: /jsons?intake_key=<HIDDEN> 
      port: 443
      tls: on
      tls.verify: on
      header: content-type application/json
      format: json
      json_date_format: iso8601
      json_date_key: time
      http.response_timeout: 30

Buffer size will need to go in the output plugin config

Already tried but the buffer size option doesn't seem to work with http output plugin (If you look at the documentation there are no option for buffer size)

Run it with

-o http --help

to check it's not just undocumented

When i try applying the configuration "buffer_size: 64K" i get this error message [error] [config] http: unknown configuration property 'buffer_size'. The following properties are allowed: proxy, allow_duplicated_headers, log_response_payload, http.response_timeout, http.read_idle_timeout, http_user, http_passwd, aws_auth, aws_service, aws_region, aws_sts_endpoint, aws_role_arn, aws_external_id, aws_profile, header_tag, format, json_date_format, json_date_key, compress, header, uri, gelf_timestamp_key, gelf_host_key, gelf_short_message_key, gelf_full_message_key, gelf_level_key, body_key, and headers_key. Doesn't seems to have any buffer related stuff in this..

there's no way to increase that buffer in out_http. That buffer is used to read the response from the server