Nelson
06/26/2025, 10:19 AMPat
06/26/2025, 10:20 AMPat
06/26/2025, 10:21 AMPat
06/26/2025, 10:21 AMPat
06/26/2025, 10:23 AMNelson
06/26/2025, 10:24 AMPat
06/26/2025, 10:29 AMPat
06/26/2025, 10:30 AMPat
06/26/2025, 10:33 AMNelson
06/26/2025, 10:43 AMPat
06/26/2025, 10:44 AMPat
06/26/2025, 10:45 AMPat
06/26/2025, 10:46 AMNelson
06/26/2025, 10:47 AMNelson
06/26/2025, 10:51 AM[SERVICE]
daemon false
grace 30
http_Port 2020
http_listen 0.0.0.0
http_server true
log_level info
parsers_file /fluent-bit/etc/parsers.conf
storage.backlog.flush_on_shutdown on
storage.metrics true
storage.path ${STORAGE_PATH}
[INPUT]
name opentelemetry
alias otlp-app
listen 0.0.0.0
port 4318
tag_from_uri false
tag app
threaded true
storage.type filesystem
[OUTPUT]
name opentelemetry
alias otlp-app-us-east-1a
match app*
host vlinsert.vlogs-us-east-1a.svc.cluster.local
port 9481
logs_uri /insert/opentelemetry/v1/logs
header AccountID 1
header VL-Stream-Fields environment,hostname,role
workers 1
log_level warn
retry_limit false
storage.total_limit_size 10G
[OUTPUT]
name opentelemetry
alias otlp-app-us-east-1b
match app*
host vlinsert.vlogs-us-east-1b.svc.cluster.local
port 9481
logs_uri /insert/opentelemetry/v1/logs
header AccountID 1
header VL-Stream-Fields environment,hostname,role
workers 1
log_level warn
retry_limit false
storage.total_limit_size 10GNelson
06/26/2025, 10:52 AMPat
06/26/2025, 10:52 AMPat
06/26/2025, 10:55 AMstorage.max_chunks_up as the default is 128:
Chunks are never much larger than 2 MB,
so with the default storage.max_chunks_up value of 128, each input is limited to
roughly 256 MB of memory.Pat
06/26/2025, 10:55 AMNathan Ching
06/26/2025, 3:52 PMretry_limit no_retries on outputs. This almost gives the failover behaviour we are after in that if one output goes down it's not limiting throughput to the other output, however it still lacks some resilience for transient issues and maybe can loose data with surges. I had half a thought to add buffering downstream, roughly like this but was wondering about simpler setup that maybe gives similar behaviour:Nathan Ching
06/26/2025, 3:57 PMPat
06/26/2025, 5:15 PMPat
06/26/2025, 5:16 PMLeonardo
06/26/2025, 5:21 PMNathan Ching
06/26/2025, 6:01 PMPat
06/26/2025, 7:03 PMLeonardo
06/26/2025, 7:41 PMLeonardo
06/26/2025, 7:41 PM