I've been out for Easter holidays as well so not been keeping up on any recent issues but worth checking the Github repo to see if anything reported

I'm trying to get metrics but it doesn't respond: [root@ip-10-1-202-170 ~]# curl -v --max-time 2 http://127.0.0.1:2020/api/v1/metrics/prometheus * Trying 127.0.0.1:2020... * Connected to 127.0.0.1 (127.0.0.1) port 2020 (#0)

GET /api/v1/metrics/prometheus HTTP/1.1

Host: 127.0.0.1:2020

User-Agent: curl/7.76.1

Accept: /

* Operation timed out after 2001 milliseconds with 0 bytes received * Closing connection 0 curl: (28) Operation timed out after 2001 milliseconds with 0 bytes received

nothing very exciting in the config (I hope): [SERVICE] Flush 5 parsers_file /etc/fluent-bit/parsers.conf [SERVICE] HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_PORT 2020 Health_Check On HC_Errors_Count 5 HC_Retry_Failure_Count 5 HC_Period 5 [INPUT] Name tail Path /var/log/httpd/*/*_access_log Tag raw_dianomi_access_log Alias tail_httpd_access_logs db /var/log/fluent_bit_access_log_tail.db [FILTER] Name aws Match raw_dianomi_access_log imds_version v2 hostname true private_ip true [FILTER] Name rewrite_tag Match raw_dianomi_access_log Rule $ec2_instance_id ^i.*$ dianomi_access_log_ec2|$ec2_instance_id false Emitter_Mem_Buf_Limit 256M [FILTER] Name rewrite_tag Match dianomi_access_log_ec2* Rule $private_ip ^.*$ private_$TAG|$private_ip false Emitter_Mem_Buf_Limit 256M [OUTPUT] Name s3 Match private_dianomi_access_log_ec2* region us-east-1 bucket dianomi-production-log-stream total_file_size 10M upload_timeout 10m s3_key_format /year=%Y/month=%m/day=%d/hour=%H/instance=$TAG[1]/private_ip=$TAG[2]/hostname=${HOSTNAME}/$UUID.gz s3_key_format_tag_delimiters | compression gzip Alias s3_httpd_access_logs [INPUT] name http listen 0.0.0.0 port 8888 Alias http_input [OUTPUT] Name kinesis_streams Match cpm_served region us-east-1 stream cpm_ad_serves Alias kinesis_cpm_ad_serves_output [OUTPUT] Name kinesis_streams Match cpm_viewed region us-east-1 stream cpm_ad_views Alias kinesis_cpm_ad_views_output [FILTER] Name aws Match app_errors imds_version v2 hostname true private_ip true ami_id true account_id true vpc_id true [FILTER] Name rewrite_tag Match app_errors Rule $ec2_instance_id ^i.*$ app_errors/error=500/instance=$ec2_instance_id false Emitter_Mem_Buf_Limit 256M [OUTPUT] Name cloudwatch_logs Match app_errors/* region us-east-1 log_group_name app_errors log_stream_prefix errors/ log_format json Alias cloudwatch_logs_app_errors auto_create_group On [INPUT] Name exec Tag fluentbit_metrics Command curl 'http://localhost:2020/api/v1/metrics' 2>/dev/null Interval_Sec 60 Interval_NSec 0 Buf_Size 8mb Oneshot false Alias exec_fluentbit_metrics_input [FILTER] name parser Parser json Match fluentbit_metrics key_name exec [FILTER] Name aws Match fluentbit_metrics imds_version v2 [FILTER] Name aws Match programmatic_bids imds_version v2 [FILTER] Name aws Match programmatic_bid_responses imds_version v2 [FILTER] Name aws Match programmatic_bid_response_rejected imds_version v2 [FILTER] Name aws Match programmatic_bid_response_win imds_version v2 [OUTPUT] Name cloudwatch_logs Match fluentbit_metrics region us-east-1 log_group_name fluentbit_metrics log_stream_name fluent-bit-metrics-fb log_format json/emf Alias cloudwatch_logs_fluentbit_metrics_output auto_create_group On [OUTPUT] Name s3 Match programmatic_bids region us-east-1 total_file_size 50M upload_timeout 5m bucket prod-programmatic-logs-incoming s3_key_format /programmatic_bids/year=%Y/month=%m/day=%d/hour=%H/$UUID.gz compression gzip Alias s3_programmatic_bids_output [OUTPUT] Name s3 Match programmatic_bid_responses region us-east-1 bucket prod-programmatic-logs-incoming total_file_size 50M upload_timeout 5m s3_key_format /programmatic_bid_responses/year=%Y/month=%m/day=%d/hour=%H/$UUID.gz compression gzip Alias s3_programmatic_bid_responses_output [OUTPUT] Name s3 Match programmatic_bid_response_rejected region us-east-1 bucket prod-programmatic-logs-incoming total_file_size 50M upload_timeout 5m s3_key_format /programmatic_bid_response_rejected/year=%Y/month=%m/day=%d/hour=%H/$UUID.gz compression gzip Alias s3_programmatic_bid_response_rejected_output [OUTPUT] Name s3 Match programmatic_bid_response_win region us-east-1 bucket prod-programmatic-logs-incoming total_file_size 50M upload_timeout 5m s3_key_format /programmatic_bid_response_win/year=%Y/month=%m/day=%d/hour=%H/$UUID.gz compression gzip Alias s3_programmatic_bid_response_win_output

the config previously worked

investigating, I've got that config running on 4.1.1 and 4.2.2 and it's working fine

the 5.0.3 release talks about http input changes, maybe that's related

ok disabling http2 doesn't help

giving it more http threads doesn't help

it seems to freeze around the time the kinesis/s3 scheduled stuff wake up

experimenting, i tried giving the tail input its own thread: Apr 23 142421 ip-10-1-213-152 fluent-bit[3557416]: [2026/04/23 142421.210] [debug] [inputtailtail_httpd_access_logs] failed buffer write, retries=0 Apr 23 142421 ip-10-1-213-152 fluent-bit[3557416]: [2026/04/23 142421.310] [debug] [inputtailtail_httpd_access_logs] failed buffer write, retries=1 Apr 23 142421 ip-10-1-213-152 fluent-bit[3557416]: [2026/04/23 142421.411] [debug] [inputtailtail_httpd_access_logs] failed buffer write, retries=2 Apr 23 142421 ip-10-1-213-152 fluent-bit[3557416]: [2026/04/23 142421.511] [debug] [inputtailtail_httpd_access_logs] failed buffer write, retries=3 Apr 23 142421 ip-10-1-213-152 fluent-bit[3557416]: [2026/04/23 142421.611] [debug] [inputtailtail_httpd_access_logs] failed buffer write, retries=4 Apr 23 142421 ip-10-1-213-152 fluent-bit[3557416]: [2026/04/23 142421.711] [debug] [inputtailtail_httpd_access_logs] failed buffer write, retries=5 Apr 23 142421 ip-10-1-213-152 fluent-bit[3557416]: [2026/04/23 142421.811] [debug] [inputtailtail_httpd_access_logs] failed buffer write, retries=6 Apr 23 142421 ip-10-1-213-152 fluent-bit[3557416]: [2026/04/23 142421.911] [debug] [inputtailtail_httpd_access_logs] failed buffer write, retries=7 Apr 23 142422 ip-10-1-213-152 fluent-bit[3557416]: [2026/04/23 142422.011] [debug] [inputtailtail_httpd_access_logs] failed buffer write, retries=8 Apr 23 142422 ip-10-1-213-152 fluent-bit[3557416]: [2026/04/23 142422.111] [debug] [inputtailtail_httpd_access_logs] failed buffer write, retries=9 Apr 23 142422 ip-10-1-213-152 fluent-bit[3557416]: [2026/04/23 142422.211] [error] [inputtailtail_httpd_access_logs] could not enqueue records into the ring buffer

it seemed to work when I enabled filesystem storage, but not sure if that's a real fix or just not hitting the dodgy code path

it used to work in fluentbit 4

in 5.03 I had to add "threaded true"

I have to admit using fluent to call itself is slightly odd, although I don't see why it shouldn't work