Deepak
08/19/2025, 9:27 AMDeepak
08/19/2025, 9:33 AMDeepak
08/19/2025, 10:12 AMDeepak
08/20/2025, 10:30 PMlecaros
08/21/2025, 2:12 PMDeepak
08/22/2025, 1:05 PMcurl -s <http://127.0.0.1:2020/api/v1/health>) is invoked, the context does not restart. Based on the log messages, it appears to hang after flushing six times on the HTTP output. Log writing also stops at this point, so the logs below are the last entries we receive.
[2025/08/19 00:00:11] [engine] caught signal (SIGHUP)
[2025/08/19 00:00:12] [ info] reloading instance pid=1 tid=0x7fef45c56ec0
[2025/08/19 00:00:12] [ info] [reload] stop everything of the old context
[2025/08/19 00:00:12] [ warn] [engine] service will shutdown when all remaining tasks are flushed
[2025/08/19 00:00:12] [ info] [input] pausing storage_backlog.1
[2025/08/19 00:00:12] [ info] [engine] service has stopped (0 pending tasks)
[2025/08/19 00:00:12] [ info] [input] pausing storage_backlog.1
[2025/08/19 00:00:12] [ info] [output:http:http.0] thread worker #0 stopping...
[2025/08/19 00:00:12] [ info] [output:http:http.0] thread worker #0 stopped
[2025/08/19 00:00:12] [ info] [output:http:http.0] thread worker #1 stopping...
[2025/08/19 00:00:12] [ info] [output:http:http.0] thread worker #1 stopped
[2025/08/19 00:00:12] [ info] [output:http:http.0] thread worker #2 stopping...
[2025/08/19 00:00:12] [ info] [output:http:http.0] thread worker #2 stopped
[2025/08/19 00:00:12] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=18005413420718882816 watch_fd=1
[2025/08/19 00:00:12] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=13034560727803756544 watch_fd=2
[2025/08/19 00:00:12] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=16505741133083574272 watch_fd=3
[2025/08/19 00:00:12] [ warn] [engine] service will shutdown when all remaining tasks are flushed [2025/08/16 00:00:12] [info] [input] pausing storage_backlog.1
[2025/08/19 00:00:12] [info] [task] tail/tail.0 has pending task(s):
[2025/08/19 00:00:12] [info] [task] storage_backlog/storage_backlog.1 has pending task(s): [2025/08/16 00:00:12] [info] [engine] pending chunk count: memory=0, filesystem=2; grace_timer=1
[2025/08/19 00:00:12] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200
[2025/08/19 00:00:12] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200
[2025/08/19 00:00:12] [ info] [input] pausing storage_backlog.1
[2025/08/19 00:00:13] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200
[2025/08/19 00:00:13] [info] [engine] pending chunk count: memory=0, filesystem-2; grace_timer-2
[2025/08/19 00:00:13] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200
[2025/08/19 00:00:13] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200
[2025/08/19 00:00:15] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status-200
[2025/08/19 00:00:15] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200
[2025/08/19 00:00:15] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200
[2025/08/19 00:00:15] [info] [output:http:http.0] clickhouse-chproxy.clickhouse-cluster.svc.cluster.local:8080, HTTP status=200
---- after this there was no log and also no data ingestion was happening---Deepak
08/22/2025, 1:06 PMDeepak
08/22/2025, 1:08 PMeduardo
08/22/2025, 2:57 PMDeepak
08/23/2025, 2:31 PMDeepak
08/26/2025, 9:59 AMgrace_timer, so that on restart it sends the remaining chunks?
[2025/08/26 09:55:19] [ info] [engine] pending chunk count: memory=0, filesystem=24; grace_timer=9970
[2025/08/26 09:55:20] [ info] [engine] pending chunk count: memory=0, filesystem=24; grace_timer=9971
[2025/08/26 09:55:21] [ info] [engine] pending chunk count: memory=0, filesystem=24; grace_timer=9972Deepak
08/26/2025, 10:57 AMHot_Reload.Ensure_Thread_Safety true
Shutdown_Grace 5 Storage.delete_irrecoverable_chunks on
Storage.backlog.flush_on_shutdown Off
[OUTPUT]
Retry_Limit no_limitseduardo
08/30/2025, 2:59 AMeduardo
08/30/2025, 3:11 AMDeepak
09/01/2025, 11:14 AM