Hung
08/13/2025, 12:36 PMHung
08/13/2025, 12:38 PMparsers:
- format: regex
name: kube-log-tag-parser
regex: (?<pod_name>[^_]+)_(?<namespace_name>[^_]+)_(?<container_name>[^_]+)_(?<container_id>[a-z0-9]+)$
skip_empty_values: true
time_format: '%Y-%m-%dT%H:%M:%S.%L'
time_keep: true
time_key: time
time_strict: true
pipeline:
inputs:
- Tag: kube.var.logs.<pod_name>_<namespace_name>_<container_name>_<container_id>
Tag_Regex: .+/(?<pod_name>[^_]+)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]+)\.log$
buffer_chunk_size: 128KB
buffer_max_size: 128KB
multiline.parser: docker, cri, java
name: tail
path: /var/log/containers/health-seo-0_seo_cluster-controller-a043c85e36791270b7e4edbc6822d4a9a103b7fc500f6300226fb86a3048e967.log
processors:
logs:
- Buffer_Size: 2M
Cache_Use_Docker_Id: "On"
Kube_Meta_Cache_TTL: "30"
Kube_Tag_Prefix: kube.var.logs.
Merge_Log: "On"
Namespace_Annotations: "On"
Namespace_Labels: "On"
Regex_Parser: kube-log-tag-parser
name: kubernetes
Merge_Log_Key: "log_merged"
Keep_Log: "Off"
filters: []
outputs:
- name: stdout
match: '*'
sample of the output with 1 text line per record instead of 1 record with multiple lines
[785] kube.var.logs.health-seo-0_seo_cluster-controller_a043c85e36791270b7e4edbc6822d4a9a103b7fc500f6300226fb86a3048e967: [[1755084391.291220757, {}], {"log"=>"2025-08-13T07:26:31.284084088-04:00 stdout F at java.base@17.0.5/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
"}]
[786] kube.var.logs.health-seo-0_seo_cluster-controller_a043c85e36791270b7e4edbc6822d4a9a103b7fc500f6300226fb86a3048e967: [[1755084391.291220757, {}], {"log"=>"2025-08-13T07:26:31.284086331-04:00 stdout F at java.base@17.0.5/java.nio.file.Files.newByteChannel(Files.java:380)
"}]
[787] kube.var.logs.health-seo-0_seo_cluster-controller_a043c85e36791270b7e4edbc6822d4a9a103b7fc500f6300226fb86a3048e967: [[1755084391.291220757, {}], {"log"=>"2025-08-13T07:26:31.284088054-04:00 stdout F at java.base@17.0.5/java.nio.file.Files.newByteChannel(Files.java:432)
"}]Pat
08/13/2025, 12:40 PMPat
08/13/2025, 12:40 PMHung
08/13/2025, 12:41 PMPat
08/13/2025, 12:41 PMPat
08/13/2025, 12:42 PMPat
08/13/2025, 12:42 PMPat
08/13/2025, 12:43 PMHung
08/13/2025, 12:43 PM[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
multiline.parser docker, cri
I should add a parser filter afterwards?Hung
08/13/2025, 1:33 PMlog field as json ? to keep it as text (even it is in json format)?Hung
08/13/2025, 1:34 PMinputs:
- multiline.parser: cri, docker
name: tail
path: /var/log/containers/bar-query-*.log
...
processors:
logs:
- ...
Merge_Log: "On"
Namespace_Annotations: "On"
Namespace_Labels: "On"
name: kubernetes
- name: multiline
multiline.key_content: log
multiline.parser: java, go, pythonHung
08/13/2025, 1:35 PMlog field as text for json linesPat
08/13/2025, 5:24 PMPat
08/13/2025, 5:24 PMHung
08/14/2025, 3:42 PMPat
08/14/2025, 3:43 PMHung
08/14/2025, 3:44 PMHung
08/14/2025, 3:44 PMHung
08/14/2025, 3:44 PMHung
08/14/2025, 3:45 PMPat
08/14/2025, 3:45 PMHung
08/14/2025, 3:46 PM- Skip_Empty_Lines: "On"
Tag: kube.var.logs.<pod_name>_<namespace_name>_<container_name>_<container_id>
Tag_Regex: .+/(?<pod_name>[^_]+)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]+)\.log$
buffer_chunk_size: 128KB
buffer_max_size: 128KB
multiline.parser: cri, docker
name: tail
path: /var/log/containers/*.log
processors:
logs:
- buffer: "On"
multiline.key_content: log
multiline.parser: java, go, python
name: multiline
- action: insert
key: kubernetes_cluster
name: content_modifier
value: ${KUBENERTES_CLUSTER}
- action: insert
key: kubernetes_host
name: content_modifier
value: ${KUBERNETES_NODE_NAME}
- Buffer_Size: 2M
Cache_Use_Docker_Id: "On"
Kube_Meta_Cache_TTL: "30"
Kube_Tag_Prefix: kube.var.logs.
Merge_Log: "Off"
Namespace_Annotations: "On"
Namespace_Labels: "On"
Regex_Parser: kube-log-tag-parser
name: kubernetesPat
08/14/2025, 3:46 PMHung
08/14/2025, 3:47 PMlecaros
08/14/2025, 3:48 PMPat
08/14/2025, 3:48 PMlecaros
08/14/2025, 3:48 PMPat
08/14/2025, 3:48 PMHung
08/14/2025, 3:49 PMHung
08/14/2025, 3:49 PMPat
08/14/2025, 3:49 PMlecaros
08/14/2025, 3:49 PMPat
08/14/2025, 3:49 PMPat
08/14/2025, 3:50 PMHung
08/14/2025, 3:51 PMHung
08/14/2025, 3:51 PMPat
08/14/2025, 3:58 PMPat
08/14/2025, 4:12 PMlecaros
08/14/2025, 4:21 PMPat
08/14/2025, 4:50 PMlecaros
08/14/2025, 8:21 PMnaresh amrutham
08/15/2025, 4:14 AMPat
08/15/2025, 8:17 AMPat
08/15/2025, 1:30 PM