Hi Beginner question ...What am I doing wrong ? - ...
# fluent-bit
a
Hi Beginner question ...What am I doing wrong ? • Trying to use fluent-bit to move default namespace logs to associated log-group • Used the following to deploy fluent-bit in EKS environment
helm upgrade fluent-bit fluent/fluent-bit --namespace logging --values values.yaml
• Works well for services with string like logs, yet, no forwarding for services with json like logs (
kubectl log pod_name
).
Copy code
serviceAccount:
    create: true
    name: fluent-bit
    annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::${AccountID}:role/FluentBitCloudWatchRole

config:
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines Off
        Buffer_Chunk_Size 64KB
        Buffer_Max_Size 128KB

  filters: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            <https://kubernetes.default.svc.cluster.local:443>
        Merge_Log           On
        Merge_Log_Key       data
        Keep_Log            On
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
        Buffer_Size         32k

    [FILTER]
        Name                  rewrite_tag
        Match                 kube.*
        Rule                  $kubernetes['namespace_name'] "^default$" cloudwatch.default true
        Rule                  $kubernetes['namespace_name'] ".*" cloudwatch.others true

  outputs: |
    [OUTPUT]
        Name                  cloudwatch_logs
        Match                 cloudwatch.default
        region                us-west-2
        log_group_template    /aws/eks/eks-qa/workload/$kubernetes['container_name']
        log_stream_template   $kubernetes['pod_name']
        log_group_name        /aws/eks/eks-qa/workload/kubernetes-logs
        log_stream_prefix     log_stream_prefix
        auto_create_group     true
        log_retention_days    90

    [OUTPUT]
        Name                  cloudwatch_logs
        Match                 cloudwatch.others
        region                us-west-2
        log_group_name        /aws/eks/eks-qa/workload/kubernetes-logs
        log_stream_template   $kubernetes['namespace_name'].$kubernetes['pod_name']
        log_stream_name       kubernetes-log-streams
        auto_create_group     true
        log_retention_days    90
p
The log content should have no impact. All logs are converted to an internal structure.
a
@Pat My apologies for the late response ... still not used with slack .... Not sure I've understood you're answer. Can you please elaborate ?