Hi . I'm using fluentbit to push EKS logs to OpenS...
# fluent-bit
n
Hi . I'm using fluentbit to push EKS logs to OpenSearch and I see these indices being generated with random incorrect timestamps. How to fix these ? appreciate your help. green open eks-genesis-test-2033.01.29 nGvSt_ngS1CAe_T9YvOUNQ 2 0 1 0 18.1kb 18.1kb green open eks-genesis-test-2014.08.02 5TPLe1M6Rm-nZRCag8hvQQ 2 0 1 0 18.7kb 18.7kb
p
What's your full config and version? There's not really enough here to help. Timestamps typically are parsed from the input data and then sent on
I'd also encourage you to use stdout to see the actual timestamp Fluent Bit is using to confirm it's not an issue with ingestion or something else in your stack
n
@Pat below is the config..
Copy code
custom_parsers.conf: |
    [PARSER]
        Name cri
        Format regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File /fluent-bit/etc/parsers.conf
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Exclude_Path     /var/log/containers/*_kube-system_*.log,/var/log/containers/*_kubernetes-dashboard_*.log,/var/log/containers/*_monitoring_*.log,/var/log/containers/*_goldilocks_*.log,/var/log/containers/*_logging_*.log,/var/log/containers/*_vpa_*.log,/var/log/containers/*_test-fb_*,/var/log/containers/*_logging-fb_*,/var/log/containers/*_confluent-connect_*,/var/log/containers/*_confluent-connect-stage_*,/var/log/containers/*_custom-metrics_*,/var/log/containers/*_logging-test_*.log
        multiline.parser cri
        DB               /var/log/flb_kube.db
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On
        Refresh_Interval   15

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        #Keep_Log            Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

    [OUTPUT]
        Name               es
        Match              *
        Host               xxxxxxxxxx
        Port               443
        Type               doc
        TLS                On
        tls.verify         On
        Logstash_Format    On
        Logstash_Prefix    eks-genesis-test
        #Logstash_DateFormat %Y-%m-%d
        Buffer_Size        False
        AWS_Auth           On
        AWS_Region         us-east-1
        Retry_Limit        5
        Trace_Error        On
        Trace_Output       On
        Replace_Dots       On
        Suppress_Type_Name On
        Generate_ID        On
p
I'd probably use the Opensearch output to be honest but also add stdout to see the actual timestamps you're getting
n
use opensearch instead of es for the output plugin ?