Sahil
07/14/2025, 4:53 AM[PARSER]
Name airflow
Format regex
Regex ^(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}) (?<log>.*)
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S
Time_Keep On
[PARSER]
Name docker_no_time
Format json
Time_Keep Off
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
fluent-bit.conf
[SERVICE]
Flush 1
Daemon On
Log_Level info
Config_Watch On
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Parsers_File /opt/bitnami/fluent-bit/conf/custom_parsers.conf
[INPUT]
Name tail
Tag airflow.task
Path /opt/bitnami/airflow/logs/dag_id=*/run_id=*/task_id=*/**/*.log
Parser airflow
Read_from_Head On
Refresh_Interval 5
Skip_Long_Lines On
Mem_Buf_Limit 100MB
DB /buffers/airflow_task.db
[OUTPUT]
# output config
Fluent Bit Logs:
[2025/07/14 04:51:11] [ info] [fluent bit] version=4.0.4, commit=cdb6c4a6e1, pid=1
[2025/07/14 04:51:11] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/07/14 04:51:11] [ info] [simd ] disabled
[2025/07/14 04:51:11] [ info] [cmetrics] version=1.0.4
[2025/07/14 04:51:11] [ info] [ctraces ] version=0.6.6
[2025/07/14 04:51:11] [ info] [input:cpu:cpu.0] initializing
[2025/07/14 04:51:11] [ info] [input:cpu:cpu.0] storage_strategy='memory' (memory only)
[2025/07/14 04:51:11] [ info] [sp] stream processor started
[2025/07/14 04:51:11] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[2025/07/14 04:51:11] [ info] [output:stdout:stdout.0] worker #0 started
[0] cpu.local: [[1752468671.539507144, {}], {"cpu_p"=>2.571429, ... }]Pat
07/14/2025, 8:52 AMPat
07/14/2025, 2:07 PMPat
07/14/2025, 2:08 PMSahil
07/15/2025, 2:53 AMSahil
07/15/2025, 2:53 AMSahil
07/15/2025, 2:53 AM