Hi, I want to collect logs of a particular docker ...
# fluent-bit
s
Hi, I want to collect logs of a particular docker container using fluent-bit container, how can i do it? compose-file
Copy code
services:
  fluentbit:
    image: fluent/fluent-bit:latest
    container_name: fluentbit
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
      - ./parsers.conf:/fluent-bit/etc/parsers.conf:ro
    restart: unless-stopped
fluent-bit.conf
Copy code
[SERVICE]
    Flush        1
    Log_Level    info
    Parsers_File parsers.conf

[INPUT]
    Name              tail
    Path              /var/lib/docker/containers/*/*.log
    Parser            docker
    Tag               all-containers

# [FILTER]
#     Name              grep
#     Match             all-containers
#     Regex             container_name bpm

[OUTPUT]
    Name              stdout
    Match             all-containers
parsers.conf
Copy code
[PARSER]
    Name   docker
    Format json
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep On
    Decode_Field_As escaped_utf8 log
Hi @Anurag Gupta Can you please help me out here?
p
You need to identify the container in some way. I'm guessing you're just running everything locally rather than K8S?
It may be better to just configure the docker daemon to send logs for that specific container via the fluent forward protocol to a Fluent Bit container to receive them
https://docs.docker.com/engine/logging/drivers/fluentd/ - fluentd can be replaced with fluent bit here, they both talk the same fluent forward protocol
s
Hi @Pat, Yes I'm running locally in docker. without driver is there other way? as I already other driver
p
How would you identify this container? By filename, if so use that?
You're being quite vague, what is it you actually need to do? Find a single container in a huge mass of container log files, or do something per container or what?
s
i have 3 container already running backend ,db and kafka i want to collect logs of backend container
p
you need the container ID of that container then to filter the logs with
s
yes but everytime i restart backend container id changes..
p
yes, that's the hard part
s
is there no way to get by name?
p
what logging driver are you using?
the container runtime is the thing responsible for naming the log files, there are potential options to do that: https://stackoverflow.com/a/64008686
s
json-file
p
otherwise you could add something into your actual log output to identify it as
backend
and filter those lines out, e.g. always log it with that
so log tags basically do that for the driver you have it seems
every log message will include an additional tag you can use to allow/exclude it, check that post I shared above
s
Sure thanks.