hi, I try to mask sensitive data, and it works for...
# fluent-bit
d
hi, I try to mask sensitive data, and it works for my syslog output, if I use something like "logger password=foo", but If I try something like
curl -k  <https://172.16.0.102:22077/backoffice/status/password=secret>
I can see password on my Graylog and Victorialogs output.
I'm not sure, what's wrong, because:
Copy code
filters:
    - Name: modify
      Match: graylog
      Add: version 1.1
      Add: host ${HOSTNAME}
      Rename: log short_message
    - Name: modify
      Match: "ns.*"
      Add: version 1.1
      Add: host ${HOSTNAME}
    - name: lua
      match: syslog
      script: pritolevel.lua
      call: pritolevel
    - name: lua
      match: "*"
      script: sensitive_replace.lua
      call: mask_fields
....
  outputs:
    - name: gelf
      match: "ns.*"
      host: 172.16.0.206
      port: 12201
      mode: tcp
      Gelf_Short_Message_Key: log
      Gelf_Timestamp_Key:    timestamp
      Gelf_Host_Key:    host
      Gelf_Level_Key:    level
    - name: syslog
      match: syslog
      host: 192.168.1.5
      port: 514
      tls: off
      tls.verify: off
      mode: udp
      syslog_format: rfc5424
      syslog_maxsize: 2048
      syslog_severity_key:  severity
      syslog_facility_key:  facility
      syslog_hostname_key:      host
      syslog_appname_key:   ident
      syslog_procid_key:    pid
      syslog_msgid_key:     msgid
      syslog_sd_key:        sd
      syslog_message_key:   message
A example
Even if I use:
logger curl -k <https://172.16.0.102:22077/backoffice/status/password=secret>
it gets masked on syslog output, but not on the other outputs.
p
I'm guessing your masking is being done by the lua script?
What does it do and I imagine that's what you need to resolve?
Also you can just use the stdout output to test, no need to deal with the backend
d
Hi Pat, nice to see you again 🙂
Yes, lua, but while driving home, I had an idea ..
I have this nice lua:
Copy code
-- File: mask_sensitive.lua
function mask_fields(tag, timestamp, record)
    local message = record["message"]

    if message ~= nil then
        -- Mask patterns
        message = string.gsub(message, "cardNo>%d%d%d%d%d%d%d%d%d%d%d%d", "cardNo>XXXXXXXXXXXX")
        message = string.gsub(message, "cvv>%d%d%d", "cvv>XXX")
        message = string.gsub(message, "cvv=%d%d%d", "cvv=XXX")
        message = string.gsub(message, "Verification%%3E%d%d%d", "Verification%%3EXXX")
        message = string.gsub(message, "R19%%3E%d%d%d", "R19%%3EXXX")
        message = string.gsub(message, "R16%%3E%d%d%d%d%d%d%d%d%d%d%d%d", "R16%%3EXXXXXXXXXXXX")
        message = string.gsub(message, "ccn=%d%d%d%d%d%d%d%d%d%d%d%d", "ccn=XXXXXXXXXXXX")
        message = string.gsub(message, "card_account=%d%d%d%d%d%d%d%d%d%d%d%d", "card_account=XXXXXXXXXXXX")
        message = string.gsub(message, "cvc_code=%d%d%d", "cvc_code=XXX")
        message = string.gsub(message, "password=[^,]*", "Password=XXX")
    end

    record["message"] = message
    return 1, timestamp, record
end
and I guess, the issue is: it works, because of "message", what for syslog is used, but the logs I want to mask also, are coming via Docker -> fluentd (Docker plugin) -> fluent-bit, which is not "message", but "log".
I tried it and cloned the config .. it works
but .. is it possible to make it independent ? Regardless if it is message / log / whatever ?
p
Sure, lua could iterate over all the keys. You could also use another filter to ensure a fixed key name is always used too
d
you mean, rename "log" to messages?
hmm
p
Sure but honestly it's pretty easy to get lua to work over multiple records, have a look at the online docs for how to iterate over the keys
d
thanks, got it working 🙂