Hello, we use at the moment NXlog, which forwards...
# fluent-bit
d
Hello, we use at the moment NXlog, which forwards messages to Graylog and old style (R)Syslogd. I'm trying out fluentd and fluent-bit in parallel. We have in our Nxlog something like this:
Copy code
$message =~ s/cardNo>\d\d\d\d\d\d\d\d\d\d\d\d/cardNo>XXXXXXXXXXXX/g;
	 $message =~ s/cvv>\d\d\d/cvv>XXX/g;
	 $message =~ s/cvv=\d\d\d/cvv=XXX/g;
	 $message =~ s/Verification%3E\d\d\d/Verification%3EXXX/g;
	 $message =~ s/R19%3E\d\d\d/R19%3EXXX/g;
	 $message =~ s/R16%3E\d\d\d\d\d\d\d\d\d\d\d\d/R16%3EXXXXXXXXXXXX/g;
	 $message =~ s/ccn=\d\d\d\d\d\d\d\d\d\d\d\d/ccn=XXXXXXXXXXXX/g;
	 $message =~ s/card_account=\d\d\d\d\d\d\d\d\d\d\d\d/card_account=XXXXXXXXXXXX/g;
	 $message =~ s/cvc_code=\d\d\d/cvc_code=XXX/g;
	 $message =~ s/password=/Password=XXX\&/g;
to replace sensitive creditcard data. How we can do the same in fluent-bit ?
p
Quite a few ways to be honest, LUA may be the simplest - I used it to hash the sensitive strings before.
Hashing is useful because you remove the sensitive data but end up with a hash you can correlate across logs/etc. if needed
d
hi, this nightfall I saw, but I would like avoid it. Better to use, what fluent-bit has 🙂
p
Yup, I don't think Nightfall is the right option because you send your private data out first to redact it lol
LUA is the easiest way I'd say and you can provide a lot of flexibility around it then
We have a simple hosted LUA sandbox here to experiment with if you want: https://fluentdo.github.io/lua-sandbox/
d
Phu 🙂 Not so easy as I assumed. Nxlog was pretty nice .. but they going dark .. so better to find a solution for it. Fluentd is much more complicated .. I will try the LUA way. thanks
p
LUA is simple once you get into it, once you have one filter you will find more uses and it will grow 😄
LUA has a simple global substitution - no regex support directly because that is expensive but substitution can do it instead. I just did something like this for another customer to extract the short name of the image as an example:
Copy code
if (record.kubernetes and record.kubernetes.container_image) then
--[[
Attempt to extract the short image name using regex: '^.*/<short name>:.*$'.
LUA does not have regex support so instead we substitute empty strings in.
]]--
short_image = string.gsub(record.kubernetes.container_image, "^.*/", "")
record['short_image'] = string.lower(string.gsub(short_image, ":.*$", ""))
record['service'] = record['short_image']
record['source'] = record['short_image']
end
If you need a hand we can always help 🙂 fluent.do
d
Thanks a lot 🙂
e
if you just need hashing, you can use Content Modifier processor with the
hash
action https://docs.fluentbit.io/manual/pipeline/processors/content-modifier#actions
p
Ah yeah, forgot we added that 🙂
d
hi, thanks for this hint, but no, really replace 🙂
p
Replace or remove is easy enough with content modifier or one of the other filters
Those NX logs look just like sed which is just global substitution same as LUA
d
Ok, thx 🙂
p
https://fluentdo.github.io/lua-sandbox/ is a sandbox running Fluent Bit to let you play with LUA and test data
There's also a few online LUA sandboxes to just play with LUA code too, e.g. https://onecompiler.com/lua/