Hi, wondering if anyone has gone through regex tes...
# fluent-operatord
Dale Hille
10/03/2023, 9:48 AMHi, wondering if anyone has gone through regex testing strategies when using the fluent-operator. We recently had a scenario where an invalid regex in a ClusterParser was merged which caused crashloopbackoff of our fluentbit pods so I’m working on ways to get that tested as part of CI. Has anybody gone down this road before?
p
Pat
10/03/2023, 9:56 AMOnly real way to test is try and verify against known input + expected output then roll back. Probably want to apply on a test pipeline then roll out if passes.
We do something like this in fleet management, although it's a more complete rollback to previous working config if it fails to prevent accidental foot guns destroying the entire fleet of agents
d
Dale Hille
10/03/2023, 10:00 AMI was going to see if I could get onigmo built as part of our build system so it could be used to validate any regex’s found in ClusterParser files. basically a simple regex validation tool. I’ve only just started looking at the onigmo repo so am not sure if that’s a workable strategy but seems like it would from a high level. Have you done anything like that before?
p
Pat
10/03/2023, 10:02 AMI wouldn't, I'd just run Fluent Bit as a container to validate things
Pat
10/03/2023, 10:03 AMI put together some regression tests at Couchbase around expected output: https://github.com/couchbase/couchbase-fluent-bit/blob/main/test/run-tests.sh
see input vs expected output here: https://github.com/couchbase/couchbase-fluent-bit/tree/main/test/logs
Pat
10/03/2023, 10:04 AMit's not pretty but you could do the same thing in principle to verify things like the parser is being applied - I made sure parsers never had th default
logexpectPat
10/03/2023, 10:05 AMyou could build onigmo but have you built the same version with the same flags/config as FB version, is there anything else outside it that is affected (e.g. string escaping), etc.? Seems a lot of work when a container could verify things much more easily, plus it can be used to test FB upgrades across versions both for accuracy and performance against your actual data + config.
d
Dale Hille
10/03/2023, 10:06 AMthose are very helpful, thank you! I’ve considered the container approach as well but I think I was getting hung up on our use of the fluent-operator and needing to have that deployed in order to generate the final config that fluent-bit expects
p
Pat
10/03/2023, 10:06 AMYou could run this as a job in the cluster that takes a config to verify then promotes it to the config to use more widely maybe
Pat
10/03/2023, 10:06 AMYou could even run operator across namespaces using one namespace to test the config then acts as a gate to apply it to the "prod" namespace.
d
Dale Hille
10/03/2023, 10:10 AMthanks. I’ve been thinking about testing things prior to having a k8s environment ready, trying to keep things simple, but maybe that in itself will end up being more complicated than just having more of an e2e type of environment setup…
p
Pat
10/03/2023, 10:11 AMfor my couchbase stuff that was just a Jenkins job, hence the bash, running a local container then a simple diff on the actual output vs expected
Pat
10/03/2023, 10:11 AMit actually came in useful to review changes as well, e.g. if you add extra metadata then the expected output needs updating to match so you can review both the actual change and what it does in a PR
👍 1
Pat
10/03/2023, 10:12 AMunfortunately all that is in the delights of Gerrit rather than Github to see 🙂
d
Dale Hille
10/03/2023, 10:14 AMon a related note, also been looking at https://docs.fluentbit.io/manual/pipeline/filters/expect but this seems a bit tricky to have if you’re also running the fluent-operator. Have you used
expectp
Pat
10/03/2023, 10:16 AMno, I used it in that offline pipeline mate
Pat
10/03/2023, 10:16 AMrealistically I think you have to use it in a job where you check the exit code then work on that, e.g. roll back or approve
Pat
10/03/2023, 10:17 AMor use it combined with downstream alerting to look for the warning output if you don't go with exit
d
Dale Hille
10/03/2023, 10:19 AMyeah we would only want the expect filters to be applied during testing which makes things a bit more complicated since the order of filters is controlled alphabetically with the fluent-operator
p
Pat
10/03/2023, 10:19 AMoh right, that's a bug I'd say - filter ordering should be completely controllable
Pat
10/03/2023, 10:20 AMyou could also just add a lua filter that does your checks and exits explicitly
d
Dale Hille
10/03/2023, 10:20 AMoh I didn’t consider the lua option.. will have to think on that a bit, thank you
p
Pat
10/03/2023, 10:21 AMNo worries, personally I think you're over complicating things with the operator - you want to verify config (and maybe FB upgrades later) works so just run a simple job or local container to do that.
Once that is done the config can be applied via the operator separately.
Pat
10/03/2023, 10:22 AMIf you want to test parsers, etc. people will thank you to do it all locally rather than via K8S and checking log/exit codes.
d
Dale Hille
10/03/2023, 10:28 AMthe local container option you originally proposed is just a fluent-bit container though right? which would need to be passed the final config to be validated. but that config is generated by the operator so I’m not sure how we could eliminate the operator from the testing setup. but it’s early here so I’m most likely missing something 🙂
p
Pat
10/03/2023, 11:25 AMthe operator config though ultimately should be fluent bit config but yeah it's best to test everything as it is deployed - you could even get the operator to provide the config map but no actual pods (e.g. a node selector that won't match or even better a dedicated option for config generation only)