Anyone using the OpenSearch output on the fluentbi...
# fluent-operatord
David Tippett
10/31/2023, 1:33 PMAnyone using the OpenSearch output on the fluentbit operator? I am getting the exception
Invalid value: "string": spec.opensearch.httpPassword in body must be of type object: "string""httpPassword": "admin""httpPassword": adminp
Pat
10/31/2023, 1:36 PMwhat does a
kubectl explainPat
10/31/2023, 1:37 PMIt looks to be a reference to plugins secret type so my guess is it is a nested object? https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/plugins/output/open_search_types.go#L50
Pat
10/31/2023, 1:38 PMYeah it seems to want a `valueFrom`: https://github.com/fluent/fluent-operator/blob/44936259dbe2bd7cc4a53647d57ad02e1b7991c9/apis/fluentbit/v1alpha2/plugins/secret_types.go#L15-L17
d
David Tippett
10/31/2023, 1:41 PMAhhhhhhhh that makes so much more sense. I am a bit new to this so I will need to create a secret that contains the httpPassword and httpUser?
David Tippett
10/31/2023, 1:41 PMThen pass the reference to the secret into the config?
p
Pat
10/31/2023, 1:41 PMlooks like it, which makes sense rather than specifying fixed values
Pat
10/31/2023, 1:41 PMI've not used it I'm afraid but
kubectl explaind
David Tippett
10/31/2023, 1:42 PMthe docsDavid Tippett
10/31/2023, 1:42 PMThanks for the kubectl expain tip tho didn't realize that was a thing
p
Pat
10/31/2023, 1:43 PMyeah, plus the docs and project are OSS so feel free to add specific examples 🙂
Pat
10/31/2023, 1:43 PMit's hard for someone to document every possible use case so a collaborative and emerging thing I think as we go along
💯 1
d
David Tippett
10/31/2023, 1:43 PM100% will be when I get it running
p
Pat
10/31/2023, 1:44 PMmy guess is the Elasticsearch example may be close enough but I could be wrong, I avoid both if I can help it 😛
Pat
10/31/2023, 1:44 PM@Benjamin Huo on the docs side, did we decide with @Anurag Gupta as well whether to push a full docs site up like fluentbit.io?
d
David Tippett
10/31/2023, 1:53 PMYeah the ES one was my starting point but seems like the spec's have diverged a fair bit.
p
Pat
10/31/2023, 1:54 PMThat's probably more a question for Elastic and Opensearch folks 🙂
d
David Tippett
10/31/2023, 1:54 PMHi nice to meet you, I'm the senior advocate for OpenSearch 🥲
David Tippett
10/31/2023, 1:54 PMhaha
David Tippett
10/31/2023, 1:55 PMOn a serious note kubectl explain was the piece I was missing
p
Pat
10/31/2023, 1:55 PMlol ok, I was going to say I know their has been explicit Opensearch contribution which is why it is diverging
d
David Tippett
10/31/2023, 1:55 PMYeah, they probably did that for security as we try to have everything be secure by default
p
Pat
10/31/2023, 1:55 PMwe did also do a fluent operator walkthrough in a sandbox for Kubecon last year that might be good to update for people to use directly then as a working example
d
David Tippett
10/31/2023, 1:56 PMYeah that is the hope, I'll add a walkthrough in the manifests/logging-stack dir when Im finished
p
Pat
10/31/2023, 1:57 PMthis one: https://github.com/kubesphere-sigs/fluent-operator-walkthrough
Quite a while ago I did this as well to help folks, maybe good to add an Opensearch one to it: https://github.com/calyptia/fluent-bit-devtools
We also use Opensearch in the Fluent Bit integration tests as well but that's more CI around Terraform and BATS.
Pat
10/31/2023, 1:58 PMfeel free to ping me direct btw as well
d
David Tippett
10/31/2023, 1:59 PMSounds good and ill look over those resources! Thanks pat!
p
Pat
10/31/2023, 2:00 PMno worries, if you're at Kubecon NA then there'll be a Fluent booth up as well (and a Calyptia one) - I won't be there though as in the UK
b
Benjamin Huo
10/31/2023, 2:03 PM@Wanjun Lei @elon cheng how should we pass the username and password for open search? Using secret?
p
Pat
10/31/2023, 2:04 PMit seems a good option to me, the alternative is optional fields for a string and secret ref to allow a hard-coded one for POCs but realistically you never want that in prod so referencing a secret to then support auto-reloading via K8S when the secret changes is good
d
David Tippett
10/31/2023, 2:46 PMSo here is how I've done it which feels good enough for most POC's
David Tippett
10/31/2023, 2:46 PMDavid Tippett
10/31/2023, 2:46 PMOnly issue is this is still not shipping logs into OpenSearch.
e
elon cheng
11/03/2023, 7:37 AMFor opensearch, es username and passwords, we use the form of secret mount, in my opinion, this is more secure, if there are any better suggestions, feel free to submit issue.
2 Views