Hey all, I am trying to use my own self signed SSL...
# fluent-bit
g
Hey all, I am trying to use my own self signed SSL certs in a docker compose, so I can test my "production" config (which has SSL enabled) against a "local" prometheus etc. I can create a CA and add it to a
ca-certificates.crt
file, and then mount that into the fluent bit container at
/etc/ssl/certs/ca-certificates.crt
which I understand to be the correct location. However, unless I set
tls.ca_file: "/etc/ssl/certs/ca-certificates.crt"
for the prometheus output, I get SSL errors.
I am not really sure what is going wrong here
If I use the
-debug
build of fluent-bit, I can open a shell and use curl to check if the CA is set up correctly.
without mounting my CA file I get the expected SSL errors from curl when doing calls to the local prometheus, but if I do mount my CA into
/etc/ssl/certs/ca-certificates.crt
then curl works as expected
so, as far as I can see, mounting
/etc/ssl/certs/ca-certificates.crt
should work.
But it doesn't for FB
Is there some other location with a CA used by FB, where I should mount my file?
I think the file is good, because as I say, if I set
tls.ca_file
on the prom output, then it works
p
It should just use OpenSSL for all this...
However debugging SSL issues is always "fun"
Can you clarify the working vs non-working config?
g
šŸ¤”
So, the only way I can make this work is by adding
tls.ca_file: "/etc/ssl/certs/ca-certificates.crt"
to the prom output
like
Copy code
- name: prometheus_remote_write
      match: metrics.*
      host: ${PROMETHEUS_HOST}
      uri: /api/prom/push
      port: 443
      tls: on
      tls.verify: on
      tls.ca_file: "/etc/ssl/certs/ca-certificates.crt"
      http_user: ${PROMETHEUS_USER_ID}
      http_passwd: ${GRAFANA_TOKEN}
p
I think the docs say you do need to define it - otherwise it defaults to the empty string
g
I actually have an nginx reverse proxy to terminate the SSL, where I have my server certs etc
p
My guess is it is not using the system default but overriding it to empty string - I have no evidence for that at the moment though šŸ™‚
g
I think the docs say you do need to define it - otherwise it defaults to the empty string
But this works for regular HTTPS endpoints, which are not self signed
p
Ah, so it is using the trust store
g
ah, right šŸ¤”
p
I dunno, would have to look - maybe it's not updating the trust store when you mount it in
g
oh well
p
I recall some funny business using self-signed before where it was not good enough to just mount the new certs in but also tell the system they should be read šŸ˜ž
g
okay, thanks, I'll keep on digging
a
I think it needs to be
tls.crt_file
then ca_file is used on top
g
šŸ¤”
Wouldn't that be for an input?
where FB is serving an SSL endpoint?
It seemed like I needed some other random files from the
/etc/ssl/certs/
files from where I updated the CA, then it works okay
p
Ah ok, any idea what?
Be good to add it to the transport security docs page
g
Well, it is a mystery file called
728616e5.0
, which is a certificate file
/etc/ssl/certs
has a bunch of file, mostly sym links to
/usr/share/ca-certificates
etc.
p
Lol ah that does sound special, might be worth a general statement though that you may need to mount other files
g
When you add the CA by running
update-ca-certificates
it takes your CA and adds it to
/etc/ssl/certs/ca-certificates.crt
.
ca-certificates.crt
should then have everything needed, and indeed to can pass this to the output with
tls.ca_file: "/etc/ssl/certs/ca-certificates.crt"
and it will work.
and actually curl will work well with just this updated file mounted into the container
but FB also needs that other random file 🤷
p
Is that file part of the image then?
g
I think the advice should something like users need to mount a complete
/etc/ssl/certs/
dir, including all sym link targets
p
Yeah so the same as mounting container logs in that may be symlinks
Let's add a section then just to cover it for the inevitable next time
g
Is that file part of the image then?
Yes, I believe the image (including the non-debug distroless image), has a "complete" (?) debian based ssl file system layout
I am actually confused as to what counts as distroless, as of course
ssl
needs to be well configured, but how this is done is apparently different between distros
p
Yeah if you want more fun, try to use bazel to build the images lol
g
Anyway, my advice would be: • don't start by mounting the files you think you need (and have to add more until it works) • do start by mounting everything, and then remove stuff that you can confirm isn't needed šŸ˜…
šŸ’Æ 3
p
We actually take the certs from the builder stage: https://github.com/fluent/fluent-bit/blob/master/dockerfiles%2FDockerfile#L199 The builder stage is a full Debian image
g
Yes, I saw that šŸ˜„
p
I wonder if there's anything we can do to improve either usage or debugging for ssl there? Problem likely though is it's hard to not break things if you start fiddling!
g
If
tls.ca_file
defaulted to
/etc/ssl/certs/ca-certificates.crt
, and was documented as such, it would give a hint as to how to override this (mounting over that file), and such a simple, single mount would then work for my use case.
If people want to use their own CA, I would imagine they just mount it under a user path and configure the output. My use case is much more "hacky", in that I want a "vanilla" prd image to work in local docker compose, with self signed certs for prometheus etc
p
I think the issue with the default is that's it's not the same on all platforms
g
right, of course