ldaps issues
# troubleshooting
k
I got a cert and copied it to the lldap data directory for testing but I am having issues binding to ldaps, specifically I'm trying to test this with bookstack. It just says it can't contact LDAP server. if I switch to plain ldap in the bookstack config it works but not ldaps. Bookstack:
Copy code
AUTH_METHOD=ldap
LDAP_SERVER=ldaps://ldapsserver:6360
LDAP_BASE_DN=ou=people,dc=domain,dc=tld
LDAP_DN=uid=admin,ou=people,dc=domain,dc=tld
LDAP_PASS=password
LDAP_USER_FILTER=(&(uid=${user}))
LDAP_VERSION=3
LDAP_ID_ATTRIBUTE=uid
LDAP_EMAIL_ATTRIBUTE=mail
LDAP_DISPLAY_NAME_ATTRIBUTE=cn
LLDAP ldaps settings:
Copy code
[ldaps_options]
enabled=true
port=6360
cert_file="/data/ldap/cert1.pem"
key_file="/data/ldap/privkey1.pem"
I can see in the logs that it does load those settings and I am setting 6360:6360 in my docker-compose.yml any ideas?
n
Just checking the basics: in the LLDAP logs, do you see at the beginning that it's listening on port 6360 with LDAPS?
Any errors in the LLDAP logs/any trace of the request from bookstack?
And finally, any error logs from bookstack?
k
i currently have verbose=true, should i turn that off?
n
no, that's fine
(that's even better, actually)
k
it does say Starting the LDAPS server on port 6360
this is what bookstack says:
Copy code
ErrorException
ldap_bind(): Unable to bind to server: Can't contact LDAP server
i'm not seeing any logs in lldap that say error
there are debug but no error
n
If you restart the LLDAP logs (to start with clean logs) and you try to connect with bookstack (so that you get the error), do you see any request in LLDAP? Anything at all?
k
wait, i had refresh off. it did this:
Copy code
2023-06-02T02:22:50.676559535+00:00  DEBUG       โ”โ” ๐Ÿ› [debug]:  | response: SearchResultEntry(LdapSearchResultEntry { dn: "", attributes: [LdapPartialAttribute { atype: "objectClass", vals: [[116, 111, 112]] }, LdapPartialAttribute { atype: "vendorName", vals: [[76, 76, 68, 65, 80]] }, LdapPartialAttribute { atype: "vendorVersion", vals: [[108, 108, 100, 97, 112, 95, 48, 46, 52, 46, 51]] }, LdapPartialAttribute { atype: "supportedLDAPVersion", vals: [[51]] }, LdapPartialAttribute { atype: "supportedExtension", vals: [[49, 46, 51, 46, 54, 46, 49, 46, 52, 46, 49, 46, 52, 50, 48, 51, 46, 49, 46, 49, 49, 46, 49]] }, LdapPartialAttribute { atype: "supportedControl", vals: [] }, LdapPartialAttribute { atype: "supportedFeatures", vals: [[49, 46, 51, 46, 54, 46, 49, 46, 52, 46, 49, 46, 52, 50, 48, 51, 46, 49, 46, 53, 46, 49]] }, LdapPartialAttribute { atype: "defaultNamingContext", vals: [[100, 99, 61, 112, 111, 116, 97, 116, 111, 115, 97, 108, 97, 100, 44, 100, 99, 61, 112, 97, 114, 116, 121]] }, LdapPartialAttribute { atype: "namingContexts", vals: [[100, 99, 61, 112, 111, 116, 97, 116, 111, 115, 97, 108, 97, 100, 44, 100, 99, 61, 112, 97, 114, 116, 121]] }, LdapPartialAttribute { atype: "isGlobalCatalogReady", vals: [[102, 97, 108, 115, 101]] }] })
2023-06-02T02:26:43.247462922+00:00  WARN     ๐Ÿšง [warn]: Sending fatal alert BadRecordMac | log.target: "rustls::conn" | log.module_path: "rustls::conn" | log.file: "/__w/lldap/lldap/${GITHUB_WORKSPACE}/.cargo/registry/src/github.com-1ecc6299db9ec823/rustls-0.20.8/src/conn.rs" | log.line: 1332
2023-06-02T02:26:43.247592492+00:00  ERROR    ๐Ÿšจ [error]: [LDAPS] Service Error: cannot decrypt peer's message
n
I think we got our clues: "Sending fatal alert BadRecordMac " and "[LDAPS] Service Error: cannot decrypt peer's message"
you didn't enable starttls on bookstack, did you?
k
no
n
I'm curious what's the message sent by bookstack that LLDAP doesn't understand
We could try tcpdump to see the traffic
it should be encrypted (the whole point of LDAPS), but if it's not, that's a clear issue
k
is it possible that i may have missed something with the certs?
n
I would expect a TLS failure to establish the connection, then
bookstack shouldn't need to access the certs, only LLDAP
k
I got a cert for ldap.domain.tld using let's encrypt through nginx proxy manager, then i copied the cert.pem and key.pem to the machine running lldap and set the config
n
but the LLDAP certs should be valid for the hostname you're trying to reach it at. So if it's
ldaps://ldapserver:6360
then the cert should be for
ldapserver
(no .domain.tld)
but then again, I'd expect a different error in case of wrong cert
(even just a self-signed certificate would work if you passed LDAP_TLS_INSECURE=true to bookstack)
k
i'm not seeing bookstack even sending anything to lldap in tcpdump
n
Unless you anonymized the data in the logs, are you sure that bookstack can reach the LLDAP host as
ldapserver
? Are they running on the same host?
(are you using docker? If so, are the containers in the same network? If you open a shell in bookstack, can you
ping ldapserver
?)
k
i used ldapserver in place of the ip in my message
i was just using the server's IP, not the hostname
i just changed it from ip to ldap.domain.tld which is what the cert is for
then added a DNS record in my local dns to point to the ip
and it is pingable on that ip
n
From the bookstack host/container, can you telnet to the address/port?
well, actually yes, LLDAP sees something
but the message sent is not correct
hmmm
can you try installing ldapsearch somewhere (ideally the bookstack host/container) and checking whether you can bind to ldaps://ldapserver:6360 ?
k
to answer your earlier question, bookstack and lldap are both running in docker on the same host
and it does work if i disable ldaps in bookstack
ok, i've install ldap-utils and configured the file as such:
Copy code
BASE    dc=domain,dc=tld
URI     ldaps://ldap.domain.tld:6360
if I run ldapsearch it returns
Copy code
ldap_sasl_interactive_bind_s: Can't contact LDAP server (-1)
        additional info: (unknown error code)
n
"configured the file"? Can't you do it as CLI parameters to ldapsearch directly?
ldapsearch -x -H ldaps://fqdn -b "dc=example,dc=com"
k
Copy code
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
just to clarify, should i be using my actual domain.tld in "dc=example,dc=com" if that's what I set in LLDAP?
n
it actually doesn't matter here
since you can't even reach the server
k
also, for reference, if i use ldapsearch to go to plain ldap on for 3890 i get:
Copy code
ldap_bind: Naming violation (64)
        additional info: Missing DN value
n
yeah, that's fine, it's saying you didn't provide your bind username (which we didn't here)
we're hoping to get that error over LDAPS
Any error in LLDAP when connecting with
ldapsearch
over ldaps?
k
Copy code
2023-06-02T03:02:59.677333779+00:00  INFO     HTTP request [ 8.27ยตs | 100.00% ]
2023-06-02T03:02:59.677335539+00:00  INFO     โ”โ” ๏ฝ‰ [info]:  | uri: /health
2023-06-02T03:03:51.298231170+00:00  ERROR    ๐Ÿšจ [error]: [LDAPS] Service Error: while handling incoming messages: while receiving LDAP op: unexpected end of file
n
Alright, I'll really need to see the tcpdump on the LLDAP port 6360 to make sense of that
hopefully it's not just "handshake + encrypted data"
k
i'm sorry, what do i run to capture that with tcpdump?
n
in the LLDAP container, something like
tcpdump port 6360
k
wait how do i get tcpdump installed in the container? I was running it from the host sorry..
n
Hmm, from the host it's going to be more complex, since you'd need to monitor the traffic going to a specific container on that port
unless you expose the LLDAP 6360 port and then use
ldapsearch
with localhost
(exposing the port will redirect the traffic to the LLDAP container, but to see it in tcpdump you'll need to talk to the main host, because you want the redirection to happen since that's the point at which you capture the traffic)
k
so in the lldap docker compose, i have 6360:6360 under ports, is that correct?
n
I think so
try the ldapsearch command but targeting localhost
k
it says 0 packets captured
n
hmmm. Then the next step is to install tcpdump in the LLDAP container and run the capture there
and maybe capture all the traffic so we see what messages get to LLDAP
k
alrighty, my docker skills are fairly new, i can get into the container through portainer, but idk how to install packages
n
do you know if you have the debian or alpine image for LLDAP?
easiest think to try: can you do
apk add tcpdump
k
aha
been doing apt this whole time
n
๐Ÿ˜„ you have the alpine version, then (it's more lightweight)
k
nice, lemme just obfuscate my domain info real quick
n
Can you run tcpdump with
-w capture.pcap
(for writing the packets to a file) and send me the resulting capture file?
(also, just checking, while capturing you ran
ldapsearch ... ldaps://ldapserver:6360
?)
k
i ran ldaps://ldap.domain.tld:6360
before i send it, will the pcap have any sensitive information in it?
n
it'll have the domain name, but that's it
if you want, you can send it to me privately
k
sent
n
the bind password, if it gets sent at all (and I think you don't even give it in the command you run), will be encrypted with the certificate that I don't have ๐Ÿ™‚
yeah, it's kinda what I was afraid of, it's handshake + encrypted data, and the server closing the connection
k
hmm, are there any configs that i should triple check?
n
The rustls error (BadRecordMac) usually means that the packet was corrupted. I don't see how that could happen due to a config error
one thing to try: can you switch to the debian-based LLDAP?
(just add
-debian
to the image tag, e.g.
lldap/lldap:latest-debian
)
k
``ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)``
could it be an issue with the cert?
oh my gosh
i fixed it
i am so sorry for wasting your time
I mistakenly thought that cert1.pem was what I was supposed to use
but it was actually fullchain1.pem that nginx proxy manager also created that I was supposed to use
n
oh ๐Ÿ˜…
I guess we really need a little guide for how to set up certificates for LDAPS, then ๐Ÿ˜„ Wanna share your setup with nginx proxy manager for others to re-use?
k
sure, i will write something up tomorrow (getting late here..lol) and send it to you
n
sounds good. Thanks for getting to the bottom of this!
I really thought it wouldn't be a cert issue, though. But I guess it kinda makes sense if the certificate is invalid that decoding the message doesn't work?
k
hopefully when I get everything tested, i can just mount the cert folder from npm into lldap and then i don't have to worry about it expiring
it was you telling me it was corrupt and this closed github issue that led me to try it: https://github.com/lldap/lldap/issues/289
they have fullchain.pem in their config
i just used cert1.pem because it most closely matched the example config lol
i do have a question, can i completely disable plain LDAP on port 3890 then? just to be sure i don't accidentally use it down the line?
n
Well, you don't have to expose the port. It's already a non-standard LDAP port, but you can change the port to an even more obscure one if you want
but no, there's no mechanism to completely disable LDAP
I guess you can use iptables to block that port as well?
k
so i can just not bind it in the docker-compose file and that will suffice?
n
you don't have to bind any port in the docker-compose file if you don't plan on accessing them outside the docker network
if you want to expose the ports on your host, then you need to bind them
k
ah, i have several things running on other hosts that I would like to use ldap for
n
by default, any docker container can access any port of any other container on the same docker network
but the ports are not exposed to the internet
if you want to expose them (i.e. have the ports accessible from the main host) then you need the bind
so yeah, if you only bind the LDAPS port, then from outside you'll only be able to access LDAPS
k
cool
n
but from inside the docker network (so only localhost) you can also hit LDAP
(you can further restrict that by setting up separate networks in your docker-compose file so that the lldap container is not visible from the other containers)
so I guess you could set up a separate docker network just for the lldap container, and force all the other containers to go through the main host to access LLDAP, in which case they'll only have access to the LDAPS port that you bound
k
so based on the docker-compose example on github, the network is lldap_default and unless i add other containers to that, or lldap to other networks, it's only the host and lldap on that network?
n
it's only lldap on that network. The host can't access it, unless you bind some ports (in which case they'll be accessible from outside, so the other containers can take the outside road to access it)
k
ah, gotcha
ty
n
๐Ÿ‘
k
i really appriciate it