https://github.com/lldap/lldap logo
heret1c404 - Hello, I noticed a weird behaviour...
# troubleshooting
h
Hello, I noticed a weird behaviour on one of my LLDAP docker deployments. After a few days of uptime, LLDAP became unresponsive, front is not loading and LLDAP does not respond to requests. After restart, it works again for a few days and becomes unresponsive. Has anyone else had the same issue?
a
message has been deleted
h
I run Docker in the Proxmox LXC container, and use basic compose:
Copy code
services:
  lldap:
    image: lldap/lldap:stable
    restart: unless-stopped
    depends_on:
      - postgres
    ports:
      - "3890:3890"
      - "6360:6360"
      - "17170:17170"
    volumes:
       - "./lldap_data:/data"
    environment:
       - UID=${UID}
       - GID=${GID}
       - TZ=${TZ}
       - LLDAP_JWT_SECRET=${LLDAP_JWT_SECRET}
       - LLDAP_KEY_SEED=${LLDAP_KEY_SEED}
       - LLDAP_LDAP_BASE_DN=${LLDAP_LDAP_BASE_DN}
       - LLDAP_LDAPS_OPTIONS__CERT_FILE=/containers/lldap/certs/fullchain.pem
       - LLDAP_LDAPS_OPTIONS__KEY_FILE=/containers/lldap/certs/privkey.pem
       - LLDAP_DATABASE_URL=postgres://${POSTGRESQL_USER}:${POSTGRESQL_PASS}@postgres/${POSTGRESQL_DB}
  postgres:
    image: postgres:14.1-alpine
    restart: unless-stopped
    environment:
      POSTGRES_DB: ${POSTGRESQL_DB}
      POSTGRES_USER: ${POSTGRESQL_USER}
      POSTGRES_PASSWORD: ${POSTGRESQL_PASS}
    volumes:
      - ./postgres-data:/var/lib/postgresql/data
n
What's your LLDAP version? Can you pull the latest stable, just in case? It sounds somewhat similar to an earlier case where our logging buffered for the entire LDAP session, and there were some long-lived sessions, leading to an out of memory situation
h
I have 0.6.1
n
Huh, that's not the same issue, then. Could you enable verbose logging, let it run until it doesn't respond, and send me the logs? It might be fairly big, depending on your traffic
h
Just enabled verbose logging, it usually takes a day or two before it becomes unresponsive
n
And check the CPU and RAM usage before restarting
What else can I check before restarting?
n
Does it respond to ldapsearch?
Does it respond from inside the container? (Hitting localhost)
h
It does not respond to ldapsearch from same host, how to check it from inside the docker container?
n
Some version of docker exec
I'm on my phone, I can't give you a precise command
Ah, and check if it responds to an http request too, just in case
h
I meant that ldapsearch wasn't installed in the image, after I installed it manually - no response to ldapsearch from within the container
n
For instance, curl
h
no response
n
Alright, I'll have a look at the logs a bit later
But I'm not optimistic, I'm guessing it's probably due to your system
h
What do you mean?
n
Some unusual setup
h
I didn't modify much from the example compose file. I have 2 identical deployments, both on the same version but only this one is acting out
n
Interesting, the last thing I see from the logs is a regular LDAP request, as part of a session. But no end of session, so it's probably something to do with LDAP and that client. It's also interesting that you have 2 deployments and only one is acting up. What is the difference between the 2? Are there services connected to one but not the other?
h
They both have services, one is my home deployment (home one is problematic) and the other is office, both are in LXC on Proxmox, both have a similar set of services connected
only major diference is compose file, office one has 3 additional services in the compose, home one has only db
n
Hmm, the next step would be to grab a tcpdump of the 3 ports that LLDAP is listening to, looking for any diff with the logs (maybe see a killer query that doesn't make it to the logs)
I must warn you that this requires a deeper investigation than I can do on my phone, and I'm on holidays and unlikely to get on the computer anytime soon
h
No problem, it is my home deployment and I can just keep restarting it
I think I may have found the issue. I suspect it is the Loki logging driver; it is the only difference between instances, and it had an obsolete configuration. Removed it and retesting now
I believe it was a Loki logging driver; it lost connection to the server and froze the entire Docker container after it accumulated a certain amount of logs and was not able to send them.