https://www.puppet.com/community logo
Join Slack
Powered by
# puppet
  • r
    Copy code
    exec { 'dear coworkers, this is fine. ssl reload of apache due to ssl change': }
    😁
  • j
    (and yes, i use the term "cow-orker" because they have been known to ork cows.) (it was a sick ostrich. allegedly.)
    homer 1
  • r
    degens from upcountry
  • j
    that's what i appreciates about puppets communities
    🍻 1
  • r
    somewhat surprised that you don't trigger a restart to get connections to reconnect due to the ssl change. hmmm, I guess if it's a long running connection it's fine because it's already been negotiated.
  • j
    some of the clients' sites have tens of thousands of client devices active (i.e. websocket connection open, but not necessarily sending messages) at the same time ... just doing a shift-change, where a few thousand people are logging in while a different few thousand are logging out over the course of 10-15 minutes, causes CPU/RAM issues ... not with nginx itself, but with the back-end service that nginx is proxy-ing the connections to. having every client device re-connecting all at once has killed a few under-spec'd servers in the field (we tell them what hardware specs to buy, but they always insist on running everything on a shared vmware esxi cluster running on under-powered and already-overloaded hardware ... so glad i'm on the "dev" side and don't have to deal with client IT departments)
  • b
    different approach: switch to haproxy which can restart without dropping connections (and reload certs if you want)
  • r
    Interesting. Might be more than you'd want to take on at the moment, but haproxy can do seamless reloads. I've used it for proxy only services in the past. It's really slick.
  • r
    jinx?
  • b
    ha!
  • b
    :D
  • b
    everybody needs a Bird+haproxy setup
  • b
    it sparks joy
  • r
    though in the case of websockets it might not really solve the problem... actually it can you just have to be tricky. You'd want to set the TTL on the old process to 10 hours. So the new haproxy will come up and start taking connections. The old process will stick around and handle the old connections. At the end of the TTL (or whatever haproxy calls it) the old process is terminated and the connections it was handling are reset. As long as the TTL is longer than a shift that probably works.
  • r
    We used haproxy to load balance PE and the long running orchestrator connections would always hit the ttl. We set it to 15m to avoid interrupting catalogs and took the hit on orchestrator connections.
  • a
    hey folks! Could you pls share how do you clean up puppetdb fact_paths table when it gets huge?
  • b
    how huge is it? puppetdb cleans it up automatically
  • b
    it can grow a lot / have a lof of churn when you have changing facts
  • b
    and huge in: many rows or huge disk usage? maybe postgres autovacuum is too slow
  • a
    sorry, my bad, it’s public.factsets
    Copy code
    public.factsets                                          | 702 GB
  • a
    sorry, my bad, it’s public.factsets
    Copy code
    public.factsets                                          | 702 GB
  • b
    how did you determine that
  • a
    Copy code
    puppetdb=# SELECT
    table_schema || '.' || table_name AS table_full_name,
    pg_size_pretty(pg_total_relation_size('"' || table_schema || '"."' || table_name || '"')) AS size
FROM information_schema.tables
ORDER BY
    pg_total_relation_size('"' || table_schema || '"."' || table_name || '"') DESC;
  • b
  • b
    ah yours look similar
  • b
    how many nodes do you have?
  • a
    1557
  • b
    ok not so much
  • b
    which postgresql/puppetdb version?
  • a
    puppetdb 6.22.1-1 psql 11
1...269270271...428Latest