This message was deleted.

Oh man, so many related issues merging in my head. @simonhoenscheid We definitely hit that same very-long-running-query but it was a symptom of another problem. I don’t think we we ever addressed anything to do with the tables or cleanup. Looking through some support tickets.. • One issue we had was a timeout on PuppetDB queries coming from compilers. I believe this may have left queries running on Postgres. If I remember (and am re-reading this ticket) correctly, it was a lack of G1GC on the Puppetservers; the PuppetDB service was timing out on input from Puppetserver during blocking GC. Our initial reading of the logs was that the timeout was caused by PuppetDB waiting to hear from Postgres, but that was incorrect, it was the GC on the Puppetserver side. In this case, look for HTTP timeouts in the compiler logs:

[c.p.h.c.i.PersistentSyncHttpClient] Error executing http request

• We definitely also hit what I’d call a bug where long-running PuppetDB queries on the console would time out the console request after 60 seconds, but the query kept running Postgres, meaning retried queries kept backing up more-or-less indefinitely until something crashed, or the user gave up.

Thank you @Reid Conti!