https://www.puppet.com/community logo
Join Slack
Powered by
I have a Puppet PE plan question. I have a plan w...
# puppet-enterprise
m
I have a Puppet PE plan question. I have a plan which only does this:
Copy code
apply_prep($targets)

  $kickstart_files = [
    '/root/anaconda-ks.cfg',
    '/root/dnf-install.log',
    '/root/install-callhome.post.log',
    '/root/install.post.log',
    '/root/install.postnochroot.log',
    '/root/original-ks.cfg',
  ]

  $result = apply($targets, '_catch_errors' => true) {
    file { $kickstart_files:
      ensure => absent,
    }
  }
my targets are fetched via a pql query in the plan. I want to run this plan on all our nodes attached to this PE instance. It works for PQL queries which return < 50 nodes. But as soon as I try to run this plan on all our nodes > 400 I just get a generic error saying
Copy code
{
  "msg": "Aborted due to orchestrator shutdown",
  "kind": "puppetlabs.orchestrator/job-aborted",
  "details": {}
}
I was under the impression that the puppet apply runs are batched according to the normal max limit of nodes, or is this really at the same time and I need to manually create batch groups of nodes and then iterate over them?
b
sounds like someone restarted pe-orchestrator-services. Did you check the logs on the primary?
m
Aha! Thank's for the hint. I'll go and peek in the logs. Just did a re-run and did 
tail -f /var/log/puppetlabs/orchestration-services/*.log | tee plan.log
AHA!!
Copy code
orchestration-services[2513182]: Aborting due to java.lang.OutOfMemoryError: Java heap space
orchestration-services[2513182]: #
orchestration-services[2513182]: # A fatal error has been detected by the Java Runtime Environment:
orchestration-services[2513182]: #
orchestration-services[2513182]: #  Internal Error (debug.cpp:362), pid=2513182, tid=2517982
orchestration-services[2513182]: #  fatal error: OutOfMemory encountered: Java heap space
orchestration-services[2513182]: #
orchestration-services[2513182]: # JRE version: OpenJDK Runtime Environment (17.0.18+8) (build 17.0.18-internal+8-adhoc..jdk17u-jdk-17.0.18-8)
orchestration-services[2513182]: # Java VM: OpenJDK 64-Bit Server VM (17.0.18-internal+8-adhoc..jdk17u-jdk-17.0.18-8, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
orchestration-services[2513182]: # Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %d %F" (or dumping to //core.2513182)
orchestration-services[2513182]: #
orchestration-services[2513182]: # An error report file with more information is saved as:
orchestration-services[2513182]: # /var/log/puppetlabs/orchestration-services/orchestration-services_err_pid2513182.log
Probably too many connections
b
just give it a gig more HEAP
m
Aye will try that now
Ugh why can't I do
Copy code
puppet_enterprise::profile::orchestrator::java_args: '{"Xmx": "2G", "Xms": "2G"}'
in hiera?!
Error while evaluating a Resource Statement, Evaluation Error: A substring operation does not accept a String as a character index. Expected an Integer (file: /opt/puppetlabs/puppet/modules/puppet_enterprise/manifests/profile/orchestrator.pp, line: 122, column: 131)
b
the datatype is Hash, you provided a string
m
So
Copy code
puppet_enterprise::profile::orchestrator::java_args:
  Xmx: '2G'
  Xms: '2G'
nah that doesn't look right
b
why not?
m
Ok yes that worked now. I was just thinking about the yaml syntax and whether I needed something like xmx: value: 'asdasd' you know? Some modules have this syntax
Yup that looks like expected 😄
meh coredumped again
b
that utilization looks a bit high
m
Honestly, it's easier for me to iterate over each patch group and do them in batches than all at once. Don't have the time or patience to troubleshoot this further. But I did learn a new thing today, so thank you!
b
There's a chance that your pe-puppetserver also has issues. The HEAP with just 2G could be too low. I suggest to check those logs as well.
and setup the puppet_operational_dashboards
m
Since this is the first time we've seen these issues I'm hesitant for more changes. Yet I agree with you to troubleshoot this further. I'll put it on the backlog for the summer break or someone else to pick that up earlier.
m
If you have enterprise support id recommend raising a ticket, they are pretty good and finding the root cause of memory and performance issues
m
I do. And I will try that and hopefully won't get turned away 😄 I'll report back in this thread what the outcome is
m
ill give them the heads up
c
isn’t there like a 
puppet infra tune
or some such? (I haven’t worked with PE in quite some time)
m
there is, it does provide some good base advice but sometimes particularly with orchestrator its a good first step but there may be something that needs tweaked beyond adding more resource
c
gotcha.
I had just found this: https://portal.perforce.com/s/article/360013148854 when you answered back. 🙂
m
i can also recommend the search at https://help.puppet.com/ it is trained on the docs and the KB and is pretty good at surfacing stuff like that
👍🏻 1
m
I mean I can start by trying the output the tune command just spat out and see if that helps
Copy code
puppet_enterprise::profile::database::shared_buffers: 6760MB
puppet_enterprise::puppetdb::command_processing_threads: 3
puppet_enterprise::profile::puppetdb::java_args:
  Xms: 3380m
  Xmx: 3380m
puppet_enterprise::profile::orchestrator::jruby_max_active_instances: 3
puppet_enterprise::profile::orchestrator::java_args:
  Xms: 3380m
  Xmx: 3380m
puppet_enterprise::profile::console::java_args:
  Xms: 1024m
  Xmx: 1024m
puppet_enterprise::master::host_action_collector::java_args:
  Xms: 512m
  Xmx: 512m
puppet_enterprise::master::puppetserver::jruby_max_active_instances: 8
puppet_enterprise::profile::master::java_args:
  Xms: 8192m
  Xmx: 8192m
puppet_enterprise::master::puppetserver::reserved_code_cache: 768m
puppet_infrastructure_tune::metadata:
  pe_version: 2025.10.0
  tune_date: '2026-05-11'
looks promising
same search is also here https://portal.perforce.com/s/product/a3g4X000009wMFBQA2/puppet
That link doesn't lead anywhere to me even though I am logged in
m
does seem to increase you considerably over current settings, could still be worth a proper root cause,
hmm try https://portal.perforce.com/s/ and just click the puppet tile
✅ 1
m
Yeah no that didn't help. I'll file a ticket now
01585231
s
What PQL query are you using? One issue is that you could be pulling full datasets for 400+ nodes and trying to consume that much data is blowing the Ochestrator heap. PQL queries can be scoped to return just the 
certname
that the Orchestrator needs to target activity. E.g: 
nodes[certname] {...}
m
Yes, we've already identified this as one of the root causes 🙂
👍 1
and by that I mean it's not the PQL query
Support was really helpful! The tune command was not, but support was able to tweak a couple values and we ended up with this:
Copy code
# Puppet infrastructure tune
puppet_enterprise::profile::database::shared_buffers: 4096MB
puppet_enterprise::puppetdb::command_processing_threads: 3
puppet_enterprise::profile::puppetdb::java_args:
  Xms: 3380m
  Xmx: 3380m
puppet_enterprise::profile::orchestrator::jruby_max_active_instances: 3
puppet_enterprise::profile::orchestrator::java_args:
  'Xms': 6144m
  'Xmx': 6144m
puppet_enterprise::profile::console::java_args:
  Xms: 1024m
  Xmx: 1024m
puppet_enterprise::master::host_action_collector::java_args:
  Xms: 512m
  Xmx: 512m
puppet_enterprise::master::puppetserver::jruby_max_active_instances: 4
puppet_enterprise::profile::master::java_args:
  'Xms': 6144m
  'Xmx': 6144m
puppet_enterprise::master::puppetserver::reserved_code_cache: 768m
puppet_infrastructure_tune::metadata:
  pe_version: 2025.10.0
  tune_date: '2026-05-11'
ran the plan a couple times and it worked. Obviously it takes a looooong time if I run it on all nodes rather than a subset with a different query. For example if I run it on half the nodes it's sub 5min and on all it took 23min 😅 Thank's all for chiming in and trying to troubleshoot this with me 🍰
catjam 1
Open in Slack
PreviousNext
Powered by