yes abandon node groups as good as possible and rely on Hiera.

It's also possible to manage node groups via Puppet, and you can export and import node groups

Yes, that's the gospel I hear everywhere. But we are kind of stuck on...how? Is there any good documentation on how to setup a modern "hiera-based" PE server? Can all the settings under PE Infrastructure, PE Patch Mgm be defined in hiera also? What would be the best module to handle those hiera settings? We are clueless right now. 🙂

in the easiest form, hiera is just a collection of yaml files

which settings did you configure under the PE Infrastructure group?

they can usually be copied 1:1 to Hiera

If you want a UI for Hiera / see which nodes get which dataset from where, you can use https://github.com/betadots/hdm

Under PE Master we define the agent to provide to nodes of various distros and versions.

if you provide the exact data, we can tell you how to put that into Hiera

I'm looking for some documentation on how to confiure Puppet itself with hiera, but in the entire documentation all I could find, with one example only was: https://help.puppet.com/pe/2025.6/topics/configure_settings_with_hiera.htm Is there nothing more on this? Feels more like "if you use hiera, you're on your own".

you throw your class parameters 1:1 into Hiera

So, as an example, how would i know the syntax to convert this into hiera settings e.g: puppet config set --section main runinterval 90m" puppet config set --section main splay true"

I doubt that's what you have in your node group

The recommended module to configure your agents within PE is puppetlabs/puppet_agent. If you use the module via node manager, you can copy the data directly into Hiera

you really should just do in-place upgrades, they are very simple and easy for single-server setups. you just run one command.

You could export and import your classifier data into the new primary, but I would also agree with just upgrading in-place https://portal.perforce.com/s/article/115004588928

We only upgraded LTS versions. You leave too much baggage in place with new postgres, java, ruby and so on. Also way easier to go back if something goes wrong

question ... code manager is doing something to avoid updating environments which are currently being used to build catalogs, and/or to tell the compiler not to use a given environment to build catalogs while code manager is rebuilding it ... are the details for how this works, available anywhere?

By default puppetserver caches the code. So you can edit the filesystem like you want, puppetserver will ignore it. Code manager basically flushes the cache

You can see that in the puppetserver.log

I think he's asking about JRuby locking? I don't know if/where it's documented, but basically it will lock all JRubies in the pool, update the code inside each, and release the locks. They cannot be borrowed to do any work while the lock is in place.

It will also wait until all borrowed, i.e. currently used, JRubies are returned to the pool before updating the code, which is a great way to find out if you have a stuck or super long-running request because the update will never finish.

i'll be honest, i'm not sure exactly what i'm asking because i have no idea how the components talk to each other ... what i know is, on the "old" PE2016 servers using a custom

r10k

-alike script, some agents will occasionally fail with an error message like "unable to find module stdlib", because they happen to request a catalog while their environment is in the middle of being rebuilt. for PE2023 the "real" PE servers (being used to configure normal machines) will be using code manager so i don't expect this to ever happen, but on the "dev" PE servers (only used to prototype and test new puppet code before it gets committed and pushed to a repo) the environments are still being built by hand, because having to commit, push, and then wait for code manager to rebuild the environment, takes long enough that i lose track of what i was doing ... instead i use a script to

rsync

my changes directly to the environment, and i can go from "save changes" to "run

puppet agent -t

on a scratch machine" in about five seconds.

i vaguely remember years ago, somebody told me that, whatever name they were using for the

r10k

wrapper now known as "code manager", had some kind of secret API that it used to (1) wait until the compiler wasn't using a given environment, (2) "lock" that environment so the compiler wouldn't use it, (3) rebuild the files in the environment, and (4) "unlock" it so the compiler could build catalogs again ... but they also said "i think that's an internal thing that puppetlabs (at the time) doesn't want to share the details of"

i'm hoping that things have changed and this stuff isn't really a big secret, it's just something nobody has ever asked about before ... although i'll be honest, i fully expect to be "perforced", which in this case means being told that it's something perforce doesn't want to tell anybody about.

(also it's quittin' time in florida, i'll check back tomorrow to see if anybody answered ... have a good evening all)

I’m typically fascinated by the types of questions you ask like you’re the Mystical Merlin or something. I’ve literally been in hundreds of environments, done hundreds of deployments, worked in small-ish environments with servers numbered in dozens as well as massive environments with servers numbered in hundreds of thousands, and not once have I run in to the types of scenarios you bring up here. Makes me think I missed something in my education, or I’m doing somnething wrong.

Code manager is a client <-> server service built into

pe-puppetserver

. The server side runs on the PE Primary and exposes a HTTP API that receives deployment requests which it handles by: • Authorizing the request using a PE RBAC token. • Running

r10k deploy environment

for each control repo branch listed in the request. This step includes logic to safely run

r10k deploy

concurrently against multiple branches along with logic to prune the r10k caches to prevent them from growing too large. • When

r10k

finishes running, post-run scripts are executed. By default, this includes running

puppet generate types

on the environment, if needed. • The result of

r10k deploy enviroment

+ post run scripts is committed to an internal Git repository known as "File Sync Storage". • Optionally, the API call may wait before returning a HTTP response until all clients ACK the deployment is live or a timeout has elapsed. The client side runs in

pe-puppetserver

and

pe-orchestration-services

and: • Polls the Primary for new deployments, every 5 seconds. This polling request also serves as the deployment ACK by notifying the Primary of the latest environment versions the client has deployed. • If there are new deployments, runs a

git fetch

operation to pull commits from File Sync Storage to local client copies of the repository. • Deploys updated environments brought in by

git fetch

. These updates are made atomic either by a very heavy JRuby read+write lock (legacy deployment) or by creating a new versioned copy of the environment and updating a symlink (lockless deployment, modern default). Lockless deployment gets a significant speed boost from modern GNU coreutils that default to

cp --reflink=auto

and a filesystem that supports reflinks (XFS, BTRFS, ZFS, notably NOT EXT4). Basically, for best performance run PE infrastructure on RHEL 9 or newer or Ubuntu 24.04 or newer (but don't use Ubuntu's filesystem default of

ext4

) . • Performs cleanup of superseded environment content and git history. Code Manager also serves a dual purpose when DR is enabled in that it syncs deployed code, CA state, and PE configuration from the Primary to the Replica.

Turning Code Manager on also activates "Static Catalogs". These contain pre-computed metadata for

file

resources that use content deployed through Code Manager. This provides two benefits: • An entire source of JRuby contention is eliminated as agents no longer have to make

file_metadata

requests to determine expected checksums, it's all just there in the catalog.

file_content

requests are also cheaper as they stay in the Java layer and hit the JGit service instead of going down to JRuby. • The agent gets file content from the same deployment that its catalog was compiled from. Not a different version that may have come down in a subsequent deployment. IIRC, the above combined into something like a 20% cut to the JRuby load in Puppet Lab's internal infrastructure when it was benchmarked years ago. Milage will vary, custom file mounts serving large blobs are still something to shift over to a dedicated file or artifact server.

i just saw these responses ... i read thought it all once but i'm not grokking it in fullness, probably because it's late in the day on a friday and my brain already has one foot (or neuron?) out the door ... i saved it into obsidian so i can refer back to it later, even if "later" means after slack blocks access to it in 90 days ... @csharpsteen thank you for this, and don't be surprised if i come back with more questions, either next week or in january (i'll be out from 12-13 to 01-04,

$DAYJOB

has a "use it or lose it" policy for PTO)