I came here to ask a related question to the discussion above. I have found a problem with the way we deploy changes to our puppet service. We're clearing the cache (or jruby pool), but there is a race condition which causes some rare problems. We keep our in-house puppet modules in and hiera data in one repo. When we update that repo, our puppet servers pick up the changes via a cron job which: 1. Does a git pull. 2. Compares the contents of the repo vs. the previous contents, then does an environment cache clear or jruby pool clear. I recently deployed a change which involved modifying puppet code to require a different data structure specified in hiera; in the same commit, I also modified the data structure in hiera accordingly. What I saw is that two clients (out of about 1000) used the new hiera data but the old puppet code; this only happened for one puppet run--subsequent runs from the same clients worked properly. This makes sense in retrospect--our updating is fast, but it's not atomic. Those two clients must have had their catalog compilation happening during (or right after) the git pull, but before the cache clear; the puppet code was cached, but the hiera data was not cached. To solve this, I see two possible approaches: 1. Before each puppet server does a code update, gracefully take that server out of service, in cooperation with the load balancer. 2. Update the puppet server atomically. I'd rather do an atomic update, if possible. From a filesystem perspective, I could atomically replace the repo, but that won't be enough, due to puppetserver's caching. I'm thinking that I could use the "environments" feature for this, so that each deployment goes into a new environment, hence will start out uncached. Right now, we're only using one

production

environment for everything. My idea is to modify our cron job to, for every update of the main repo: 1. Make a new

git worktree

(named by the commit hash) in

/etc/puppetlabs/code/environments/

2. run

puppet generate types

3. prune unused worktrees (older than the previous worktree) so they don't accumulate 4. trigger our ENC (via a config file) to start specifying the new environment Has anybody else run into this problem? Does my idea seem reasonable as a solution? Or is the current recommended method of deployment different and somehow immune to this? I'm another one of those "old school puppet2x" users and I'm not always aware of newer practices.