it's ENC

I'm not expecting somebody to know the solution, but hoping for advice on places to check.

But the ENC says "development" if I run it standalone.

So the question then becomes, what does the puppet run actually do?

Do you have more then one environment defined in ENC

puppet sends a request for a catalog and provides a cert, a requested environment and a fact set

on which puppet version are you?

This is on puppet 5

The puppetserver puppet.conf "master" section contains external_nodes = /etc/puppetlabs/puppet/node.rb

That's the ENC as I understand it.

ugh that version is EoL since ages

Yep. And there's been a change that pretty well eliminates that by skipping the operations that lead into the first switch.

nods

Yup, 6.25.0: https://puppet.atlassian.net/browse/PUP-10539

Yeah, I have a new puppet 7 environment up and running, and I'm in the process of making it possible to switch everything over. So I won't have puppet 5 questions forever. 🙂 But it would make things a lot easier to upgrade if I could assign environments in the old puppet setup.

(So I can test code changes without deploying them everywhere at once.)

I don't know why would it switch it each run. From what I know you assign host a puppet environment

and it doesn't change

The core of the problem is that first switch is happening after a

GET

request for

/puppet/v3/node

That request carries no facts as it executes before pluginsync which is before Facter is called up to load facts. So, if your ENC is dependent on facts to put a node in the "right" environment and doesn't have access to a cached source of facts like PuppetDB, then it may give the "wrong" answer to that first request leading to a flip-flop.

I doubt the foreman folks would know the answer to this either, because it's very specific to this setup. I wouldn't be surprised if somebody who used to work here made a hack somewhere that always switches the environment back to "test".

What I want to do is to better understand how Puppet interfaces with the ENC so I can find out if I can see where that switch is happening.

The fix in Puppet 6.25.0 was to skip that request to

/puppet/v3/node

if there was a cached catalog present locally on the agent on the assumption that the

environment:

in that catalog will be a much more accurate starting point than whatever an ENC might generate without facts.

So, in general ALL the hosts connected to this puppet 5 setup are in environment "test" all the time, always.

I set up a new "development" environment so I can actually test changes.

But from a puppet perspective—it has to get ENC integration info from somewhere. And the only place I know of is that "external_nodes" line. But that script, run with the name of the host I'm trying to reassign to development, correctly shows "development" as the environment—UNTIL I run puppet again on that host.

Notice: Local environment: 'production' doesn't match server specified node environment 'development', switching agent to 'development'.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Notice: Local environment: 'development' doesn't match server specified environment 'test', restarting agent run with environment 'test'

In this sequence, the first "server specified node environment" of "development" is the correct one. The second one is wrong. Somewhere in between the first and the second, because of the steps in between, the ENC changes from saying "development" to saying "test."

The only thing passed to an

external_nodes

script is the certname of the node: https://www.puppet.com/docs/puppet/5.5/configuration#external_nodes So, the answer to whether or not the script pulls in additional sources of info when making a decision lies within the script its self.

The point is that whatever info it pulls in, must change between the first call to the ENC script and the second, which is in puppet's purview.

it probably always pulls ins facts, but the don't exist on the first run?

Sure, but the quickest way to identify what those items may be is still to examine the script.