Puppet Community

if you end up with 25k facts you just have too big servers!

there's no concept of opt-in facts, sadly. only opt-out

or puppet 7 has a bug <https://tickets.puppetlabs.com/browse/PUP-11685>

I think we can confine the fact to an external fact... So it won't run at all unless the fact file is created.

Timing is bad for me today. I'm going to be out of the office for the middle of the day.

I'm not enthusiastic about another major version bump

I think a patchrelease would be fine as well

I don't think anybody relies on the fact already

I'm also a bit surprised that people have managed to update to version 4 already. They must be ignoring version constraints

Is it not in the $PATH of the shell puppet was run under?

/sbin/udevadm , the udevadm fact also works

we still use librarian-puppet in foreman-installer too

<https://github.com/voxpupuli/librarian-puppet> &amp; <https://github.com/voxpupuli/librarian> are more recent

don’t get me started on how much I loathe the foreman install documentation

it's not perfect, but it's better than manually tracking dependencies

meh, nothing’s perfect.  It’s just a sysadmin Monday.  Time to gripe.  :smile:

so I wonder who's going to FOSDEM and/or cfgmgmtcamp

<@U1QVBAUS2> is that correct? <https://github.com/voxpupuli/puppet-systemd/blob/master/lib/facter/udev.rb#L15>

```  Error: Facter: Error while resolving custom fact fact='udevadm', resolution='&lt;anonymous&gt;': break from proc-closure
  Error: Facter: Error while resolving custom fact fact='udevadm', resolution='&lt;anonymous&gt;': break from proc-closure```

Bah. One of the code comments is now out of date.

each node gets warning with systemd v4:
```  Warning: The current total number of facts: 8382 exceeds the number of facts limit: 2048```
It's due to a bug I posted earlier , <https://tickets.puppetlabs.com/browse/PUP-11685|PUP-11685>

That warning is there because facts eventually add up to levels that can overload the infrastructure.

One fact is cheap. 1000 facts multiplied by 10,000 nodes can be very expensive.

I agree, but here it's a bug, I counted actual facts count is way less

Hmm, looking at PUP-11685 it sounds like the bug there is an undercount.

The thing that starts getting expensive with arrays is tracking the "paths" through each fact. A three-element array would have three paths: `foo.0`, `foo.1`, `foo.2`.

Do you have a workaround, by any chance? I have started migrating to PE2021 and stepped on this. So far only docker running systems have been affected

I thought to open PE support ticket, but then found that one, <@U112RM52S>, found it, actually

Hmm, the fix to PUP-11685 would cause the count to go up, not down. What is the `systemd` module reporting that is adding thousends of entries to the factset? How many of those are used during catalog compilation?

imho, none. recent module addition adds parsing of  udevadm info

Yeah... I'm inclined to say the warning is spot-on for that case. Pushing large amounts of data through the system that isn't used in configuration decisions is what gets folks into trouble.

is there a way in puppet 7 to block offending fact?

Yeah, the Facter 4 blocklist is fully configurable.

in puppet 6 you can limit only predefined facts

documentation still indicates it's the case

<https://www.puppet.com/docs/puppet/7/configuring_facter.html#configuring_facter-facter-conf-facts>

```blocklist — Prevents all facts within the listed groups from being resolved when Facter runs. Use the --list-block-group command line option to list valid groups.```

Maybe using the ability to define new block groups:

<https://www.puppet.com/docs/puppet/7/configuring_facter.html#configuring_facter-facter-conf-fact-groups>

I assume facter.conf is not something I plugin-sync ?

so either during provisioning or two runs

<@U2S1MUUQ7> I will do something about the fact later today. Probably going to disable it by default.

it's not just this one. docker makes one, yum, bunch of modules nowadays push lots of stuff into facts

this fact is extra large.  We decided to give it a try and see if any thing blew up

<@U11MDJNGZ>, would be nice if 'offending' fact was printed in those warnings

I think the better question is how can we push a hash that is opaque to the $facts tree into the system

I see value in a lot of these ideas, but it seems like having a way to make those available as a _single_ fact instead of as traversable would give a solution both ways

it would also allow for passing a large amount of reporting data for PE's compliance and patching products

i wrote the first version of the fact for storage configuration but inventory would be a nice bonus

There isn't a good way to push "opaque" data through. When PE bumped into this problem with the package inventory, the solution was to special-case it in the facts terminus:

<https://github.com/puppetlabs/puppetdb/blob/main/puppet/lib/puppet/indirector/facts/puppetdb.rb#L30-L32>

For one customer that had doubled down on reporting something `O(scary)` in size and couldn't switch to a different approach, the solution ended up being to run the fact through `xz` and then base-64 encode it.

One large string was better than a hash with millions of paths.

But. In general, this sort of usage starts spending your performance budget. It's easy to paint yourself into a corner if you're not very careful with what gets reported as a fact.

I wonder if that's something that could be configurable in the puppet configuration

Honestly, this sort of data probably needs to go to a different system. Pushing it through `/puppet/v3/catalog` puts pressure on several weak points of the system that did not expect to deal with large volumes of non-configuration data riding along on the catalog request.

Apply enough pressure and the infrastructure components start imploding.

Which is the reason this warning was added. Folks overloaded their systems with fact data and said "why didn't the system warn us?"

it's good, just need to be able to easily find bad facts now :slightly_smiling_face:

and easily block them before they are collected

Just as a curiosity, years ago there was a mantra about not managing users with puppet. Barely tangential to that, and while I have zero love for NM, I find myself wondering the very basic question: why use a tool that (in its most common usage) requires networking as the very tool that sets up that networking? To me, it has always been a provisioning-time thing much like user management is an ldap thing. I'm not being critical, but I would like to understand the thinking behind it, please. Obviously anytime drop-in files can be used, I prefer it.

puppet ventured into provisioning for awhile, so it might be remnants of that

Actually, through multple twists and turns around systemd-networkd, I mistakenly thought this comment: <https://puppetcommunity.slack.com/archives/C0W1Y5VL0/p1675026607383829> was part of this thread

I hate that threads also ignore my notification settings

in slack, I mostly do to, but when you spend the vast majority of your time in a different slack community, it makes it easier to get notifications of those conversations which you actually were part of

some NM or networkd settings can be needed post factum though

yes, I can see that, also a global change can be required on already in-prod hosts

I better leave fast.... :stuck_out_tongue:

that will only modify running servers :slightly_smiling_face:

unfortunately, network configuration isn't only at provisioning time. For example, I have k8s running ceph (IOW - state I don't like to loose) which get vlans and statics routes updated.

this code seems to be working properly
```  if $facts_blocklist and versioncmp(fact('aio_agent_version'),'7') &gt;= 0 {
    file { '/etc/puppetlabs/facter':
      ensure =&gt; directory,
    }
    -&gt; hocon_setting { 'blocklist facts group':
      ensure  =&gt; present,
      path    =&gt; '/etc/puppetlabs/facter/facter.conf',
      setting =&gt; 'fact-groups.blocked-facts',
      value   =&gt; $facts_blocklist,
      type    =&gt; 'array',
    }
    -&gt; hocon_setting { 'blocklist facts':
      ensure  =&gt; present,
      path    =&gt; '/etc/puppetlabs/facter/facter.conf',
      setting =&gt; 'facts.blocklist',
      value   =&gt; ['blocked-facts'],
      type    =&gt; 'array',
    }
  }```

<@U2S1MUUQ7>, is there a particular reason for using `fact('aio_agent_version')` from stdlib instead of the builtin `$facts['aio_agent_version']` or is it just personal preference?

<@U2HGUG2CD> those two are completely equivalent, but `fact('my.personal.nested.fact')` is more like  `$facts.dig('my', 'personal', 'nested', 'fact')` which is like `$facts['my']['persona']['nested']['fact']` but with nil exception checking at each level

good to know if something ever blows up. :slightly_smiling_face:

also, it makes command line facter usage and Puppet code fact usage far more similar

I try to not pull in any deps until the moment I'm forced to

also personal preference :slightly_smiling_face:

yeah. tbh, I'd prefer if we shipped stdlib with the product to begin with, but that's a different conversation :grin:

We did ship stdlib bundled in PE 3.0. And the world burned because it conflicted with environment content added by users.

oh I remember. Imho, that's a separate conversation though.

I'm still advocating for an isolated enterprise environment instead of polluting user content.

The Puppet 4 loader might handle it in a saner way. But stdlib conflicts was a support case that was just on repeat week after week.

And while we're at it, we should rename the default environment name from `production` to `default` :stuck_out_tongue_winking_eye:

&gt; we all have testing environments. Some of us are lucky enough to also have a production environment.

Yup. "not shipping directly to production" is more of an exception than a rule :lolsob:

we have hundreds of environments here, and each team just points their hosts to their envs. "What is a PR?" "What is code review?" :shrug:

<@U2HGUG2CD> I have started putting in r10k branch filtering and github branch protection rules

we have multiple puppetservers and I'm planning to start taking away feature branches/envs in the most critical "enclaves" to force code review/merging into prod branches

we give each environment its own control repo and layer their environment on a base module pack that does infra.  So the Infrastructure/Application Support/DBA teams manage the basemodulepath and the 3 packs of modules there and the dev teams only ever get the main branch in non-dev/test envs

in dev and test we use r10k dynamic branches to give them access to testing / PR spaces

but also, puppetserver per tier, so dev nodes are on a dev puppet cluster (ca/1+ compile/puppetdb)

The issue we have started running into is people switching hosts onto a feature branch... which becomes long lived and is not rebased upon master

it is sort of a fine line as if we take away branches completely then things will probably stop being puppetized

that's why I don't offer feature branches to be deployed above test :smile:

the big thing is, it prevents splitting the tree

but we have physical hardware which is also under development in "prod"

we used to have a dev -&gt; stageus/stageuk -&gt; produs/produk

and said physical hardware isn't identical to what is in the test stands... it is largely simulated in the test envs

can you not attach the physicals to a pre-prod pm?

the stage us and uk/prod us and uk branches diverged badly by the end

I started having to cherry pick between them

in this case, a copy of the full up physical hardware would be about $200M

I don't mean duplicating the hardware, just changing which puppet cluster owns the configs for it

or is it literally a prod server that has dev going on on it because there's no way to duplicate it into dev

part of it is lack of discipline in workflow, some of it is that there is different physical hardware that the servers are connected too

and that physical hardware has firmware which is also under dev and changing

but most of the issue is cultural / inertia and not technical

sorting it out is going to have to wait until unrestricted sudo can be largely taken away from dozens of people

oh that's one of my favorite problems... (to execute on)