Puppet Community

hi, i just updated a 0.1.1 update to <https://forge.puppet.com/modules/kn/doas> where i can manually download this version, but `mod 'kn-doas', '0.1.1'` in Puppetfile followed by `librarian-puppet update` keeps failing with
```[Librarian]   Failed to resolve kn-doas (= 0.1.1) &lt;<https://forgeapi.puppetlabs.com>&gt; (from Puppetfile)```

does it take time for the api to catch up or is there something wrong i'm not seeing? the forge website shows 0.1.1 and the downloaded tarball has 0.1.1 in its metadata.json, so I don't understand why librarian fails to fetch/resolve it

Every time someone says they're using Librarian, I'm shocked.

I've been using Librarian's _replacement_ for getting close to 10 years now

hey librarian-puppet is still maintained :stuck_out_tongue:

`puppet module install kn-doas` fetches 0.1.1 as expected, so librarian seems off.

See the `kn-doas` module at <https://forge.puppet.com/kn/doas?src=slack&amp;channel=puppet>

guess i'll try r10k as a librarian replacement

pretty sure the path should be manifests plural, not manifest.

I was hoping that was it, but I changed it to plural and still same error.
```local_modules/
├── profile
│   ├── files
│   ├── lib
│   └── manifests
│       └── manifest
│           └── nginx
└── role
    └── manifest
        └── nginx.pp```

OK, figured it  out.  It was the extra `manifest` dir I had under the `pofile/manifest`  Plus I fixed the `role/manifest` to `role/manifests` too.

Side comment: pls always paste stuff as text, including execution outputs. Better visualization and ability to copy/paste are some advantages.

Another side comment. (or two):
Are you leveraging `pdk` for your code generation?  That will help you with making sure all of your directories and files are in the right place (particularly with `pdk new class` .  Also, `pdk validate` is amazingly useful to make sure you're syntax is correct before you try to deploy.

legacy facts were removed as part of Puppet 8 ... you'll need to replace those with $facts['networking']['fqdn'] I believe. There are way to reenable legacy facts, but I'd suggest the refactor to structured facts.

Ah, how do I send puppet a note that the puppet site still says @fqdn will work in 8

shoot me the link and I'll forward to the correct folks to fix that

<https://www.puppet.com/docs/puppet/8/lang_template_erb.html>

I've notified our docs team for this one.

Just for reference, you can see all the legacy facts here: <https://github.com/puppetlabs/puppet-lint/blob/main/lib/puppet-lint/plugins/legacy_facts/legacy_facts.rb>

And the process to re-enable them is listed in the release notes here: <https://www.puppet.com/docs/puppet/8/release_notes_puppet.html> (you could also use the puppet_agent module to manage that setting across your estate).

@fqdn is only the legacy fact fqdn I think

<https://www.puppet.com/docs/puppet/8/release_notes_puppet.html#enhancements_puppet_x-8-0-0-PUP-11430>

A brk() is apparently used to change the size of the heap.  Can I do something to start with a bigger heap?

any chance you have a few file resources with purge =&gt; true and/or recurse =&gt; true?

Few? Many?  I don't think so. Files are mostly created.

was it ever fast? did you change anything recently?

can you run puppet apply with --debug --evaltrace

Should I send the output somewhere? What am I looking for?

open a second terminal and look for memory usage. maybe at some point the debug output stops. the last few lines might be interesting. in parallel check if the allocated memory of the apply process starts to increase when debug output stops

I see a lot of
``` Adding autorequire relationship with File```
then
```Skipping automatic relationship with File```

(apache module) - it's creating vhosts .conf files

I can see that the Puppet agent ram use went up after the run, yes.   From 7xx MB to 9xx

"after"? did it finish? or is it still running and just the output stopped?

```Notice: Applied catalog in 336.01 seconds```

you can add --summarize to get information about theamount of time spend per resource type

Ok, will be bak in 330 seconds :slightly_smiling_face:

that will write the report to /opt/puppetlabs/puppet/cache/state/

that will create a huge yaml file with all resources and the time spend on each resource

```Notice: Applied catalog in 303.32 seconds
Application:
   Initial environment: cleanupdevusers
   Converged environment: cleanupdevusers
         Run mode: user
Changes:
Events:
Resources:
            Total: 8538
Time:
       Filebucket: 0.00
        Mailalias: 0.00
          Yumrepo: 0.00
     Selinux port: 0.00
         Schedule: 0.00
           Anchor: 0.00
        File line: 0.00
        Selmodule: 0.00
   Ssh authorized key: 0.00
            Group: 0.00
   Selinux fcontext: 0.01
       Selboolean: 0.05
      Ini setting: 0.05
             User: 0.07
             Exec: 0.09
      Concat file: 0.11
          Vcsrepo: 0.33
   Concat fragment: 0.42
     Alternatives: 0.45
           Augeas: 0.54
          Service: 1.45
         Last run: 1686347840
   Firewalld service: 3.60
   Transaction evaluation: 300.94
   Catalog application: 303.32
          Package: 5.83
   Config retrieval: 56.98
             File: 8.31
            Total: 304.25
Version:
           Puppet: 7.24.0
           Config: atqrh8phpd1-cleanupdevusers-1bd705738ae```

is there maybe some legacy code that can be removed?

Not really, just creating virtual hosts for the dev environments.  One per dev, per app.

How can I tell which modules, classes or resources are the most time consuming?

I don't see how much time it has spent on it, but there are timestamps

and I dont find proper docs for those metrics

I need to head out for today. you can take a look at the yaml files in the state dir, maybe they provide some more insights

<@U11567REE> from the --summary output, is there a documentation for the different timings, for example transaction evaluation? I only found <https://tickets.puppetlabs.com/plugins/servlet/mobile#issue/PUP-10531|https://tickets.puppetlabs.com/plugins/servlet/mobile#issue/PUP-10531>

No worries, thanks a lot and have a great week-end!

Adding `time` in front of my `puppet` command shows more than 7 minutes.

It looks like you’re copying a lot of files (vhost contents maybe?).. if so I’d suggest to deploy the contents with git or a tarball maybe.. that will reduce amount of resources in puppet

That would defeat the purpose of auto-generated vhost conf files.

there might be some room for improvement, but that time isnt bad for 8k resources. so unless it has to be faster, I would leave it as is

So what’s managed is just vhost configs, right? No www root dirs?

I mean that no www contents is managed with puppet

It's ok, you can stop putting efforts on this. I jus wanted to know it it was "normal". Thanks a lot everyone

In my main init.pp manifest I have:
```
    file { '/opt/dspace/assetstore':
      ensure =&gt; 'directory',
      owner  =&gt; 'tomcat',
      group  =&gt; 'tomcat',
    }```
And in the hiera for an NFS mount I have:
```  storage:
    '/opt/dspace/assetstore':
      ensure: 'mounted'
      device: 'tur-nas1:/vol/unix_storage1_t1a/qds-research/assetstore'
      fstype: 'nfs'
      options: 'nfsvers=3'
      atboot: true
      require: File['/opt/dspace/assetstore']```
I’m calling create_resources(‘mount’, $storage) but end up with:
```Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find resource 'File['/opt/dspace/assetstore']' in parameter 'require' on node tur-ds-dev4```

If I remove the require in Hiera then things proceed, but I end up with an ordering issue (i.e. Directory might not exist prior to the mount being attempted)

the `require`<https://www.puppet.com/docs/puppet/7/metaparameter.html#require| metaparameter> wants a resource reference, but in Hiera you are rather limited and `File['/opt/dspace/assetstore']` is indeed the String `"File['/opt/dspace/assetstore']"`. Luckily some mogic seems to happen and save the day, but I would avoid this kind of patterns if I can.

I avoid create_resources altogether.  It's like DNS.....  when fixing an issue, it's always create_resources.

it's better to do a .each loop over storage and then actually put a mount resource in with a `* =&gt; $options` type of thing

<@U1PC2RAH0> or <@U11HA7VJ7> do you know who can do this, or maybe you can do this? or how is the release cycle?

that's something for the CAT team, for example <@U0332SZQLKZ>

See the `puppetlabs-mysql` module at <https://forge.puppet.com/puppetlabs/mysql?src=slack&amp;channel=puppet>

This currently says:
```Error: Could not run: Could not retrieve dependency 'Exec["echo hi"]' of File[/home/animesh/TrashThings/testing/puppet/file-link]```
which is obvious as I've not defined that anywhere...

as you said,  you don't have it defined anywhere :slightly_smiling_face:

which you cannot do on the CLI in that way

it manages a single resource. if you want to play with a bit of Puppet DSL, for example two resources or some conditions, you need to use `puppet apply`

Um, ok, but it might be actually useful to run command on changes that way actually :thinking_face: even if its currently not implemented like so...

that's really what puppet apply is made for

I mean its what puppet is for, but this cli interface can be useful for imperative management while using puppet provider ecosystem which support many os.

yes. and you can do that with puppet apply as well

Anyway, I would like to try adding a --on-change "cmd" in fork if I'm able to, would be great to understand puppet from inner side as well...

puppet manifest can notify variety of resources, though if using from cli, notifying a command is enough i guess.

the whole concept of `puppet resource` is to manage a single resource

Hi. Is it possible to set the default/fallback environment for puppetserver if an agent connects to the Puppet master with a non-existing environment? By default it is of course 'production' but is it possible to configure it somehow?

```#!/opt/puppetlabs/puppet/bin/ruby
#
# This is a YAML-based node classifier which can be used as a Puppet
# External Node Classifier (ENC). It identifies the environment for
# a Puppet client, based on rules listed in a user-provided
# configuration file, '/etc/puppetlabs/puppet/environments.yaml'.
#
# INPUTS/OUTPUTS
# This script should be called with a Puppet client FQDN as a
# single command line argument.
#
# It will return a YAML-formatted string with a single key,
# 'environment'.
#
# For example,
#
#   cmdline$  /usr/local/bin/set_environment foo1.bar
#   ---
#   environment: production
#
#
# CONFIGURATION
# Each rule in '/etc/puppetlabs/puppet/environments.yaml' should have
# the following format:
#
#      &lt;regex/hostname&gt; : &lt;environment&gt;
#
# where,
#
# &lt;regex/hostname&gt;: May be either a regular expression or a simple
#                   string that matches against the FQDN of the system.
#
#      NOTE: The rule *must* have '/' as the first and last character
#            if it is a regex.
#
# &lt;environment&gt;: A string representation of the environment. This is
#                what will be passed back to the Puppet Server
#
# Rules are processed from the top down and the first match wins!
# If no match is found or 'environment.yaml' does not exist, this
# script will return the default environment: 'production'.
#
# EXAMPLE
#
# ---
# 'foo1.bar' : 'production' =&gt; Matches only 'foo1.bar' and places it in the
#                              'production' environment.
#
# '/^foo\.bar/' : 'foobar'  =&gt; Matches all hosts that start with foo.bar and
#                              places them in the 'foobar' environment.
#
# USAGE
#
# 1. Ensure this script can be executed by the Puppet server.
#
# 2. Configure Puppet to use this script as an ENC
#    a. Add, or change, the line 'node_terminus' to 'exec' in the
#       [master] section of puppet.conf.
#    b. Set the 'external_nodes' entry in the [master] section of
#       puppet.conf to point to this script.
#
#    The resulting lines should look something like this:
#
#      ...
#      [master]
#      ...
#      node_terminus = exec
#      external_nodes = /usr/local/bin/set_environment
#      ...
#
# 3. Add the file '/etc/puppetlabs/puppet/environments.yaml' to
#    your system.
#
#    NOTE:  You can verify that the file is properly formatted
#    and yields the classification desired by running this script
#    manually.  For example,
#
#    /usr/local/bin/set_environment foo1.bar
#
# 4. Restart the Puppet Server process
#
#
require 'yaml'
module Simp
  class YamlNodeClassifier
    DEFAULT_CONFIG_FILE = '/etc/puppetlabs/puppet/environments.yaml'
    DEFAULT_ENVIRONMENT = 'pending'

    def run(args, config_file = DEFAULT_CONFIG_FILE, default_env = DEFAULT_ENVIRONMENT)
      fqdn = args[0]

      raise("You must pass the FQDN of the host as the first argument") unless fqdn

      # Set result to the default environment if there is no match
      result = { 'environment' =&gt; default_env }

      if File.readable?(config_file)
        environment_data = YAML.load(File.read(config_file)).to_hash

        environment_data.each_pair do |host,value|
          if host[0].chr == '/'
            if host[-1].chr == '/'

              clean_host = host[1..-2]

              if Regexp.new(clean_host).match(fqdn)
                result['environment'] = value.to_s.strip
                break
              end
            else
              fail("Rule '#{host}' is not a valid hostname or regex")
            end
          else
            if host == fqdn
              result['environment'] = value.to_s.strip
              break
            end
          end
        end
      end

      puts result.to_yaml
      return 0
    end
  end
end

if __FILE__ == $0
  exit Simp::YamlNodeClassifier.new.run(ARGV)```

You can see that my default environment is PENDING. And i'm passing everything through the environments.yaml, which looks like...
```# Production environment
'/^obs\d{4}/': 'production'
'/^klr\d{4}/': 'production'
'/^tre\d{4}.+/': 'production'
# Dev environment
'/^ib\-\d{4}\-\d{2}/': 'dev'
# Master and Compiler to puppetmaster environment
'/^master.*/' : 'puppetmaster'
'/^compiler.*/' : 'puppetmaster```

by doing this I don't need to set any environments in puppet.conf. in fact even if you set env in puppet.conf they're overruled by the ENC.

It also really, really helps for testing because you can just put the name of the node you want to test on and direct it somewhere

Cool. Thanks for the examples. Will have a better look when I'm working again tomorrow. Thanks again!

<@U035NAH9QUU> You don’t mention anything in your script about the Simp components.  Also, the script doesn’t run.  Ruby is still looking for another “end” somewhere, and I’m no Ruby guy to find it.

here ya go:  put an “end” at the end of your paste-in.  Also, do:  /opt/puppetlabs/puppet/bin/gem install simp

not necessarily, you can just supply environment

If fact count keeps increasing every run, you've likely got a bug in one of your custom facts. I'd run `facter -p` and diff the output from two different runs.

first WAG is that someone wrote something that has a random element in the name of the fact

or, a richness. one should aspire to such a fact-filled life.