Puppet Community

I'm looking to puppetize SAMBA but unfortunately the more popular <https://forge.puppet.com/tags/samba|projects> are 4-6 years old.  I would be most appreciative if you'd share what samba module would be recommended.

<https://forge.puppet.com/modules/rehan/samba|https://forge.puppet.com/modules/rehan/samba> code looks okay, as far as I can tell that within a minute

Ahhh....thanks.  I didn't look at the projects with less than top-5 popularity.

Can you construct a variable name dynamically? Say I had two variables in my class $my_class::a and $my_class::b. I want to iterate over the values ['a','b'] and lookup $my_class::$item where $item is 'a' or 'b'.

Yeah, using the `getvar()` function makes that easy: <https://puppet.com/docs/puppet/7/function.html#getvar>

```$example1 = getvar("my_class::${a}")
$example2 = getvar("my_class::${b}")```

(use string concatenation to build the actual variable’s name, then put that in `getvar()`)

That’s not the current tuning recommendation. Especially not with Puppetserver 6+.

<https://puppet.com/docs/puppet/7/server/tuning_guide.html#number-of-jrubies>

On the topic of tuning Puppetserver, I highly recommend giving this entire ticket a read, but the comment I highlighted in the link explains why 1:1 jruby:core isn’t recommended anymore:
<https://tickets.puppetlabs.com/browse/SERVER-2771?focusedCommentId=754291&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-754291>

I think I have my mx set to 24g. I’ve got 28 cores and if I want to leave 8 of them free of puppetserver, it sounds like (based on that comment), I should probably set it to about 10 instances.

I was running into some EOF issue today and came across a convo from a few years ago about setting max-queue-length, which I just did.

My instances are currently set to 7, though, so I could probably help myself by increasing those.

Does this server do anything else except run `puppetserver`?

Also, `max-queue-length`? Is that the right name? Did you mean `max-queued-requests`?

And yeah, 7 jrubies seems low. I’d recommend trying 10-12 `max-active-instances` and seeing how that fares.

Ah good, then everything I linked and mentioned above applies.

FYI, in PE world pe_tune is available, you can _borrow_ logic from there, but it would be nice if someone did something similar for OSS

I came across something to help with it, I think.

It’s weird. My masters are suddenly being hammered today when they were fine before.

monitor when agents requests catalogs and avoid a thundering herd problem

and check if your JVM start options configure error  logging and then if you have error logs next to puppetserver.log. EOF errors also appear when the HEAP/codecache is too small. then an erorrlog would be written

also I can recommend reading <https://tickets.puppetlabs.com/browse/SERVER-3217|https://tickets.puppetlabs.com/browse/SERVER-3217> and the linked tickets

Thank you for the pointers. I'll be looking into those. The timestamps if these EOFs directly tie to the timestamps of error 500 on PUTs to the reports endpoint. There are no errors showing in the puppetdb side.

However, they started showing up when we threw a drive and have continued even after it was replaced

okay. if you still have any issues and want do debug stuff, let me know

I have some non-work stuff I have to focus on. I put the masters in 'store' mode last night so that the nodes wouldn't get so many failures, but I'll have to come back later today and look at it

<@U1PC2RAH0> Not sure what your TZ is. Dug in a little on the weekend. I’m not sure if tuning should be my actual focus yet. This had been working smoothly (as far as I can tell) prior to losing part of a mirror on the 14th.
We didn’t detect it right away and PostgreSQL performance dropped significantly and there were non-clean shutdowns involved.
I’m thinking that at the very least we got some index corruption. I saw some relationship errrors and am trying to find those again.

which postgresql version are you running?

My first clue of the issue on the 14th was that the stockpile/cmd/q had filled up

Before I knew it was disk, I removed older entries from that (I’d come across some advice on that)

I’m digging in the PostgreSQL logs for that timeframe to see what the errors were there.

```2022-09-14 03:19:20.452 EDT [3225611] puppetdb@puppetdb ERROR:  canceling statement due to statement timeout```

I want to see what the messages post-“recovery” look like.

ha, had the same errors a few days ago at a PE customer

I’m also getting ‘attempt to read body out of block’ on some concat::fragment (file) resources from the file_ endpoints. I have to assume they’re unrelated, but it’s also going on. :disappointed:

I’m tempted to blow away the DB and let PuppetDB re-create it.

Getting these errors since I disabled report posting on Friday evening:
```2022-09-18 20:00:09.434 EDT [427119] puppetdb@puppetdb ERROR:  relation "reports_20220919z" does not exist at character 13
2022-09-18 20:00:09.434 EDT [427119] puppetdb@puppetdb STATEMENT:  INSERT INTO reports_20220919Z ( status_id, environment_id, catalog_uuid, receive_time, hash, transaction_uuid, puppet_version, noop, logs, report_format, start_time, producer_timestamp, cached_catalog_status, end_time, producer_id, report_type, configuration_version, code_id, noop_pending, certname, metrics, job_id ) VALUES ( $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22 )
        RETURNING *```

mhm that one sounds odd, like if the whole table is missing

And seeing this frequently:
```2022-09-18 21:12:19.411 EDT [1437] LOG:  checkpoints are occurring too frequently (20 seconds apart)
2022-09-18 21:12:19.411 EDT [1437] HINT:  Consider increasing the configuration parameter "max_wal_size".```

or maybe the postgresql instance is just overloaded

I’m using puppetdb module on that node.

I suggest to check max_wal_size, it should be in the postgresql.conf file. and probably double it

Is that a package or just a metric in stats?

you can check iowait by running `iostat 1` on the box

there will be %iowait st the top, refreshing every second. the tool comes from the sysstat package

iowait means "cpu wants to do something but has to wait for the storage" and usually indicates a lack of IO performance for a cpu core

and when the max_wal_size is too small, postgres needs to write the wal log more often which causes a lot of IO

mhm, not sure what the default unit is. for my 260GB puppetdb I have a max wal size of 32GB

postgresql can write a checkpoint that will be use for recovery, after a DB crush

Maybe there’s a weird TZ thing going on between EDT and Zulu, though.

Yeah. This was two halts in a row with a bad disk present.

if you dont have exported resources, just throw the DB away

Well, we do, but they’ll likely re-export.

I’ll check with the module creator on that one, but I think it’s for nagios alarm definitions.

thats fine. just keep in mind that exported resources are gone for the first puppet run after the db cleanuo

so if you use exported resources for stuff like firewalling, that might break stuff

a workaround would be to stop all agents, pruge and recreate db, then run all agents once via noop to get resources back in the db, then without noop and there should be no changes

Yeah. I don’t have that kind of agent control at this point. Hoping to at some point.

I also highly recommend setting up <https://forge.puppet.com/modules/puppetlabs/puppet_operational_dashboards|https://forge.puppet.com/modules/puppetlabs/puppet_operational_dashboards> to get matrics for puppetserver/puppetdb/postgresql

I think I’ve already got Telegraf available.

We turned up those `wal` settings and also ran some patches. I’m going to turn report posting back on and hopefully we remain stable.

Do you know if we can publish reports that are stored on disk? I think they’re stored as JSON or YAML. Can’t recall

what do you mean, import old reports into puppetdb?

you can submit them via curl to the puppetdb api if you like

I think I’ll need to adjust auth.conf to allow other hosts to submit on behalf of a certname.

Hopefully this tuning fixes my issue. I’m somewhat pessimistic as the problem didn’t exist before the 14th and the size of our server population hasn’t changed radically.

8 masters talking to a single puppetdb host

okay that is quite large and I know for sure that the default values from puppetlabs-postgresql and puppetlabs-puppetdb are are made for suge a huge env

See the `puppetlabs-postgresql` module at <https://forge.puppet.com/puppetlabs/postgresql?src=slack&amp;channel=puppet>

A common problem is that the default config works. ut is at a tipping point. puppetdb does some cleanup on a regular basis. when available IO drops, because a disk fails, or you require slightly more IO, cleanup wont work -&gt; db size grows -&gt; usual operations take longerand will eventually fail

you can monitor that with the dashboard I linked. and ensure to monitor the disk growth and disk IO for the partition with the postgreswl data

hi All,
I have a question regarding the API

Is there any way to query puppet if there was any change in the code?

What I basically would like to achieve is detect frequently, if there is a change on the code, so a puppet agent should be started to apply the changes.

Is there any way to obtain that information from the puppet server?

You mean new module code? Puppet’s default interval is 30 minutes, so it should pick it up fairly often.

a webhook triggers r10k, r10k deploys code, calls choria, choria initiates puppet run

How are you running the agent currently?
Is it on a timer (cron/systemd/scheduled_task/built-in-service), or do you always manually start a Puppet run?

its not intensive on agent (usually), but quite expensive on the puppetserver side

This is not true.  I have come to learn exec evaluations aren't "nothing", and in fact don't run the same priority as puppet itself if set (ie to low).

It depends on the number of nodes and masters, of course. Decreasing interval can mean more catalog compiles and could affect load there.

&gt;  If nothing changed in the code, the agent won't do anything.
It checks if the local configuration matches to the one on the server, isn't that? Like is the content of the file the same like on the server?

In my environment it takes about 10 sec, even if it changes nothing

eg. collecting facts is a resource intensive part of the task.

Well, you could _lower_ the cron interval, and then have something on the master that watches the results of r10k or codemanager to see if it picked up something new and if it does, trigger a bolt task to run puppet on the nodes you want to go early.