Puppet Community

thanks for the response
This is the line that throwing the error
`lookup('classes', {merge =&gt; unique}).include`
Merge doesn’t work for some reason
Also I’ve changed my env to prod

also this lookup throw the same error
`puppet lookup classes --merge unique --environment prod --explain --debug`

I’m checking all my Hiera files, global and env level and module level can’t see where is it getting the `undef`

so, your hiera is not configured in prod environment

This won't fail
```lookup('classes', Array[String], 'unique', []).include```

```$ puppet lookup classes --node <http://mynodename.mydomain.com|mynodename.mydomain.com> --environment prod
---
- role_puppetserver```
It is configured

Just tried `lookup('classes', Array[String], 'unique', []).include`
Same error

Same still
`Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Lookup of key 'classes' failed: The second element of the merge has wrong type, expects a value of type Scalar or Array, got Undef (file: /etc/puppetlabs/code/environments/prod/manifests/site.pp, line: 11, column: 1) on node nodename`

```# cat /etc/puppetlabs/code/environments/prod/manifests/site.pp
# global defaults

Exec { path =&gt; '/usr/bin:/usr/sbin/:/bin:/sbin' }

File {
  show_diff =&gt; false,
}

# install the classes
#lookup('classes', {merge =&gt; unique}).include
lookup('classes', Array[String], 'unique', []).include```

```$ cat /etc/puppetlabs/code/hieradata/nodename.yaml

---
classes:
  - role_puppetserver

role_puppetserver::passwords:
  postgres: &gt;
    ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBAD
    AFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAKOZuOD10sGxUa0uWIMH80XIx/5
    bHgoY7wMjxm77KubGx95vsoR9AQpQ+CWuW2rIWPpf0N3lZvZzGZeN7mlaCB5
    hmCdpKI+x7lMb34YshJjleM/J+6AjBt6OWXNSnfM/XTGsKlFiimf8NH/Vgoq
    Xj+TEDnjJKM/v1t1VYlmWWZ2CKVRViuCArsqZ0Uo3H7BaQfRq4l23iIlcIyE
    PPeUBxYc8JJuZo4bIR6VOB4hPjGgU9TQxEjPmUv0TGYdjH01EvAWr7c7Ns9X
    Ufy7AH8d5+96Ol9+HPuu9QobqRcjUiAF9KSg1p6ckDP86z0KVtR9l/M1O8ij
    K+I79dIgtxt3D3ojBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBCb2aSWyR
    fX5jXv+3oVe8ocgCBAc1kY9yj40T4KPAzree19WR5DuXxp5XmVDc2mXfaBfQ
    ==]```

```puppet lookup classes --node <http://mynodename.mydomain.com|mynodename.mydomain.com> --environment prod --explain```


```    Hierarchy entry "Host secure data"
      Path "/etc/puppetlabs/code/hieradata/mynodename.mydomain.com.eyaml"
        Original path: "%{facts.networking.hostname}.eyaml"
        Found key: "classes" value: [
          "role_puppetserver"
        ]```

no its one file only `eyaml` it was my typo

but the key `classes` only has `role_puppetserver`
and then we pass a parameter to that class

it can lookup the classes,
It can’t merge it for some reason

and there is no 'classes' attribute anywhere else?

```cat ./code/environments/prod/data/common.yaml
---
classes: []```
I have an empty one here

also tried to remove it to see if it makes a difference

thought `[]` would be interpretted as an array

also, by default lookup would use production environment, not prod

*Update on the scheduled maintenance:* :wrench:  *Back online!* The Forge infrastructure upgrade was successful. Done! :white_check_mark:

Also the <https://puppet.statuspage.io/> has been updated for those of you that subscribed to the Updates :slightly_smiling_face:

Yep works well, if there are updates :slightly_smiling_face:

I have experience with Skype, Slack, DingTalk, Symphony.. and Slack is the best of things I tried for the teamwork.. if (and only if) email is used in a company too.. otherwise everything sucks

ah, there was Teams as well.. but just to deal with corporate IT for short time.. and it was awful, yes :slightly_smiling_face:

the Teams API is absolutely crap for anything usable

I'd do chatops in slack or discord any day of the year

I honestly wish discord would make a separate Discord for Business product line

some of us don’t game, therefore we weren’t exposed to Discord, therefore we don’t use it when we already have half a dozen chat products to use.

My kids keep trying to get me to use discord, but I have no use for it.

since we all are still in on IRC we could just keep using that

I think when I went to my latest new computer, I never brought over an IRC client

heck, we even manage a Slack for our family communications

for me, it's the ability to use roles and real permissions that is Discord's killer feature, for gaming communities it means people can be given a role that matches their role in game with permissions and channels related to it, for (small) businesses you can create roles for the different teams/jobs and each of them can be given different accesses and permissions (like support folks can read/react to the ops channel, but not post etc)

I can see that.  We manage one for our church staff too, and it *is* all or nothing.  You’re in the group and can read it or you’re out.

I was doing ChatOps in IRC in late 199x :smile: by writing scripts for mIRC

so chatops is not that new tech, right? :wink:

I_Guess.gif

I mean if you write die on a big chunk of stone and get something to drop it on someone, it's an operation and you chatted into it

We already have an RFC for IP by carrier pigeon

That’s <https://www.rfc-editor.org/rfc/rfc2549|RFC2549>, for reference

I wonder i that would apply to dragons…. hmmmm

I've said it before, but everything went down hill after Usenet went out of fashion

If the exported resources are from a node that is no longer supposed to be running, then revoke the certificate and de-activate it.

If the exported resources are from a node this is supposed to remain running, update the classification to remove classes that are exporting the resources.

it worked. I could not find the node, which was exporting ....

this is a good place for it. Happy to chat. I've seen both.

Have my fair share of opinions on that as well, so please, rant right away :popcorn:

Organizational behavior is always a hard (but interesting) topic.

From our experiences in Puppet’s customer-facing teams, when the directors or top managers _are not onboard_ in automation or improving their infrastructure processes then nothing will change. And something similar happens when those managers are onboard but they are married to what they know or what they have heard it works (_change is always hard isn’t it?_), for example using the good old Bash scripts or  another tool (Ansible).

If you are part of the enablement team, you might take the chance to showcase Puppet’s benefits such as the declarative language that saves you from writing if-then-else cases to manage software, the ease of scale from 100 nodes to 100,000 nodes without scaling the teams, reports, the variety of modules, the community (:hello: hello all of you), and the extensibility options such as the ad-hoc reusable tasks (Orchestrator API), Impact Analysis- CD4PE (it’s getting better eh!), our available tooling. It’s all about long-term value creation and the ability to maintain a healthy and predictable workflow for your infrastructure.

*Managing upwards*

Showing a POC or a demo also helps to solidify the message, and try to align the demo with the goals of top management, as we usually say Puppet seems “hard to grasp” at the beginning but it’s one of the easiest ways to scale in Production.

However, if the top managers don’t get it or they have the preference as I mentioned before for a different tool, there is not much you can do.

One thing from personal experience: once management engages and decides to, after learning of and being convinced by the benefits,  jump the NEWTECH train (It's Puppet here, but in general) it's paramount you keep a close eye on people actively working against that upcoming change (there's no upper bound for hideousness here) and combat and sanction such behaviour.

keeping a close eye on those people is important for several reasons.  Are they adverse to change, hoping you fail, trying to get you to fail, or waiting to step in to be a hero?  What I have seen is these are the folks that stand up new tech, script a bunch of stuff, tout their automation to mgmt, but there is no coherency around it, beyond their 1 off. It isn't part of the puppet modeling, and its just another snowflake.  You end up with never meeting the comprehensive puppet mgmt.

One of the problems I've seen is not bringing in multiple tools. A couple of years ago I was the Puppet Guy at a company. A team approached me about using Ansible or similar for some deploy and runbook like tasks. I was fine with it though had a set of principles that I wanted them to follow to unsure consistency.

They were surprised and had expected pushback. My take was the Puppet operated better when it does what it's good at. Keeping inappropriate use cases out of Puppet was part of the job.

Wow, was not expecting such a reception for my aches! So here I go:
Yesterday I learned my company decided to progressively abandon Puppet and switch to Ansible. Obviously, a management, not a technical decision.
I have nothing against Ansible, or whatever technology one can use to make things work. But it's like <@U12006DC1> said: each one has its use, and Ansible's is not the same as Puppet's.
Indeed, I was assigned the task of presenting a comparison between both tools. My conclusion is both are great, but one can't replace the other. I now wonder if this presentation ever got higher than my manager...
I told my immediate leader about all that, but he just made clear it's a decision well above his own head. Our hopes are that, during the migration process, someone understands how terrible the idea was.
For reference, I'm talking about thousands of servers, with several dozens of house-developed modules (along with a huge lot of community ones). From what I learned in the comparison, I look forward to see people converting the most intrincated codes...

&gt; keeping a close eye on those people is important for several reasons.
&gt; Are they adverse to change, hoping you fail, trying to get you to fail, or waiting to step in to be a hero?
Indeed a good reminder also for myself: you are absolutely correct, that one has to very carefully look at and interview people and try to really grasp where their resistance is coming from. If they are acting in good faith, I think addressing concerns and taking them aboard by going the extra mile is well worth your time. That being said, my paragraph was mainly aimed at people where those efforts have failed or cannot be resolved reasonably.

&gt; I told my immediate leader about all that, but he just made clear it's a decision well above his own head.
That is unfortunate indeed. Rarely have I seen instances where, what is predicted at lower levels as the consequences of that decisions (usually technically sound) have not manifested and led to pain and frustration.

&gt; Our hopes are that, during the migration process, someone understands how terrible the idea was.
I wish you all the best with that, for sure, but my experience is that once this becomes clear, ass-covering sets in, the sunk-cost-fallacy is invoked and it is tried to beat the horse over the finish line.
What works, on the other hand, is expressing your technical arguments in a language that upper management understands: man hours and hard $$$.
Would be interesting to hear why they think this is a good move, because it sure sounds wrong, even from afar.

<@U048W3UL9EU> They main explanation is that "too few people in the company knows Puppet". Puppet development was kinda constrained in the Automation team (mine). Other involved teams just didn't jump in the train, and started using Ansible. Now, the company feels Ansible knowledge is well spread, what eventually became true.
The hard truth is that most teams weren't comfortable even with basic shell scripting, let alone convoluted code with flexibility demand. Then, instead of directing the teams to learn and master Puppet, the company just accepted it's "too hard".

OTOH, I think some "mea culpa" is in order. Puppet learning curve is indeed more steep, what shouldn't be a problem for a big company - if documentation can fill the blanks. Unfortunately, there are areas where Puppet documentation is lacking.
My recent experience trying to learn to write custom resources and their tests was a living hell. I intended to generate internal guides for that, but I wonder if they will scare the teams even more.
I was also involved in some discussion about supporting DNF modules, and I tend to conclude there is some support in `package` resource, but a too confusing one. I could understand by the examples given here in Slack - but should every user go that far to write code for DNF modules?

The other "mea culpa" is for the company itself, by seeing automation as an assembly line for quite a while.
It started as a sequence of "automate X in Y months", instead of creating a codebase and growing from that. You can guess the amount of hardcoded and unreusable stuff and piles of "execs". Plus, the waterfall approach left no space for updating, documenting, and overall improving existing modules.
Now we ended up with a hardly mantainable base and a broken process. Solution? Switch the tool, of course.

<@U3XLGRAEM> those are some great insights into your current system. I'd write them up.

I once did a CTO brief about Puppet I titled as "Why not Chef?" to address the elephant in the room. Puppet at the time was unstable and not well understood internally. And version 0.25. I suggested that current state was a company/culture problem rather than a tooling a problem. Then walked through the process of how we were going to dig ourselves out of the current hole. I did end with, "once we have a system we understand it will be simpler to move to a new tool if we desire. Doing it today with the number of unknows we have would be harder IMO. Also all the testing, tooling, and infra improvements would directly apply to a new system."

<@U12006DC1> Nice one. We'll end up either trying to translate the whole beast, which requires Puppet knowledge anyway, or rewrite everything from scratch.

I'd write it up the learnings and mea culphas and use them to ask the hard questions about how to avoid these in the future. People who don't understand systems are going to make a mess in any automation tooling.

One concept that was useful was the "95%, 5%, and all the knobs" concept. We spent a fair amount of time packaging up profiles around modules. By default it would solve the primary case, 95% of the users, with no input. The secondary case with 1-2 params would solve the other 5%. Lastly make sure we could support an all the knobs approach for the outliers.

I'm going to stop after this (really really), but found this description of layering useful in thinking about role, profile, module. <https://blog.europython.eu/kraken-technologies-how-we-organize-our-very-large-pythonmonolith/>

I think it also highlights the value of good architectural control for whatever system you're creating.

We have some large organisations that are handling 100,000+ nodes.

But it was a very intentional action and they are using an XL architecture with vitamins. I don’t want to romanticize the idea that it will be :sparkles: perfect :sparkles:  without problems because it’s not true, but it’s doable.

Also Puppet’s Product team is aware of the scalability topic and since last year Engineering is working on improving it.

<https://www.puppet.com/docs/pe/2023.2/supported_architectures.html#supported_architectures-xl-installation>

I mean it’s expected to run Puppet on 1000 of nodes :slightly_smiling_face:

speaking of 10000 and more it’s different topic.. anything need heavy lifting here (except choria maybe :smile:)

The main thing that starts to become important above 10,000 and is absolutely critical by 100,000 is that you have to start paying attention to metrics and small inefficiencies.

Some custom function or external classifier that was riding along with multi-second response times might be fine at 1,000 nodes, but for 10,000 nodes each additional second of compile time added to the node population is going to cost at least 6 JRubies of capacity to cover.

Similarly, shipping a 100 MB installation artifact to the node population via the Puppet File Server might work with 1,000 nodes, but you definitely need to move that traffic to a dedicated service like Artifactory or Nexus at 10,000+ nodes.

Large node counts multiply small inefficiencies and turn them into problems.

And, depending on how people deal with the development process, we can get huge inneficiencies. Waterfall and tight deadlines can increase them by a factor of OMG.
Do "Go Horse" have a meaning in English?

hehe oh yeah, Rspec is such an amazing tool.

And I can’t agree more with you, it takes time to wrap your head around Rspec’s lingo (`context` VS `describe`, examples, etc).

I’d suggest to spread more words and examples on how to use it and how great it is.. because when you’re into the topic it’s easy.. but again I can remember how hard it was to get into the topic for me :slightly_smiling_face:

it’s even harder for less experienced ppl

check that you've enabled autovacuum in postgres and that it runs from time to time (postgresql logs that). and you should enable pg_repack

puppetdb does not store a history of facts, only the last factset per node. so if it constantly grows a) you r cleanup isn't working b) your facts are growing/changing all the time

so check through the nodes for facts that you don't need and disable them

Thank you for the reply. Autovacuum is enabled and does work. pg_repack is not installed though. I will have a look at that.
We have indeed a lot of facts, because of custom facts. Although the number of agents doesn't increase, the size of the database does.

Thanks again for the suggestions. We will give pg_repack a try.

We are currently running Puppet 7.24 with puppetdb version 7.13.0.

so for my ~500 nodes I've 60MB in the factsets table

disabling legacy facts is a good way to get rid of some facts, but it sounds like you've a few thousand facts per node or large nested hashes

We have some nodes with &gt;2000 facts. This is caused mostly by a custom fact `pkg_&lt;package_name&gt;` with the value of the package version. Any installed package gets a fact. We already discussed internally to disable this, but some guys still find this useful.
Disabling legacy facts is already planned, because of Puppet 8. We have still several places in our own modules using them. I'm currently replacing them with modern facts, but this take some time.

`packages =&gt; { 'pkg1' =&gt; 'version'..}` would help a bit

I agree. I try to get rid of it, but the decision is done by others.
Is a large facts hash more efficient than a lot of individual facts?

If I remember it correctly it depends a bit how many levels a hash has

it helps if all servers have the same package version, so they would have the same facts. puppetdb normalizes it

but in general you can't do much when you need to keep the facts. get fast storage for your postgres and try to run postgres14 or newer :slightly_smiling_face:

The postgresql is currently version 12.6, managed by our database team. If a newer version is better, I'll ask for it at the DB-Team.
I will advocate again internally for getting rid of the pkg_ facts. I think, it creates more hassle than use.

Thanks again for your input. :slightly_smiling_face:

your welcome. in general newer postgresql is always better

Aye, facts are useful but they have an aggregate price in storage and handling that adds up. `pg_repack` can help by smoothing over bloat left behind by "bursty" facts that report a lot of data one puppet run and then nothing the next.

Beyond that, you have to look at what is used by catalog compilation, what is critical for reporting purposes, and prune the rest.

<@U11MDJNGZ> do you remember what was worse? a fact hash with many elements or a deep hash?

Josh mentioned something like that some time ago

would love to run PostgreSQL 14, but <https://tickets.puppetlabs.com/browse/PDB-5379>

yeah the puppetdb module is in a pretty bad shape. but postgresql 14 in general works with puppetdb 7

Long arrays that changed frequently (like installed packages) _used_ to be bad because PuppetDB spent a lot of time de-duplicating the data.

That's much better now that facts are stored as a big hash. But, the hash depth can get you. Basically, that huge index is tracking all the "paths" through the factset --- so the more leaf values you have, the bigger it is.

ah okay. so try to avoid long nested hashes and changing arrays (or changing facts in general)

Yeah, deeply nested hashes where the values change often will inflate the size of that `idx_factsets_jsonb_merged` because the set of "paths" through the data changes.

wow….

so repeat after me…  Puppet is *NOT* a patch management or packaging solution.

It can be, you just have to think about efficiency in order to do it at scale. For example, don't report the version number of every package. Report the number of pending package updates, possibly split up by criticality. Give folks Bolt Tasks to gather large amounts of detail such as "which package names and which versions?" that are only needed briefly for decision-making.

It should be noted, <@U11MDJNGZ> that I was quoting a former Puppet employee with that one.  :smile:

After a while of telling people "you can do that, but you have to be careful" and watching them repeatedly not be careful, the answer can change to "don't do that. ever." :wink:

or my personal albatross…. “just let me do it”

The core problem is that computers are fantastically good at making resources appear infinite until you've painted yourself into a corner. Then, there's months or years of decisions that have to be re-visited and re-factored for efficiencies.

And it's tough to find the middle ground between "premature optimization is a waste of time" and "don't tolerate workflows that consume O(n^2) resources just because it happens to be working out ok today"

Also, Moore's law has done a fantastic job of absolving us all of our sins for the past few decades because faster compute just pops out of a birthday cake and allows us to sweep stuff under the rug.

Maybe. I sure got some scars :stuck_out_tongue_winking_eye:

I wanted to work for you guys in that “heyday” (12-18)… now in retrospect, maybe I didn’t.  :smile:

PE does package inventory reporting in facts, and patch management

yeah. we use it and find it very useful, but only managing about 2000 nodes

PE also splits that package fact out of the dataset and stores it in a special PE-specific table that is optimized for the needs of a "all packages and versions" dataset :wink:

Did a full vacuum of the whole postgresql DB:
```puppetdb=&gt; vacuum full verbose analyze;```
Size of the table factsets went down from 25GB to 81MB:
```puppetdb=&gt; select pg_size_pretty(pg_total_relation_size('factsets'));
 pg_size_pretty 
----------------
 81 MB
(1 row)```
Index `idx_factsets_jsonb_merged` is now 30MB in size:
```puppetdb=&gt; select pg_size_pretty(pg_relation_size('idx_factsets_jsonb_merged'));
 pg_size_pretty 
----------------
 30 MB
(1 row)```

Non-full vacuum doesn’t return free space to the OS usually (only is last page is freed iirc)

Or periodic full vacuum (on weekend night maybe)

I dont think that the autovacuum was afressivr enough. I cant imagine that almost 30GB were locked

I did a non-full vacuum of the whole DB beforehand, which freed ~2.5 GB of disk space, but left the size of the index untouched.
The full vacuum took ~1 minute. The DB was 33GB in size before and 3.5G after.

We will add pg_repack to all our postgresql instances to keep things tidy.
Thanks for all replies.

<https://github.com/puppetlabs/puppetlabs-pe_databases/blob/main/manifests/pg_repack.pp|https://github.com/puppetlabs/puppetlabs-pe_databases/blob/main/manifests/pg_repack.pp> this is designed for PE but it shows you the tables that are repacked and with which config/interval

you can try to use `CREATE INDEX .. CONCURRENTLY` then drop old and rename new one.. I was using this trick 10 years ago with non-PRIMARY indices :slightly_smiling_face:

REINDEX was locking the table that time.. worth to recheck the docs as it might be improved these days

Reindexing without concurrently works. Don't know, if concurrently is necessary.
```puppetdb=&gt; select pg_size_pretty(pg_relation_size('idx_factsets_jsonb_merged'));
 pg_size_pretty 
----------------
 95 MB
(1 row)

puppetdb=&gt; reindex table factsets;
REINDEX
puppetdb=&gt; select pg_size_pretty(pg_relation_size('idx_factsets_jsonb_merged'));
 pg_size_pretty 
----------------
 30 MB
(1 row)```

`REINDEX` locks the table for the time of reindexing.. that might cause notable delays in SQL queries (especially in UPDATE/INSERT ones). So you should do this only during the low load time.. but better to use `CREATE INDEX new CONCURRENTLY` + `DROP INDEX old` + `ALTER INDEX new RENAME TO old` (IIRC)

Ha! In postgresql 15 you can use REINDEX CONCURRENTLY!

it was introduced in PostgreSQL 14 actually! Nice addition!

though it works only in REINDEX INDEX idx_name CONCURRENTLY form.. so no way to do REINDEX TABLE xxx CONCURRENTLY.

We are on PostgreSQL 12.6 currently. The reindex takes ~5 seconds.

well.. then you can make it a cronjob/systemd timer and call nightly maybe

as long as bloat is not that huge it should be quick enough

usually the manual reindex should not be required when autovacuum works correctly + pg_repack is running

yep, autovacuum + pg_repack is kind of general solution

I'm a beginner in postgresql, but when I read the documentation correctly, non-full vacuum does not rebuild indexes. Only full vacuum, cluster and reindex does: <https://www.postgresql.org/docs/12/routine-vacuuming.html>
&gt; Plain VACUUM may not be satisfactory when a table contains large numbers of dead row versions as a result of massive update or delete activity. If you have such a table and you need to reclaim the excess disk space it occupies, you will need to use VACUUM FULL, or alternatively CLUSTER or one of the table-rewriting variants of ALTER TABLE. These commands rewrite an entire new copy of the table and build new indexes for it. All these options require an ACCESS EXCLUSIVE lock.
We will introduce pg_repack for regular maintenance. autovacuum is enabled already.
```puppetdb=&gt; show autovacuum;
 autovacuum 
------------
 on
(1 row)```

check the postgres log. I don't think autovacuum cleans up enoguh dead tuples

Haven't checked the logs yet, but the documentation states:
&gt; The standard form of `VACUUM` removes dead row versions in tables and indexes and marks the space available for future reuse. However, it will not return the space to the operating system, except in the special case where one or more pages at the end of a table become entirely free and an exclusive table lock can be easily obtained.

yes. pg_repack will help to return the space

and vacuum is quite conservative with the default setttings, to not require much IO. because it doesn't want to influence the server performance

so quite often it only deletes a fraction of dead tuples

In our instance, the outcome of auto vacuum isn't being logged, because this is disabled:
```puppetdb=&gt; show log_autovacuum_min_duration;
 log_autovacuum_min_duration 
-----------------------------
 -1
(1 row)```
Setting this to 0 will log all auto vacuum operations. We consider enabling this to get a clue, what's going on.

it’d be nice if puppetdb module would be able to configure the maintenance things too.. though environments are different

I've this at $customer:
```max_connections = 400 # (change requires restart)
shared_buffers = 4096MB # min 128kB
dynamic_shared_memory_type = posix # the default is the first option
max_wal_size = 6144MB
min_wal_size = 80MB
log_destination = stderr # Valid values are combinations of
logging_collector = on # Enable capturing of stderr and csvlog
log_directory = log # directory where log files are written,
log_filename = 'postgresql-%a.log' # log file name pattern,
log_rotation_age = 1d # Automatic rotation of logfiles will
log_rotation_size = 0 # Automatic rotation of logfiles will
log_truncate_on_rotation = on # If on, an existing log file with the
log_line_prefix = '%m [db:%d,sess:%c,pid:%p,vtid:%v,tid:%x] ' # special values:
log_timezone = 'Europe/Berlin'
datestyle = 'iso, mdy'
timezone = 'Europe/Berlin'
lc_messages = 'en_US.UTF-8' # locale for system error message
lc_monetary = 'en_US.UTF-8' # locale for monetary formatting
lc_numeric = 'en_US.UTF-8' # locale for number formatting
lc_time = 'en_US.UTF-8' # locale for time formatting
default_text_search_config = 'pg_catalog.english'
listen_addresses = '*'
port = 5432
ssl = on
ssl_cert_file = '/opt/puppetlabs/server/data/postgresql/14/data/certs/_local.cert.pem'
ssl_key_file = '/opt/puppetlabs/server/data/postgresql/14/data/certs/_local.private_key.pem'
ssl_ca_file = '/etc/puppetlabs/puppet/ssl/certs/ca.pem'
effective_cache_size = 21552MB
maintenance_work_mem = 2048MB
wal_buffers = 8MB
work_mem = 8MB
max_slot_wal_keep_size = 12288MB
checkpoint_completion_target = '0.90'
checkpoint_timeout = 10min
log_min_duration_statement = 5000
autovacuum_vacuum_scale_factor = '0.08'
autovacuum_analyze_scale_factor = '0.04'
autovacuum_vacuum_cost_limit = 1250
wal_level = logical
max_worker_processes = 40
max_replication_slots = 20
max_wal_senders = 20
shared_preload_libraries = pglogical
autovacuum_max_workers = 5
autovacuum_work_mem = 1005MB
log_checkpoints = on
log_connections = on
log_disconnections = on
log_lock_waits = on
log_temp_files = 8192
log_autovacuum_min_duration = 0
log_min_messages = warning
log_min_error_statement = error
log_error_verbosity = default
jit = off
ssl_min_protocol_version = 'TLSv1.2'
ssl_max_protocol_version = 'TLSv1.3'
ssl_ciphers = 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256'```

Our postgresql instance is not managed by the puppetdb module, because it's running on a different server. But for local instances, it would be nice, yes.

In our DB the default auto vacuum threshold is used with 50. The scale_factor at factsets is set to 0.80. Don't know, where this come from. I think, this means, that auto vacuum will start with 40 updated/deleted tuples.
```puppetdb=&gt; show autovacuum_vacuum_threshold;
 autovacuum_vacuum_threshold 
-----------------------------
 50
(1 row)

puppetdb=&gt; select relname, reloptions from pg_class where relname = 'factsets';
 relname  |              reloptions               
----------+---------------------------------------
 factsets | {autovacuum_vacuum_scale_factor=0.80}
(1 row)```

Yeah, reduction from 25 GB to 81 MB is a sign that autovaccum was not keeping up with growth in that table at one or more points. With a size of 25 GB, the `autovacuum_vacuum_cost_limit` was likely forcing frequent timeouts while autovaccum was running, which further compounded the problem.

In general, while there is some tuning you can do to autovacuum, the pattern of data coming out of Puppet is inherently bursty as adding or removing a module or fact can produce large changes in the data being reported.

Because of that, implementing something like scheduled `pg_repack` runs is the first thing to do. Tune autovacuum if you still have problems after that.

`ensure =&gt; latest` means to upgrade to the latest version each time Puppet runs.

`ensure =&gt; present` will install the current version, but won't upgrade it further.

`ensure =&gt; $version` will install a specific version.

Correct, but with dnfmodule, latest is the name of the stream.

```$ dnf module list |grep nvidia
nvidia-driver          latest          default [d], fm, ks, src          Nvidia driver for latest branch
nvidia-driver          latest-dkms [d] default [d], fm, ks               Nvidia driver for latest-dkms branch
nvidia-driver          open-dkms       default [d], fm, ks, src          Nvidia driver for open-dkms branch
nvidia-driver          530 [e]         default [d], fm, ks, src          Nvidia driver for 530 branch
nvidia-driver          530-dkms        default [d], fm, ks               Nvidia driver for 530-dkms branch
nvidia-driver          530-open        default [d], fm, ks, src          Nvidia driver for 530-open branch```