Puppet Community

API requests are handled concurrently. As of Puppet Server 6.7.0, a mutex is traded around disk updates to serialize edits to shared state like the CRL, serial, and inventory files:

<https://tickets.puppetlabs.com/browse/SERVER-115>

Postgres' standard split would be to put the WAL logs on one device and the database files on another one. Maybe a further spit to isolate the location where temporary files are created for large joins or sorts.

For the PuppetDB service its self, the only place where significant I/O happens is around the work queue under the `/opt/puppetlabs/server/data/puppetdb` directory.

No. To be honest, the potential gains are marginal unless I/O is the resource bottlenecking a system and the pain of getting complex device configurations deployed correctly in an enterprise setting is extremely high.

So, this sort of split is normally treated as a bespoke implementation for Puppet Enterprise and is usually done by services for the handful of installations that need it.

I am currently facing performance issues with PuppetDB and struggling to find the root cause and thought this might be able to help

<@UCUE8H9U7> is this a virtual machine or physical?

It won't help unless your disk I/O is saturated.

yep. first, check if you've IO problems (for example iostat 1 -m)

<https://forge.puppet.com/modules/puppetlabs/puppet_operational_dashboards/readme> is good to get some metrics. also check how big the HEAP is und how much of it is used, ensure you use java 11 not 8, run postgres 13 or newer

the issue is mostly only noticable though when trying to run puppet tasks via the console

console times out trying to load the task when I start it

why do you think this is related to puppetdb?

Its more/less a permissions thing from what I can understand. As an admin user, I don't see the issue. however if I login with a non admin user, I can run tasks but can't see the result due to the console timeout issue

but its not a permission issue at the same time because the user has the correct permissions

I suggest to take a look at <https://forge.puppet.com/modules/puppetlabs/puppet_operational_dashboards/readme> and then check the logfile from console-services for errors/timeouts

Its a performance issue with the PDB from what I can understand but maybe I am wrong

image.png

As far as I can recall, PuppetDB isn't involved in the "Run Task" loop.

Aside from maybe using a query to gather targets for the task.

it's loading while it loads the list of available tasks?

check `/var/log/puppetlabs/*/*.log` for errors while opening the console

like its trying to understand the parameters of the class or something

Do non-admin users have several RBAC rules around which tasks can be executed? That's the biggest potential difference I can think of off the top of my head.

no, but they are restricted on which node groups they can run tasks on

I grant all tasks on a specific node group that the user should have access to

so in this particular case using extra scsi adapters wouldn't make any difference as you suggested

I don't think you've an IO problem, so no

I assume something is running in a timmeout

hence, check `/var/log/puppetlabs/*/*.log`

`tail /var/log/puppetlabs/*/*.log -fn0` works fine, then try to run a task

this sounds odd.
```==&gt; /var/log/puppetlabs/console-services/console-services_gc.log &lt;==
[2906051.327s][info][gc,start       ] GC(138283) Pause Young (Allocation Failure)
[2906051.332s][info][gc,heap        ] GC(138283) PSYoungGen: 341059K-&gt;3488K(343552K)
[2906051.332s][info][gc,heap        ] GC(138283) ParOldGen: 145076K-&gt;145491K(244736K)
[2906051.332s][info][gc,metaspace   ] GC(138283) Metaspace: 128882K-&gt;128882K(1157120K)
[2906051.332s][info][gc             ] GC(138283) Pause Young (Allocation Failure) 474M-&gt;145M(574M) 5.334ms
[2906051.332s][info][gc,cpu         ] GC(138283) User=0.02s Sys=0.00s Real=0.01s```

I am seeing that quite a bit but even without running a task

this would indicate that the console doesn't have enough memory right?

ignore the *_gc* files. is there something else?

That's completely normal for GC. Failure to find free space for a requested allocation is the usual cause for starting a GC cycle. Collection occurs frequently in the young generation.

this is about all I can get atm.
```==&gt; /var/log/puppetlabs/orchestration-services/orchestration-services.log &lt;==
2023-01-10T02:43:03.289+11:00 INFO  [pool-6-thread-91] [p.o.job] Transition job 4,326 to state failed```

If running the task job as the Admin consistently succeeds, but running the same task with the same targets and parameters as a non-admin user consistently times out, then there's something going on that we can help look into if you open a support case at <https://support.puppet.com>

thanks. I have actually got a support ticket open. but there hasn't been any success for 2 months

its currently with engineering but I have been told they are not all back from leave yet

:+1: I'll check up on it. Lots of folks have been out for end-of-year leave, but they should be trickling back in this week.

thanks I completely understand that. :slightly_smiling_face: just hoping to try get this issue resolved as its hindering our staff being able to use the product effectively

I will be planning an upgrade to Puppet 7 soon. Just got to make sure I get all the pre-reqs sorted first. including the change to filebucket algo :slightly_smiling_face:

I don't believe upgrading to Puppet 7 will resolve this issue though?

PE 2021 has a few years of changes and improvements relative to PE 2019, but I don't know off the top of my head if there were any significant re-writes of the task execution pipeline.

you don't happen to know what Puppet are doing with Ruby 3 do you? Ruby 2.7.x is EOL in a couple of months which Puppet 7 is using

Ruby 3.2 is planned for Puppet 8. For Puppet 7, we're backporting security patches for Ruby 2.7 through the lifetime of the PE 2021.7 LTS.

yeah ok. as I thought, there won't be any use of ruby 3 in Puppet 7

thanks for the help. I am going to call it a night. midnight here. :slightly_smiling_face: will wait for support to get back to me about my task/console issue. thanks for your help anyway

Yeah, bumping the Ruby version is very much a "major version" sort of change.

Is there any official documentation from Puppet about splitting PuppetDB onto separate SCSI controllers for performance benefits? Typically DBs would be on at least 1 separate SCSI controller to the OS as a standard convention. I am asking because I can't seem to find one but believe that there should be something and that I shouldn't be alone in asking this question.

hello, a query, when creating a smart-proxy to connect with foreman it gives me the following error "ERROR: Unrecognised option '--foreman-proxy-content-puppet' does anyone know how to fix it?

woops - i was cleaning out old channels :slightly_smiling_face:

took out one more than I meant to :slightly_smiling_face:

Is there a way with PQL to get a list of nodes where a specific resource was failing?

Or do you have to pull back all the failing reports and do some JQ magick?

I think the only DB that has that info is reports, and it's buried in resource_events.data

The `events` API should get you most of the way: <https://www.puppet.com/docs/puppetdb/7/api/query/v4/events.html#query-fields>

```events[certname] { status = "failed" and resource_type="Foo" and resource_title="bar" }```
You'll want some way to time box that query... I'm sure there's a way to get that in there... haven't looked too deeply myself.

<https://voxpupuli.org/docs/pql_queries/#get-all-nodes-where-one-specific-resource-changed|https://voxpupuli.org/docs/pql_queries/#get-all-nodes-where-one-specific-resource-changed>

Ah right, `latest_report? = true` to search through only the latest reports.

if you write a new awesone query, please send us a PR :)

<https://github.com/voxpupuli/voxpupuli.github.io/blob/master/_docs/pql_queries.md|https://github.com/voxpupuli/voxpupuli.github.io/blob/master/_docs/pql_queries.md>

`events [count(certname)] { resource_type = "Package" and resource_title = "&lt;packagename&gt;" and latest_report? = true and status = "failure" }` count?

`events[count()] {&lt;the query&gt;}` if you want just a count of results.
More examples are scattered throughout <https://www.puppet.com/docs/puppetdb/7/api/query/v4/pql.html#count>

I mean does that count as "awesome". Heh.