Puppet Community

The puppetdeb plugin defaults to using `uri = certname` <https://github.com/puppetlabs/bolt/blob/f2b1ec5724c4b4d5a3c6ad46c68fdbc8d7730435/lib/bolt/plugin/puppetdb.rb#L97>

URI will override a `host` in any transport config

Something like that?
```    target_mapping:
      config:
        ssh:
          host: facts.hostname```

```    target_mapping:
      name: certname```

:meeting: _:teacher:Puppet Core Team_ is about to start up in 15 minutes

Hi folks, Have a question about open source puppet, ask away!

aka we spin up a new node, and the first x minutes of the first puppet agent run are spent just syncing between the puppetservers and the node

puppet enterprise has an option where the puppetserver creates a .tar.gz file and the agent downloads that, with the pluginsync data

you can also do `puppet plugin download` to just do a pluginsync, that works in open source and PE

ah very cool, i'll check that out, thank you

from the docs: `For example, *server* and *run_mode* are valid settings, so you can specify *--server &lt;servername&gt;*, or *--run_mode &lt;runmode&gt;* as an argument.` what is `--run_mode`? it's mentioned throughout the docs but doesnt seem to ever be defined

first thing; setup <https://forge.puppet.com/modules/puppetlabs/puppet_operational_dashboards/readme>

do you use java 8 or 11 for puppetserver/puppetdb? Switch to java 11. configure reserved code cache to 2 GB. have at least 1 GB HEAP per jruby instance. If you have catalogs with 1500 or more resources, consider 1.5GB HEAP per jruby instance. running puppetserver 6? upgrade to 7.

if those quick changes won't fix the problems, we need to look closer at the actual errors and check the dashboard

• Right now we are using Java 8.
• I've already updated the reserved code cache to 2GB.
• Not sure about the HEAP setting for jruby instances. So I'll take a look at that.
• We are already upgrade to puppetserver 7.
I'll look into the Puppet Operational Dashboard.  Seems like a useful tool.

the heap is set with -xmx and -xms for the whole jvm process

usually defined in /etc/sysconfig/puppetserver

<@U04UDDNAGEL>, similar setup, about 3500 agents, one server. Around the time we crossed the 3000 agent threshold, I had the same problem. I set max-active-instances to `CPU - 1` and heap size to  `1GB + (max-active-instances * 1GB)` and everything ran perfectly smoothly from then on. This is on a 16 CPU/ 64 GB RAM server. I could potentially downsize it after those tunings, but if it ain't broke...

this is still with java 8 (11 isn't available on this old platform)

Oh, awesome!
Thanks <@U2HGUG2CD> I'll give that a try. Sounds like a very similar setup. Same server resources as well. :slightly_smiling_face:

<@U2HGUG2CD>
Didn't end up resolving my issue.
Back to the drawing board. Thanks anyways

Hrmm, are you using puppetdb (I am not). I also do not ship reports (network latency too high).

<@U04UDDNAGEL> did you install the dashboard?

<@U1PC2RAH0> - I'm just booting up my non-prod puppet server to try it out.
I haven't actually had to install a puppet module on the server itself before. So going to try it out and see how it goes.

how do you usually install modules? r10k?

Yeah, we usually have a Jenkin's pipeline that executes r10k to copy down some code from github.

We typically don't manage any modules on the server itself though, kind of new territory for myself.

add the module to your Puppetfile, let Jenkins call r10k

and when do you see timeouts? every 30 minutes? once a week? does the puppetserver.log contain any errors? do you have splay enabled in your puppet.conf? If so: does it have the same value as runinterval?

Timeouts are intermittent but are becoming more and more prevalent. I would say they are occurring very consistently now effecting most puppet runs.

The puppetserver.log does contain some timeout errors when attempting to generate catalogs.

All of our devices have splay enabled, our runinterval is currently set at 30 minutes for all devices.

can you share the puppetserver.log parts?

and please verify if splay is set to 30min

```2023-03-17T18:14:20.650Z ERROR [qtp2059159079-263] [p.r.core] Internal Server Error: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
	at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1083)
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:321)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.read1(BufferedReader.java:212)
	at java.io.BufferedReader.read(BufferedReader.java:286)
	at java.io.Reader.read(Reader.java:140)
	at <http://clojure.java.io|clojure.java.io>$fn__11538.invokeStatic(io.clj:337)
	at <http://clojure.java.io|clojure.java.io>$fn__11538.invoke(io.clj:334)
	at clojure.lang.MultiFn.invoke(MultiFn.java:239)
	at <http://clojure.java.io|clojure.java.io>$copy.invokeStatic(io.clj:406)
	at <http://clojure.java.io|clojure.java.io>$copy.doInvoke(io.clj:391)
	at clojure.lang.RestFn.invoke(RestFn.java:425)
	at clojure.core$slurp.invokeStatic(core.clj:6951)
	at clojure.core$slurp.doInvoke(core.clj:6942)
	at clojure.lang.RestFn.invoke(RestFn.java:439)
	at puppetlabs.services.request_handler.request_handler_core$body_for_jruby.invokeStatic(request_handler_core.clj:78)
	at puppetlabs.services.request_handler.request_handler_core$body_for_jruby.invoke(request_handler_core.clj:54)
	at puppetlabs.services.request_handler.request_handler_core$wrap_params_for_jruby.invokeStatic(request_handler_core.clj:86)
	at puppetlabs.services.request_handler.request_handler_core$wrap_params_for_jruby.invoke(request_handler_core.clj:81)
	at puppetlabs.services.request_handler.request_handler_core$jruby_request_handler$fn__41590.invoke(request_handler_core.clj:269)
	at puppetlabs.puppetserver.jruby_request$wrap_with_jruby_instance$fn__36905.invoke(jruby_request.clj:49)
	at puppetlabs.puppetserver.jruby_request$wrap_with_error_handling$fn__36901.invoke(jruby_request.clj:34)
	at puppetlabs.services.request_handler.request_handler_service$reify__41615$service_fnk__5011__auto___positional$reify__41630.handle_request(request_handler_service.clj:47)
	at puppetlabs.services.protocols.request_handler$fn__41528$G__41524__41531.invoke(request_handler.clj:3)
	at puppetlabs.services.protocols.request_handler$fn__41528$G__41523__41535.invoke(request_handler.clj:3)
	at clojure.core$partial$fn__5839.invoke(core.clj:2624)
	at puppetlabs.trapperkeeper.authorization.ring_middleware$fn__26381$wrap_authorization_check__26386$fn__26387$fn__26388.invoke(ring_middleware.clj:290)
	at puppetlabs.ring_middleware.core$fn__23841$wrap_bad_request__23850$fn__23853$fn__23859.invoke(core.clj:187)
	at puppetlabs.ring_middleware.core$fn__23939$wrap_uncaught_errors__23948$fn__23951$fn__23956.invoke(core.clj:233)
	at puppetlabs.ring_middleware.core$fn__23509$wrap_request_logging__23514$fn__23515$fn__23517.invoke(core.clj:47)
	at puppetlabs.i18n.core$locale_negotiator$fn__124.invoke(core.clj:357)
	at puppetlabs.ring_middleware.core$fn__23538$wrap_response_logging__23543$fn__23544$fn__23545.invoke(core.clj:53)
	at puppetlabs.puppetserver.ringutils$wrap_with_puppet_version_header$fn__37154.invoke(ringutils.clj:83)
	at puppetlabs.services.master.master_core$fn__43100$v3_ruby_routes__43105$fn__43106$fn__43123.invoke(master_core.clj:1175)
	at bidi.ring$fn__17722.invokeStatic(ring.cljc:25)
	at bidi.ring$fn__17722.invoke(ring.cljc:21)
	at bidi.ring$fn__17707$G__17702__17716.invoke(ring.cljc:16)
	at puppetlabs.comidi$make_handler$fn__19638.invoke(comidi.clj:245)
	at puppetlabs.metrics.http$fn__41883$wrap_with_request_metrics__41888$fn__41892$fn__41894$fn__41895$fn__41896.invoke(http.clj:152)
	at puppetlabs.metrics.http.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at puppetlabs.metrics.http$fn__41883$wrap_with_request_metrics__41888$fn__41892$fn__41894$fn__41895.invoke(http.clj:152)
	at puppetlabs.metrics.http.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at puppetlabs.metrics.http$fn__41883$wrap_with_request_metrics__41888$fn__41892$fn__41894.invoke(http.clj:148)
	at puppetlabs.comidi$fn__19703$wrap_with_route_metadata__19708$fn__19709$fn__19711.invoke(comidi.clj:332)
	at puppetlabs.trapperkeeper.services.webserver.jetty9_core$ring_handler$fn__29738.invoke(jetty9_core.clj:455)
	at puppetlabs.trapperkeeper.services.webserver.jetty9_core.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102)
	at puppetlabs.trapperkeeper.services.webserver.normalized_uri_helpers$fn__29308$normalize_uri_handler__29313$fn__29314$fn__29315.invoke(normalized_uri_helpers.clj:74)
	at puppetlabs.trapperkeeper.services.webserver.normalized_uri_helpers.proxy$org.eclipse.jetty.server.handler.HandlerWrapper$ff19274a.handle(Unknown Source)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1363)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:190)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1278)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767)
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
	at com.puppetlabs.trapperkeeper.services.webserver.jetty9.utils.MDCRequestLogHandler.handle(MDCRequestLogHandler.java:36)
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:500)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
	at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)
	at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more
	Suppressed: java.lang.Throwable: HttpInput idle timeout
		at org.eclipse.jetty.server.HttpInput.onIdleTimeout(HttpInput.java:801)
		at org.eclipse.jetty.server.HttpChannelOverHttp.onIdleTimeout(HttpChannelOverHttp.java:406)
		at org.eclipse.jetty.server.HttpConnection.onReadTimeout(HttpConnection.java:495)
		at org.eclipse.jetty.io.AbstractConnection.onFillInterestedFailed(AbstractConnection.java:172)
		at org.eclipse.jetty.server.HttpConnection.onFillInterestedFailed(HttpConnection.java:502)
		at org.eclipse.jetty.io.AbstractConnection$ReadCallback.failed(AbstractConnection.java:317)
		at org.eclipse.jetty.io.FillInterest.onFail(FillInterest.java:138)
		at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillableFail(SslConnection.java:579)
		at org.eclipse.jetty.io.ssl.SslConnection.onFillInterestedFailed(SslConnection.java:407)
		at org.eclipse.jetty.io.ssl.SslConnection$2.failed(SslConnection.java:167)
		at org.eclipse.jetty.io.FillInterest.onFail(FillInterest.java:138)
		at org.eclipse.jetty.io.AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:407)
		... 9 more```


I guess that from time to time more agents hit the server than you've free jruby instances

how many jruby instances do you have? 2500 nodes is quite a lot fora single puppetserver (if runinterval is 30min and the catalogs are big/take long to compile)

We just set the max active instances to 15 yesterday to see if that alleviates any issues, but it didn't. :disappointed:

I bet that's too low when splay isn't working perfectly

Splay is set to 10 minutes, and the runinterval is currently 30 minutes for our agents.

that means that if you restart all your servers at the same time, all puppet will hit the puppserver within 10 minutes

I highly recommend setting splay to the same value as runinterval

let's say you've 2500 nodes and runinterval is 30 minutes. that's 83 catalogs a minute. spread across 15 jruby instances that's 5 catalogs per jruby instance per minute

now it highly depends on how big your catalog is. some can be compiled in 5 seconds, some take a minute. puppetserver.log logs the compilation time

but even when your agents are perfectly distributed across the 30 minutes, each compilation needs to stay below 60/5=12 seconds

the dashboard will tell you how many jruby instances are requested and how many free instances you have. I guess there's a bottleneck. it will get better when changing splay to 30min. but it maybe won't fix the problem

there's always the possibility that my math is wrong :smile:

:businessparrot: Next up is _Content and Tooling_ in <https://www.timeanddate.com/worldclock/converter.html?iso=20230321T150000&amp;p1=202&amp;p2=179&amp;p3=919&amp;p4=307&amp;p5=3332&amp;p6=236&amp;p7=240|165 hours>

Alright folks, that's a wrap for today, see you in two weeks!

that should not be the case. the agent does the pluginsync on each run. when the code on the puppetserver didnt change and nobody deleted the lib dir on the agent, it nothing will be downloaded

thie behaviour happens often when the environment changes for the puppet run and different environments contain different code

I recommend checking the puppet agent log for a day and check for the used environment during each run

If you are using a package resource and `ensure` was set as `present` or `installed` it would only ensure it was there and not upgrade even if the repos changed but more specifically you should be able to use this fine with

```apt::mark {'locales'':
  setting =&gt; 'hold',
}```
If you don't mind could you raise a ticket at <https://github.com/puppetlabs/puppetlabs-apt/issues> to highlight lack of example and documentation there

Thanks! I finally chose this simple statement:
`package {'locales':  mark =&gt; "hold",}`

usually /etc/sysconfig/puppetserver or /etc/defaults/puppetserver

and theforeman/puppet is a puppet module you can use to manage it

See the `theforeman-puppet` module at <https://forge.puppet.com/theforeman/puppet?src=slack&amp;channel=office-hours>

I would like to change `JAVA_ARGS` which is currently set for my puppet server as:
```JAVA_ARGS="-Xms20g -Xmx20g -XX:ReservedCodeCacheSize=1024M -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger"```
Any idea what was the *default* setting for the JAVA_ARGS? I have puppetserver version: 5.3.3

theforeman/puppet will also provide you with defaults

Thanks! I finally chose this simple statement:
`package {'locales':  mark =&gt; "hold",  }`

Thanks! I finally chose this simple statement:
`package {'locales':  mark =&gt; "hold", }`

if its puppet enterprise, use code manager

:fry_dancing: _Content and Tooling_ is about to start up in 15 minutes

Hey everyone! :slightly_smiling_face: :wave__:

Welcome Everyone to the Wonderful World of the CAT Open Hours!!!

The resource_api gem currently fall's under the Phoenix/Agent team

do you happen to know if they have an office hours?

Should be on the community calendar: <https://www.puppet.com/community/calendar>

<@U2J8X3V7Y> is there a particular question you have?

I just have a couple of PRs that have been sitting out there for months. wanted to see if the resource API was getting any attention and chat with someone if there were any questions/concerns on them.

although it has been so long now I've forgotten just about everything that isn't in the PR itself. lol

They do look nice... We can give them a quick look over ourselves and then ping phoenix if you want?

afaik, they aren’t actively working on it but I’m sure <@U11567REE> will be able to help :slightly_smiling_face:

The resource_api is something we have some interest in ourselves at the moment.

it has been a few months, but iirc last time I asked Josh in office hours I think he said it belonged to a different team.  of course, I'm sure there have been all sorts of team/organizational changes the past few months.

Yeah there has - but we will certainly help you get your PRs in

the contributions are very welcome :slightly_smiling_face:

I'm not exactly in a hurry for them. but it sounded like someone else was quite interested and I'd hate to have them sit so long they needed to be significantly rebased.

currently it's <https://github.com/puppetlabs/puppet-resource_api/blob/main/CODEOWNERS#L2>