its running choria --version 0.29.4

Which error did you get when you tried to install it with puppet?

this didnt worked

the choria class has no "broker" parameter. The broker is managed with choria::broker. Here is the conf I use for end-to-end testing of choria when I update the FreeBSD ports: https://github.com/smortex/freebsd-puppet-test-infrastructure/tree/production/site-modules/profile/manifests/choria

hello! i'd like to propose the following PR for your review: https://github.com/choria-io/go-choria/pull/2282 we've hit a situation with an overloaded choria broker during a "reconnection storm" and increasing the connection timeout on the choria servers - along with tls_timeout and auth_timeout on the broker - resolved the issue.

Hmm, how many nodes are reconnecting? They should have random reconnect delays and expo backoff to help already. Do you have big RSA keys maybe? Like 4K or similar?

6K nodes. 2K keys. broker running on an aws ec2 instance (t3a.medium). i could see in the logs the backoff algorithm running. but in the environment i have, that wasn't enough. i went ahead, built a choria package with the connect timeout option, installed in a choria server, and that server could connect to the broker. these are the sort of messages i see in the broker during the reconnection storm:

{
  "component": "network_broker",
  "level": "error",
  "msg": "redact:41534 - cid:29307 - TLS handshake timeout",
  "time": "2025-10-15T06:24:23-05:00"
}

and this is what i see in the server:

{"component":"server","connection":"redact","identity":"redact","level":"info","msg":"Sleeping 3.062s till the next reconnection attempt","time":"2025-10-14T11:53:50-05:00"}
{"component":"server","connection":"redact","identity":"redact","level":"error","msg":"NATS client encountered an error: EOF","time":"2025-10-14T11:53:50-05:00"}
{"component":"server","connection":"redact","identity":"redact","level":"info","msg":"Sleeping 8.839s till the next reconnection attempt","time":"2025-10-14T11:54:08-05:00"}
{"component":"server","connection":"redact","identity":"redact","level":"error","msg":"NATS client encountered an error: EOF","time":"2025-10-14T11:54:08-05:00"}
{"component":"server","connection":"redact","identity":"redact","level":"info","msg":"Sleeping 4.7s till the next reconnection attempt","time":"2025-10-14T11:54:24-05:00"}
{"component":"server","connection":"redact","identity":"redact","level":"error","msg":"NATS client encountered an error: EOF","time":"2025-10-14T11:54:24-05:00"}
{"component":"server","connection":"redact","identity":"redact","level":"info","msg":"Sleeping 10.622s till the next reconnection attempt","time":"2025-10-14T11:54:38-05:00"}
{"component":"server","connection":"redact","identity":"redact","level":"error","msg":"NATS client encountered an error: read tcp redact:57536-\u003eredact:4222: read: connection reset by peer","time":"2025-10-14T11:54:38-05:00"}

Yeah sounds like typical RSA certs that are very heavy. Not sure if Puppet can make different algo certs yet but that would really help. It’s one broker or a cluster? 6k nodes really not that bad.

Oh those are really crappy instances. With 2k key size certs you should probably go for something bigger. More CPUs would help as the TLS will use that well.

Can add another option but obviously prefer to identify the issue cos that’s not normal

Puppet has an option for elliptic curves, but that's rarely used. Worth a try

it's one broker only. in practical terms, increasing the instance size might tackle the problem. but why giving aws more money if a configuration option would also handle it? 😛 under usual load, the broker is performing fine, responses from the fleet are processed quick enough. so increasing the instance for a situation that does not happen often does not seem ideal. and if i can go a little further, conceptually it seems we really need the connect timeout. since we are providing a way to tune the timers on the broker side (tls_timeout/auth_timeout), we'd also need a way to reflect that on the server side. i hope you agree 🙂

Yeah. Will review next week am ooo atm

oh, no problem. enjoy your time off. and thank you so much for being open to discuss/review this. really appreciate it.

Hey Guys, so I just build a custom choria module using rpm, its 0.29v. I configured broker using broker.conf file and on another server I configured server using server.conf and client.conf. It works, when i did

choria ping

it returned me the nodes hostname. Now I want to get facts like ip addresses and stuff but it returns

null

I also added this

plugin.choria.facts_source = facter

plugin.choria.facts_file = /etc/choria/facts.json

in server.conf file but it still not getting the facts although it return the IP address when I do this

sudo jq -r '.ipaddress' /etc/choria/facts.json

192.168.20.30

no one??

Puppet would have configured the correct items for you

how did you install it?

I am not using puppet module, I installed choria using yum rpm and configuring it manually

support here is for puppet installation method only

so actually I was already using mcollective agent package, puppet and service and its already configured, now as Its EOL and i want to move to choria. So what will be the best approach to move without disturbing old configs and Infra? I did tried to use and install choria using puppet but it also tried to replace my configured mcollective (which I dont want) thats why I moved to custom module.

use the choria puppet modules

you cannot migrate without downtime, its a new thing

not protocol compatible, cant use the same server etc

hence new name etc, compatibility is that old client code written in ruby will keep working, old agents will keep working, but apart from that its a new thing, new configs, new protocol, new security new everything

hello, guys. is there sort of a date for a new choria release? not trying to bother, just want to plan the deploy of the newest code, and want to understand if i should wait for the official release 🙂

i dont have immediate plans to do a release

This channel is migrating to the Vox Pupuli slack, please join using https://short.voxpupu.li/puppetcommunity_slack_signup Responses here will be limited

set the channel topic: https://choria.io — This channel is migrating to the Vox Pupuli slack, please join using https://short.voxpupu.li/puppetcommunity_slack_signup

@Michael Harp has left the channel