This message was deleted.

o/

only have a primary in this setup

there are a few different timeouts. for the whole deployment, for fetching modules from the forge, for syncing them locally, I'm not sure which timeout you hit

yeah me neither... our timeout_sync was increased from 300 -> 600 already so that shouldn't be it.. but normally doing a

puppet code deploy

takes like 5 min... but we are seeing runs taking 12 minutes...

okay that's quite slow

I have like 29 repos... yielding 166 environments 😞

lol

yikes, man.

it's like 3 hosts per environment... not real crazy... think our total node count is like 300

I was just thinking about all the environments. that’s still a lot. No telling what the compiler is doing to itself to parse and service all of those environments.

yeah we used to have split setup with 3 compilers... but since each primary should be able to handle like 4,000 hosts when we moved back to all-in-one we got rid of the compilers... might be useful to bring them back ... just not sure if "syncing" across compilers would now cause more or less of issue

My guess would be more. You have all the work the primary currently is doing, plus then having to distribute and sync them to the compilers.

you could have multiple primaries and only deploy some envs on each

Yeah so it's probably best not to add them... only thing that it would improve is right now when we make changes like add a project to r10k... it's done via hiera and requires

pe-puppetserver

to restart ... with compilers that wouldn't cause agent issues

do you have modules that have the same version in each env? you could put them in a shared module dir

yes tons of the environments will have the same core modules... some might be different versions though... but there are tons that use stdlib, firewall, snmp, etc... so there is def overlap

meh

a branch gets cut for a release in dev, qa, prod and the previous one gets destroyed and then deleted

if I am understanding what you are saying correctly, that would be about the worst possible design performance-wise for how I understand Code Manager works

@bastelfreak couldn't even break apart environments on different primaries based on dev,qa,prod because it still would require all the same repos to be pulled in from r10k... just limits the use of which branches are used

limiting the number of branches/environments for a given primary probably would help

so it's be having to break out the primaries based on "project" which having 29 primaries is crazy... and trying to figure out which to combine together ... is just an unarbitrary mess...

maybe engage PSE again

or throw puppet out the window 😉

which PE version is that?

The reason we aren't using shared repos for projects was because not wanting to break other projects with updates to "shared modules"... so putting shared modules in another repo probably wouldn't be ideal because that was the original reason why the per project repos were created. Everyone can agree that the resulting number of environments isn't "ideal" but based on the requirements for separation no one has any way to improve the layout 😉 ... just loves to talk about how bad it is 😛

there are some tuning options in r10k and puppetserver, there might be room for improvement. and maybe consider at least two independet setups? that would reduce envrionments per server by half

yeah there's just no delineation for breaking into 2 or more primaries