https://www.puppet.com/community logo
Title
w

Willem Basson

05/25/2023, 6:31 AM
When you have a large fleet of hosts with the same role, but you need wide variance between instances, what would be a better hieradata backend than a git repo with yaml files?
b

bastelfreak

05/25/2023, 6:33 AM
which issues do you have with git or with yaml?
w

Willem Basson

05/25/2023, 6:57 AM
the manually editing part of it šŸ™‚
for instance, over the next 6 to 12 weeks, we will be upgrading around 3000 hosts to use mariadb 10.5, rather than 10.4. which means I will have to create about 2000 yaml files with specific data in it, update about a 1000 (for the ones that already have fqdn settings. and at some point, change my defaults and remove the entries and the files.
becomes pretty tedious and error prone for a human to do that
i do like that it is easy to audit how values change, though
b

bastelfreak

05/25/2023, 7:17 AM
cant you group those nodes?
and assign the hiera data based on the group
of course you can use different backends like vault, but if you set this on a per server level, a different backend won't help you
y

Yury Bushmelev

05/25/2023, 7:22 AM
Or you may use another facts to split large fleet by groups.. we were using DC, country and env (qa/prod) to do smaller changes
e.g. first candidate usually was QA machines in one of Singapore DCs because it was smallest subset
r

ramindk

05/25/2023, 4:24 PM
In cases like these we're often done big array of machines
$is_105 = [
  'fqdn1',
  'fqdn2',
]
The other option is to pull version out and assume that the upgrade will be handled outside of Puppet.
profile::maria::server::package: 'installed'
This works fairly well as long as the upgrade window is small, couple of weeks at most. New servers get latest version, current servers don't upgrade, and Puppet doesn't fight with external upgrade process.
n

natemccurdy

05/25/2023, 4:44 PM
I'll clarify what we're all sort of getting at: This isn't a Hiera backend problem. This is a process problem. It sounds like you need more ways to group nodes into smaller chunks (assuming you want to do the upgrade with Puppet rather than outside of Puppet like what ramindk mentioned). Since the servers all have the same role, you don't want to use that info since it's too broad. And grouping by FQDN is tedious because it's too fined grained. As Yury mentioned, a popular middle ground is grouping by geography, tenant, tier, and/or specialization. Normally that'd be done with facts or some bits of data set by an external node classifier (ENC). And you'd add a layer to your hierarchy that uses that middle-ground data to progress through the upgrade.
šŸ‘ 3
šŸ‘šŸ» 1
r

ramindk

05/25/2023, 4:48 PM
Nate always following up with these banger summations. :magic:
w

Willem Basson

05/29/2023, 10:40 AM
Thanks, all of that helps. Let me elaborate on why it only kind of helps šŸ™‚ We use the mariadb_version to determine what to put into the apt_source for mariadb. So we can't just go with 'installed' although that would have been very nice, because, like you said: new machines would get the new version from the start. grouping by geolocation or domain has worked for us up until a point, but our largest datacentre has 50 percent of our fleet. I wonder if we can go with a static custom fact where we give servers an upgrade batch number between say, 1 and 10, generated by fqdn_rand. roughly 10 percent of the fleet would be in each batch number, and we use that to group. ie, this week upggrade_batch_0 gets the upgrade, next week upgrade_batch_1, etc
i've also been playing with the idea of adding some kind of fact like 'what version of the golden image did you start your life on'. then we can say 'if you are version 10 or above, you get mariadb10.5, because we want you start off with that version. or something like that
y

Yury Bushmelev

05/29/2023, 10:54 AM
I’d go with batch number fact, why not
Actually you may use some hash function against an ip or a hostname to put your host in a bucket
w

Willem Basson

05/29/2023, 12:36 PM
thanks for the oppurtunity to bounce some ideas, guys