This message was deleted.
# puppets
Slackbot
05/25/2023, 6:31 AMThis message was deleted.
b
bastelfreak
05/25/2023, 6:33 AMwhich issues do you have with git or with yaml?
w
Willem Basson
05/25/2023, 6:57 AMthe manually editing part of it š
Willem Basson
05/25/2023, 6:59 AMfor instance, over the next 6 to 12 weeks, we will be upgrading around 3000 hosts to use mariadb 10.5, rather than 10.4.
which means I will have to create about 2000 yaml files with specific data in it, update about a 1000 (for the ones that already have fqdn settings. and at some point, change my defaults and remove the entries and the files.
Willem Basson
05/25/2023, 6:59 AMbecomes pretty tedious and error prone for a human to do that
Willem Basson
05/25/2023, 7:04 AMi do like that it is easy to audit how values change, though
b
bastelfreak
05/25/2023, 7:17 AMcant you group those nodes?
bastelfreak
05/25/2023, 7:17 AMand assign the hiera data based on the group
bastelfreak
05/25/2023, 7:18 AMof course you can use different backends like vault, but if you set this on a per server level, a different backend won't help you
y
Yury Bushmelev
05/25/2023, 7:22 AMOr you may use another facts to split large fleet by groups.. we were using DC, country and env (qa/prod) to do smaller changes
Yury Bushmelev
05/25/2023, 7:23 AMe.g. first candidate usually was QA machines in one of Singapore DCs because it was smallest subset
r
ramindk
05/25/2023, 4:24 PMIn cases like these we're often done big array of machines
Copy code
$is_105 = [
'fqdn1',
'fqdn2',
] Copy code
profile::maria::server::package: 'installed'n
natemccurdy
05/25/2023, 4:44 PMI'll clarify what we're all sort of getting at: This isn't a Hiera backend problem. This is a process problem.
It sounds like you need more ways to group nodes into smaller chunks (assuming you want to do the upgrade with Puppet rather than outside of Puppet like what ramindk mentioned). Since the servers all have the same role, you don't want to use that info since it's too broad. And grouping by FQDN is tedious because it's too fined grained.
As Yury mentioned, a popular middle ground is grouping by geography, tenant, tier, and/or specialization.
Normally that'd be done with facts or some bits of data set by an external node classifier (ENC). And you'd add a layer to your hierarchy that uses that middle-ground data to progress through the upgrade.
š 3
šš» 1
r
ramindk
05/25/2023, 4:48 PMNate always following up with these banger summations. magic
w
Willem Basson
05/29/2023, 10:40 AMThanks, all of that helps. Let me elaborate on why it only kind of helps š
We use the mariadb_version to determine what to put into the apt_source for mariadb. So we can't just go with 'installed' although that would have been very nice, because, like you said: new machines would get the new version from the start.
grouping by geolocation or domain has worked for us up until a point, but our largest datacentre has 50 percent of our fleet.
I wonder if we can go with a static custom fact where we give servers an upgrade batch number between say, 1 and 10, generated by fqdn_rand.
roughly 10 percent of the fleet would be in each batch number, and we use that to group.
ie, this week upggrade_batch_0 gets the upgrade, next week upgrade_batch_1, etc
Willem Basson
05/29/2023, 10:42 AMi've also been playing with the idea of adding some kind of fact like 'what version of the golden image did you start your life on'.
then we can say 'if you are version 10 or above, you get mariadb10.5, because we want you start off with that version.
or something like that
y
Yury Bushmelev
05/29/2023, 10:54 AMIād go with batch number fact, why not
Yury Bushmelev
05/29/2023, 10:55 AMActually you may use some hash function against an ip or a hostname to put your host in a bucket
w
Willem Basson
05/29/2023, 12:36 PMthanks for the oppurtunity to bounce some ideas, guys