Puppet Community

Hello, I am trying to understand how lvm module works in Puppet? I have a requirement to expand one of the partitions and don't see the partition table for device /dev/sdc. Usually, when I manually create something using fdisk/parted, it will create it as /dev/sdc1 and then I create pv/vg and lv etc... Why in the lvm module example, it was mentioned /dev/sdc? Also if I wanted to expand the file system looking for your recommendation on how I can proceed?

```[root@testsrv~]# fdisk -l /dev/sdc

Disk /dev/sdc: 322.1 GB, 322122547200 bytes, 629145600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

[root@testsrv~]#

[root@testsrv ~]# pvs
  PV         VG        Fmt  Attr PSize   PFree
  /dev/sdc   spacewalk lvm2 a--  250.00g    0```

It's quote normal to not have partition on non-boot disk

that's why you use the whole device as your pv

```lvm::volume_group { 'spacewalk':
    physical_volumes =&gt; '/dev/sdc',
  }```

Okay... what if we need to expand? Any alternate?

yes. I have expanded the disk to 300GB, but in LVM its showing only 250GB

you need to re-scan your scsi bus, probably and then pvresize

nothing to do with puppet though, imho. it's too low level

thanks. that's it, pvresize worked. Thanks <@U2S1MUUQ7>.

`lvm::logical_volume` can increase FS if space is available

<https://fosstodon.org/@puppet/110675302616305141|https://fosstodon.org/@puppet/110675302616305141>

`$x.keys.sort.each |$a| { notice($x[$a]) }` &lt;- this is the best I can do right now

you could implement a sort_keys function if you like

by default `sort` method sorts by `keys` .
whatever you have written is appropriate.
Another way you can write is `$x.sort.each{|k,v|  notice($x[k]) }`

Also avoid using `$`  unless you intend for having that variable accessible *globally.*

I was asking about Puppet DSL, not Ruby..

I believe it was this one: <https://forge.puppet.com/modules/thekevjames/homebrew>

Though it was not updated for few years already.. I’d check forks: <https://github.com/TheKevJames/puppet-homebrew/network/members>

last time I saw ppl using this module they were keeping an internal fork with local changes

yep, they haven't had a release for awhile, you should use git repo to install

or use Brewfile + exec (as suggested by <@U2S1MUUQ7> while ago)

image.png

did you configure puppetdb to use the postgres db? is the db up and reachable?

```database.ini

[database]
# The database address, i.e. //HOST:PORT/DATABASE_NAME
 subname = //puppet:5432/puppetdb?ssl=true&amp;sslfactory=org.postgresql.ssl.LibPQFactory&amp;sslmode=verify-full&amp;sslrootcert=/etc/puppetlabs/puppetdb/ssl/ca.pem
# subname = //puppet:5432/puppetdb?ssl=true&amp;sslfactory=org.postgresql.ssl.NonValidatingFactory

# Connect as a specific user
 username = xyz

# Use a specific password
 password = zxy
# How often (in minutes) to compact the database
# gc-interval = 60


#[read-database]
#subname = //10.201.0.16:5432/puppetdb
#username = puppetdb
#password = admin

#[jetty]
#host = 0.0.0.0
#port = 8080```

```jetty.ini

# IP address or hostname to listen for clear-text HTTP. To avoid resolution
# issues, IP addresses are recommended over hostnames.
# Default is `localhost`.
# host = &lt;host&gt;

# Port to listen on for clear-text HTTP.
#port = 8080

# The following are SSL specific settings. They can be configured
# automatically with the tool `puppetdb ssl-setup`, which is normally
# ran during package installation.

# IP address to listen on for HTTPS connections. Hostnames can also be used
# but are not recommended to avoid DNS resolution issues. To listen on all
# interfaces, use `0.0.0.0`.
ssl-host = 10.201.0.16

# The port to listen on for HTTPS connections
ssl-port = 8081

# Private key path
ssl-key = /etc/puppetlabs/puppetdb/ssl/private.pem

# Public certificate path
ssl-cert = /etc/puppetlabs/puppetdb/ssl/public.pem

# Certificate authority path
ssl-ca-cert = /etc/puppetlabs/puppetdb/ssl/ca.pem

# Access logging configuration path. To turn off access logging
# comment out the line with `access-log-config=...`
access-log-config = /etc/puppetlabs/puppetdb/request-logging.xml
client-auth = want```

```puppetdb.conf

[main]
server_urls = <https://puppetdb:8081>
server_url_timeout = 10
soft_write_failure = false```

```puppet.conf

[server]
vardir = /opt/puppetlabs/server/data/puppetserver
logdir = /var/log/puppetlabs/puppetserver
rundir = /var/run/puppetlabs/puppetserver
pidfile = /var/run/puppetlabs/puppetserver/puppetserver.pid
codedir = /etc/puppetlabs/code
fileserver = true
storeconfigs = true
storeconfigs_backend = puppetdb
reports = store,puppetdb
autosign = true```

and the puppetdb hostname resolves properly?

yea so bacsically puppet and puppetdb are on the same server and puppetdb is resolved

and does puppetdb run on the public interface?

then it womt work if puppetdb resolves to the public imterface ip :)

if both services run on localhost, I recommend using 127.0.0.1

then you wont have potential external traffic and no dns resolution

both services means postgres and puppetdb?

I meant traffic between puppetdb and postgresql

thanks a lot the puppetdb is connected to postgres

do you use puppet modules to configure puppetserver/puppetdb. Seems logical to use puppet to configure puppet. theforeman/puppet is a good module, it uses puppetlabs/puppetdb and puppetlabs/postgresql

well my use case here is using puppet to copy the files and install all the python packages as well as ubuntu dependencies on the agent nodes. Heard that puppetdb can help with that . Do you think this approch is good?

I dont see how puppetdb helps. thats just a DB that stores reports, catalogs and facts

So how do you think i should achieve this, i didn’t find any solution that can install all the python packages easily . I have to give a list of packaged manually and then the agent would install it.

puppet can install packages via different package managers by default, ine of them is apt/dplg

also <https://forge.puppet.com/modules/puppet/python/readme|https://forge.puppet.com/modules/puppet/python/readme> is good for managing python venvs

and the puppet package resource supports pip by default <https://www.puppet.com/docs/puppet/7/types/package.html#package-provider-pip|https://www.puppet.com/docs/puppet/7/types/package.html#package-provider-pip>

Yup i have tried the installation with package manager as well as pip. My issue is i have to manually liat all the packages in manifest to install them. I had also tries to use the exec to run a custom command and install rhe packages but that is not what i want

do the packages have dependencies to each other?

then puppet will resolve those and install all deps

or if that's a python project, use a requirements.txt

Yea my last option is requirements.txt . My main issue is with python packages only ubuntu packages are not my main priority for now

somthing has to define the deps. You've three options:
• you define them
• your packages define them (or you build a meta package that pulls in all others)
• use a requirements.txt

tried using the python module facing this issue.

it will only install the pips when forceupdate is true or when the file resource for the requitements.txt receives a refresh

how will it know when we update the requirements fiel?

so puppet knows it when puppet updates it

then you just need to tell puppet that it should refresh the pyhton::requirements resource when the file resource changes

what I do for python projects:
• put them in a git repo
• use the vcsrepo resource to deploy the git repo to a server
• let the vcsrepo notify the python::requirements resource

i did the setup with vcsrepo on thing in noticed is that it doesn't automatically pull on agent server after there is a  push/commit in a branch in repo. Is there a way to do that

did you check the documentation for the vcsrepo resource

yeah i used ensure to latest. But how does that work in this

does it check for commits in given branch or is there any other way it .

ok then i don't have to set a run interval for puppet agent to check every minute for commit

i used it with latest it's not pulling the lates changes on agent server .

can you paste your code here and show the agent output with --debug

```include sparkpyrequirements
class { 'python':
  version    =&gt; 'system',
  pip        =&gt; 'present',
}


file {'testing':
  path =&gt; '/home/uat/xyz',
  ensure =&gt; 'directory',
  source =&gt; 'puppet:///testing/',
  recurse =&gt; 'remote'

}

#package {'python3-pip':
#  name =&gt; 'python3-pip',
#  ensure =&gt; 'installed',
#  provider =&gt; 'apt',
#  before =&gt; Exec['list_python_packages']
#}

#$package_list = exec { 'list_python_packages':
#  command  =&gt; 'pip3 freeze | cut -d "=" -f 1',
#  path     =&gt; ['/usr/bin'],
#  logoutput =&gt; false,
#  returns  =&gt; [0],
#  before =&gt; Package['install packages']
#}

#$my_packages = "$package_list"
#$packages = split($my_packages, "\n")


#$packages.each |$package| {
#  package { $package:
#    ensure   =&gt; installed,
#    provider =&gt; 'pip3',
#  }
#}

vcsrepo { 'cloning repo':
  path =&gt; '/home/uat/manu2',
  ensure   =&gt; latest,
  provider =&gt; git,
  source   =&gt; 'git@github.com:jay-test-org/manu2.git',
  remote =&gt; 'origin',
  branch =&gt; 'main',
  safe_directory =&gt; true,
  user =&gt; 'uat',
  identity =&gt; '~/.ssh/id_rsa',
  notify =&gt; Class['sparkpyrequirements']
}

class sparkpyrequirements {
  python::requirements { '/home/uat/manu2/requirements.txt' :
    pip_provider =&gt; 'pip3',
    manage_requirements =&gt; false,
    forceupdate =&gt; true
#  virtualenv =&gt; 'system',
#  owner      =&gt; 'root',
#  group      =&gt; 'uat',
  }
}```


I did a commit at 2:14 there is log for execution at that time after that i did multiple commit s nothing is happening

if i run the agent now it will do the changes

now it updated to main version but i had to do it mannually by running the agent

I'm not sure what kind of logoutput you shared in the first screenshot, this isn't the output of a whole puppet run

not the debug output, the screenshot before. that looks like a filtered puppet log

yea that was just to show you that it ran once after a commit and not after that

You understand my issue right, what i m trying to achieve is that whenever there is a commit in specified branch the agent should automatically sync there branches. without trigerring them to run manually or run them at given interveal

It cannot magically know that you changed a git repo and do things

If you want to deploy changes instantly, you need to configure a hook in your git server

Hey guys, does someone know how to set a fact as default value  for a custom type (type/provider pattern). I know how to do it for a defined type but i am stumped about how to do it for the provider pattern.

Hmm, I'm not sure what to offer on that one --- it's been many, many years since I looked at ParsedFile. I'd have to re-learn it from scratch at this point and I don't have the free time for that.

Maybe take a look at the cron type? That uses ParsedFile and has logic for handling multiple crontabs.

Does the problem still recur if you pass `name =&gt; $key,` in _addition_ to `key =&gt; $key,`?

I don't mean that as a possible solution, but it could be an interesting test.

It's weird to me because it looks like all you are doing in that specific PR is changing the type and provider to use a new parameter `key` instead of `name`. This looks as if maybe something is still considering `name` to be important.

I didnt understand the whole ParsedFile implementation but my impression is that name is a reserved keyword/is special

Maybe this:
<https://github.com/puppetlabs/puppet/blob/3170daa/lib/puppet/provider/parsedfile.rb#L218>
Looks like:
1. `self.prefetch` calls `match_providers_with_resources`
2. ...which uses `resource_for_record`
3. ...which seems to expect records to have a `:name` field. 

<@U11MDJNGZ> Thanks for the hint with the cron type/provider, I will have a look at this one

<@U03ULG4CKKK> good catch, lets see if the cron type/provider gives some insights on that

<@U03ULG4CKKK> I tried your recommendation, the result is exactly the same.
<@U11MDJNGZ> The cron type comes with a local filetype.rb, will try to understand the code.

I'm not surprised, after looking at the `resource_for_record` method.

The cron provider overrides it--I'm not sure if you saw:
<https://github.com/puppetlabs/puppetlabs-cron_core/blob/dd42efd/lib/puppet/provider/cron/crontab.rb#L110>

...but that method wraps _around_ the original method to filter out unwanted matches, whereas in your case I think the original method is not even finding the matches at all.

It's tricky for me; I'm not experienced with this. I can suggest what I would try, though, a bit later (some time conflicts right now).

I appreciate this. You seem to have much more understanding what the code used by the provider is doing than I am. 

Looking at it from a high level, to state and make sure I understand the problem:
1. You can't use `name` to identify the configuration key anymore, since multiple files will specify the same configuration key.
2. To solve this at a high level, you can make `name` be an arbitrary unique string and pass the `key` as a `key` parameter.
3. Unfortunately, the underlying "parsedfile" provider implementation relies on a field called `name` for its own internal identification, and `name` must be unique (presumably within each provider, at least).

Brainstorming options:
• Define methods to override parsedfile internals to use a combination of `target` and `key` for resource storage/lookup/uniqueness. This doesn't look very viable to me--the use of `name` looks quite entrenched in several long methods.
• Add some code to (somehow) map `key` to `name` within the `postgresql_conf` provider, so that the parsedfile could continue to use `name`. I don't think this would end up working, though, since parsedfile appears to be written to prefetch all records from all targets, and `name` collisions would happen.

Ideally, this could be done without modifying (or locally re-implementing) parsedfile, but I don't know what else is possible.

<@U112YGDUM>! now that I saw you are online, do you maybe have 10min to read through the backlog here and share your thoughts?

From a design perspective, I think the main limiting factor is that parsedfile stores records within a list instance variable `@records`. I think that if `@records` were instead a hash keyed by target, that would open up more possibilities.

Again, I'm not an expert on this; I've only read the code and I haven't added instrumentation or tested anything.

<@U03ULG4CKKK> thanks for pulling this apart! I will try to follow your ideas tomorrow 

I think I'm correct that parsedfile objects are shared among all targets within each provider. I added a bit of instrumentation:
```--- /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/parsedfile.rb.orig	2023-07-11 11:55:13.025013515 -0700
+++ /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/parsedfile.rb	2023-07-11 11:55:24.157185504 -0700
@@ -270,9 +270,12 @@
 
   def self.prefetch_all_targets(resources)
     records = []
+    puts("start prefetch loop")
     targets(resources).each do |target|
+      puts("prefetching #{target} for #{@resource_type.name} object id #{object_id}\n")
       records += prefetch_target(target)
     end
+    puts("end prefetch loop")
     records
   end
 ```
...and on a test host with our local puppet codebase, that prints:
```start prefetch loop
prefetching root for cron object id 109660
prefetching nobody for cron object id 109660
end prefetch loop
start prefetch loop
prefetching /etc/fstab for mount object id 109820
end prefetch loop```
Note how the two `cron` resources are shared and the `mount` resource is separate from those.

You took a hit because the bot is always doing this with - - - in code blocks 

Having different targets share the same parsedfile object seems strange and counter intuitive to me. I can understand why different resources of the same target _must_ share the object, but I don't see the point of different targets of the same resource type being shared. Maybe it's for systems which use a collection of config files, like nagios.
<https://github.com/puppetlabs-toy-chest/puppetlabs-nagios_core/blob/master/REFERENCE.md>

Maybe there is some way to make `postgresql_conf` _not_ share objects between different targets. That would solve the uniqueness issues. Then hopefully the only thing left would be to find a way to map `key` at the high level to `name` within parsedfile.

there are maybe some instance variables we could turn into local ones

Wow, that got much more complex than I ever expected 

I think we scared <@U112YGDUM> away with this :D

The sharing is likely required to get correct behavior out of multiple resources that operate on the same file.

ah yeah,  but there should be no sharing across different targets

Right, but the base ParsedFile assumes a singular target is in use. That's the core issue.

there are multiple types around that inherit ParsedFile. So it is somehow possible to have different targets? just not per type?

for postgres the regex to parse a line in three blocks is quite simple, maybe we should replace it with something that does not inherit ParsedFile

&gt;  Right, but the base ParsedFile assumes a singular target is in use. That's the core issue.
If I understand correctly, it does work with multiple targets, but it puts all records for all targets of a given provider in the same namespace with respect to `name`. This would be fine, since at a high level, all resources of a given type must have a unique `name`. The trouble is from how ParsedFile also requires a field to be named `name` and then uses that field as the per-record `name` .

There does need to be some kind of a per-record key, though--that part makes sense on its own; the file parsing doesn't know anything about the names of resources (which may or may not exist). This is why I am suggesting two maybe-possible approaches to maintain the internal uniqueness while still allowing the same key across multiple targets; either:
• make `@records` be a hash keyed by target
• make `postgresql_conf` not share ParsedFile objects between multiple targets

Yeah this is the sort of thing I think I knew at one point, but it's all "I have no memory of this place" :male_mage: now.

Aye. If you would have asked me about ParsedFile in 2015, I could have given some helpful pointers...

...but that was a long time ago. ParsedFile has not come up in quite a while.

maybe another reason to replace it with a smaller postgres conf specific type/provider

Some of the guts of ParsedFile come from Puppet::Util::FileParsing, so if you do your own implementation, you might still be able to re-use some code.

One thing I would be wary of would be making changes that cause the config files to be generated in a different order. As a downstream user of the postgreql puppet module, I would find that distasteful, unless it was a one-time unavoidable change to make all parameters be in alphabetical order.

You mean parsing the file and resort all active values in alphabetical order if they are not already?

Yes. I mean, I'm not recommending that, I just mean that from a user standpoint:
• I see that the current code apparently allows configuration parameters to exist in the file in arbitrary positions. The current code also preserves comments.
• I'd be annoyed if upgrading the puppet module resulted in configuration generated in a different order, even if the overall data turned out to be the same. I'd have to validate the output, apply the change, and then restart (or reload?) servers to make sure I don't have any latent bugs.
• ...that said, I don't at all mind puppet modules that generate configuration without comments and either preserve order from hashes in hiera or alphabetize the configuration keys. I'd consider the change to be an acceptable annoyance if the end result were something canonical and not subject to further such changes.

I created a new provider, but it still has a few issues: <https://puppetcommunity.slack.com/archives/C0W298S9G/p1691065378523349>

why do you want to switch to r10k when you have a working tool?

if you want to replace your git submodules you can list them in a Puppetfile and install them via r10k

the latter - replacing git submodules seems like a better pattern at this point.  It is becoming a nuisance to revisit every module, clone it to locally, push to internal vcs,  think its time to allow puppet component to talk to internet for sanity sake.

or tell your git server to mirror upstream modules

I think it is too.  but how does that fit into things.  is it just put a puppetfile into repo root, and run the r10k puppetfile install/check/purge?\

when you tell r10k the git repo, you csn use stuff like r10k deploy environment $branchnamr

and it will deploy that branch + all modules from the Puppetfile from that repo

and theres also a webhook you could use to r10k automatically when changea appear in git

does r10k persist say cloning a forge module into say git submodules (and check into localvcs), or each time a branch/env is created it goes and fetches everything?

i guess that is fine.  the likelihood someone deletes an upstream repo is prob really low.

thank you for your clarifications.  this sounds like a viable first course of action.  i think puppetfile will work, and i will be able to trigger it easily.

no problem. just ask if you have other question

There is g10k also which is faster but not 100% compatible.. worth to try 

Usually <@UPVF2CKRP> is the person to ask

Q3 was promised.. I’m waiting for arm64 agent too

Arm64 agent work is on the Q3 roadmap.  That probably means it will hit OSP in Q4.  I'm just back from two weeks PTO and haven't seen the Q3 roadmaps yet to see if there's any updates.

Jey Edwin, will there be any changes to the forge that could be interesting to us users?

<@U1PC2RAH0> I would say the main benefit for users will be :sparkles:  Reliability :sparkles: there are no changes on the frontend.

There are a lot of improvements to the Forge infrastructure :slightly_smiling_face: Basically making it more resilient :muscle:

Yeah, everything should be exactly the same, functionality-wise. If you see something afterward, be sure to let us know! We changed a lot in the infra to make it more performant/scalable and reliable

Theres nothing currently on the road map.

<@U01MXBB09RC> I'm not sure if this is something we can make more visible in our docker cleanup ticket for images not maintained

:businessparrot: _Content and Tooling_ is about to start up in <#CFD8Z9A4T|>

:the_more_you_know: _:fox_face:Vox Pupuli monthly sync; see calendar event for info_ is about to start up in <#CFD8Z9A4T|>

does anyone know if this works:
```if (a){
 $my_var = undef
}else{
 $my_var = 'myuser'
}

        File { '/usr/local/myfolder' :
          ensure =&gt; 'directory',
          owner  =&gt; $my_var,
          group  =&gt; $my_var,
        }```


I imagine the behavior would be type/provider dependent, but in this case it looks like it does what you're trying to do.

Can test it outside a class if that helps: `puppet apply -e 'file { "/usr/local/myfolder": ensure =&gt; directory, owner =&gt; undef, group =&gt; undef, }'`

confirmed, setting user/group to undef will cause puppet to not manage the state of that parameter

That feature was added in the PE 2023 release:

<https://www.puppet.com/docs/pe/2023.2/release_notes_pe.html#release_notes_pe_x-0-new-features-pe-x-0>

Thanks, right now we are not in 2023, but the task fail after 5 minutes running

In that case, it's likely a command the task is running has timeout logic of its own.

That's my concern I will reach the task owner thanks

:allthethings: _:teacher: Bolt_ is about to start up in <#CFD8Z9A4T|>