Puppet Community

is there an easy way to convert a puppet hash into a json var ?
like:
  $my_hash = {
    "key1" =&gt; "value1",
    "key2" =&gt; "value2",
    "key3" =&gt; "value3"
  }

  $my_json = json($my_hash)

yes, in stdlib there's a to_json function

See the `puppetlabs-stdlib` module at <https://forge.puppet.com/puppetlabs/stdlib?src=slack&amp;channel=puppet>

excellent!  guess my searching foo evaded me here

they're my favorite 2 functions in stdlib

a module is just a .tar.gz file. you can download it and copy somehow into your environment

and most modules come from a git repo, you can mirror those as well ans use in your Puppetfile

i thought dsc modules dont have backing git repos

thats possible. I didnt check it. but it wont block you from downloading the forge release, untar, throw in a git reo

I would assume so since you can use them in bolt too. I don’t have much experience with that though

bolt also just uses the forge releases by default

When using <https://www.puppet.com/docs/puppet/8/create_types_and_providers_resource_api#create_types_and_providers_resource_api|custom resource API> , is it possible to read a parameter value from inside `get` method?
I need to limit the `get` method to retrieve a single resource - as opposed to all resources from the custom type. Other resources aren't needed, and querying all of them is quite costly.
I can do it in the <https://www.puppet.com/docs/puppet/8/types_and_providers_method|low level method>, but wanted to make sure it isn't overkill.

I RTFM and STFW to exaustion, but couldn't find the slightest hint on that...

Do you have custom facts configured for your agent’s facter? Looks like it’s trying to load those facts (execute scrip in from the facter directory. In linux it’s `/etc/facter/facts.d/`

thanks for your answer <@U05MAESTM61> yes I have custom facters and it work on all other Agents except this one. its unfortunately windows agent. I clear the whole `cache folder` and run `puppet agent -t`  again it still hang there...

AFAIK there are two type of custom facts (on linux): one is the module related facts `..cache/lib/facter/` (as I can see they run well). And another facts are in the different folder (pure facter configuration on the host). Can you check both?

Thats good idea. I delete the one in Cache. I'll check it.

in windows both are in  cache:  `cache/facts.d/`  and `cache/lib/facter` and as I told you, I remove the complete cache folder.. and when I run `puppet agent -t` its hanging at same place Loading external facts

I would also check you `facter` configuration. On linux you can run `facter -l debug` to see how facts are loading. As I can see location of external facts is configurable for facter.
Also here is output from linux:
```Debug: Facter: Loading external facts
Debug: Facter: Executing command: /etc/facter/facts.d/puppet5_from_r10k
Debug: Facter: Executing command: /etc/facter/facts.d/nodeclassifier.sh
Debug: Facter: Fact file /etc/facter/facts.d/nodeclassifier.sh was parsed but no key=&gt;value data was returned
Debug: Facter: Executing command: lsb_release -a
Debug: Facter: Executing command: uname -m &amp;&amp;```

below is the output:

```C:\Windows\system32&gt;facter -l debug
[2023-08-14 11:30:27.030004 ] INFO Facter - executed with command line: -l debug
[2023-08-14 11:30:27.030907 ] DEBUG Facter - Facter version: 4.3.1
[2023-08-14 11:30:27.033576 ] DEBUG Facter::FactManager - Resolving facts sequentially
[2023-08-14 11:30:27.034483 ] DEBUG Facter::FactLoader - Loading all internal facts
[2023-08-14 11:30:27.034998 ] DEBUG Facter::FactLoader - Loading custom facts
[2023-08-14 11:30:27.042265 ] DEBUG Facter::FactLoader - Loading external facts```
again it stops at loading external facts, but when I run facter operatingsystem :
```
C:\Windows\system32&gt;facter operatingsystem
windows
C:\Windows\system32&gt;```

you can try to run `facter -l debug --no-external-facts`

Also if you are sure you do not need any external facts try to use `--external-dir` CLI param to set facter to some empty dir (for clear test)

```C:\Windows\system32&gt;facter -l debug --no-external-facts
[2023-08-14 11:36:20.210939 ] INFO Facter - executed with command line: -l debug --no-external-facts
[2023-08-14 11:36:20.211713 ] DEBUG Facter - Facter version: 4.3.1
[2023-08-14 11:36:20.214133 ] DEBUG Facter::FactManager - Resolving facts sequentially
[2023-08-14 11:36:20.214746 ] DEBUG Facter::FactLoader - Loading all internal facts
[2023-08-14 11:36:20.215219 ] DEBUG Facter::FactLoader - Loading custom facts```

unfortunately  yes, im checking to get minimal facters output if there any  arguments

and if you use `--no-custom-facts` instead of `--no-custom-facts` or together with it.

I test that, is still the same, so I will use now both

```C:\Windows\system32&gt;facter -l debug --no-custom-facts --no-external-facts
[2023-08-14 11:47:13.710549 ] INFO Facter - executed with command line: -l debug --no-custom-facts --no-external-facts
[2023-08-14 11:47:13.711653 ] DEBUG Facter - Facter version: 4.3.1
[2023-08-14 11:47:13.715180 ] DEBUG Facter::FactManager - Resolving facts sequentially
[2023-08-14 11:47:13.716405 ] DEBUG Facter::FactLoader - Loading all internal facts```
Still hangs :-(

hmm. I would use `strace` on linux now. but windows ended for me on Windows XP lol

I dont like windows too, but we have about 8 servers which use Windows...

even if I run `facter -l --debug --trace --no-custom-facts --no-external-facts --no-block --no-cache --no-ruby` I get same error.

looks like something is wrong with the server. Can you restart it (usually it helps for Win)

haha yes its best solution for windows :grin: by the way, even `puppet facts` doesnt work.. anyway thank you so much for your help :slightly_smiling_face:

it may actually be what happens AFTER the external facts load

you are right <@U02C3NLGPLG> what can I do ? do you have any suggestion ?

can you run each of the facts that exist in C:\ProgramData\PuppetLabs\facter\facts.d\ one by one to see if the individual facts can run outside of Facter manually?

Or, if you’re specifying a facts path, whatever is there.

<@U11JXDFJ9> in my case directory is empty; `C:\ProgramData\PuppetLabs\facter\facts.d\`  but in `C:\ProgramData\PuppetLabs\puppet\cache\lib\facter`  I see some facts  rb-files, can I run them  manually ? #

that lib directory is the system facter.  You don’t need to mess with that.  The official documented location for facts is C:\ProgramData\PuppetLabs\facter\facts.d\ as I mentioned before.  There is also a specification as to what a fact should return.

Let me ask you this, the directory C:\ProgramData\PuppetLabs\facter\facts.d\…  what’s the permissions on it?  Can Puppet even get to that directory on a run?

External facts:

<https://www.puppet.com/docs/puppet/8/external_facts.html?_ga=2.4283824.289986232.1692016958-82695772.1689109506&amp;_gl=1*1sbl06y*_ga*ODI2OTU3NzIuMTY4OTEwOTUwNg..*_ga_7PSYLBBJPT*MTY5MjAxNjk1OC44LjEuMTY5MjAxNjk2OS40OS4wLjA>.

BUt the directory is empty! there is nothing on it to run it! And I have full access to the directory.

Can Puppet even get to that directory on a run?  I have to check

Also, therre are some limitations to custom facts:

```While external facts provide a mostly-equal way to create variables for Puppet, they have a few drawbacks:
An external fact cannot internally reference another fact. However, due to parse order, you can reference an external fact from a Ruby fact.
External executable facts are forked instead of executed within the same process.```


Just a side note. In the Puppetserver7 is was getting warnings when the facts directory was completelly empty. Had to add there simple script with no logic.

That’s fune, but on a Puppet run if you hav external or custom facts that are destined for that directory, wht do you think Puppet is going to do?  It will attempt to access that directory to sync your facts to it.  If it cannot open the directory, it will hang.

I was about to suggest this, <@U05MAESTM61>

You see, <@UNFQYMU8L>, several of us are consultants and we’ve seen TONS of weirdness out of Puppet over the years.  We’re scratching out our memories of things we’ve encountered to strike off of the list.  If Puppet can’t access the directory, it won’t be able to place the facts there via pluginsync, and it will hang.  If there are no facts there, it might error, and require you to place a dummy script as Maxim noted. If the facts that *are* attempting to resolve cannot run on the destination platform, that could be it as well.

Everything you’re getting here is both clear, calculated, and the results of thousands of hours of experience with this ecosystem.  We’re trying to help you eliminate one potential issue at a time.

on puppetserver my custom facter is in `/etc/puppetlabs/code/environments/production/modules/jxsm/lib/facter` maybe thats why!

And if I add simple script with no logic, it should be also `.rb`?

How does that express on your other windows servers?

it works until now! all other system have that custom fact, which I use it in foreman too ehm

That was a *HUUUUUGE* piece we should’ve known from the beginning

You have to think about Puppet in a little different light by keeping it in the context of using Foreman.  It doesn’t modify Puppet or anything, but it is VERY helpful to know.  Youcouldn’t have known that.

Foreman adds environment variables, additional facts, an ENC, and some other things.  I don’t use it…. I swore off ever having to use it again.. It was just more trouble than it was worth.  I could get the ENC functions elsewhere and use other tools for visualization.  It just doesn’t offer the value it once did.

that doesn’t solve your problem though.

On one of your other Windows machines that works correctly, can you check the permissions on C:\ProgramData\PuppetLabs\facter\facts.d\ to make sure they match?  It might be a red herring, but it’s worth a try.

which visualization tool do you use?  and thanks for the helpful informations!

Also, check your Puppet log on the client system as well as the PuppetServer in regards to this system’s Puppet run

Looks like the agent logs on your  system is in the event log:

<https://www.puppet.com/docs/puppet/8/services_agent_windows.html#configuring_win_agent-logging-puppet-agent-windows-systems>

In your Puppet server, look in /var/log/puppetlabs/puppetserver/puppetserver.log

The permissions are the same.  How can I add simple dummy file to the facts directory ?

I don't Windows, so you'll need some input from one of those folks.

have a look at the logs in the Puppet server and also in your Event Viewer on the agent

1. in Server logs I dont find any errors
2. in event logs I find: 
Using environment 'production'
Retrieving pluginfacts
Retrieving pluginc
Loading facts

3. I create `test_fact.ps1`. in `C:\ProgramData\PuppetLabs\facter\facts.d\`   I'll restart the service and test it again

it doesnt help!  anyway I'll restart the server..  thats why I hate windows!

check also your time sync on that machine, DNS resolution,

Yes the time and dns works good, it works until 12.08.2023... and I think restart will solve it!

and thank you so much for your help <@U11JXDFJ9> <@U05MAESTM61>

I've just seen a lot of weirdness where time sync wasn't in place and also when DNS was intermittent or just flat wrong.

Hey, <@UNFQYMU8L> how did that reboot go?

<@U11JXDFJ9> I asked the department if we can restart it! So I cannot restart the server. When we do that I’ll give you/here my feedback.. maybe other will have the same situation. Thank you again for your help 

[root@US02-5037MR-003-02 puppetlabs]# puppet-code deploy --all
Deploying all environments.
2023/08/14 07:55:02 ERROR - [POST /deploys][500] Errors while collecting a list of environments to deploy (exit code: 1).
"ERROR\t -\u003e Unable to load sources; the supplied configuration does not define the 'sources' key\n"

Yes, but I'm not sure if my Code Manager is having any issues.

did you run puppet after setting the options? can you show your config?

Which configuration file do you need to view?

the one where you configured code manager

cat hiera.yaml
---
version: 5

defaults:
  datadir: "data"

hierarchy:
  - name: "Yaml backend"
    data_hash: yaml_data
    paths:
      - "nodes/%{trusted.certname}.yaml"
      - "common.yaml"

and did you configure code manage anywhere?

My code manager and my primary server are not on the same device.

at least that's not supported. how did you configure that?

I followed these steps to operate:
<https://www.puppet.com/docs/pe/2023.2/add_content#create-control-repo-from-template>

but how did you configure code-manager and how did you manage to run the service on another system?

Do I need to install GitLab before configuring Code Manager?

you need any git repo. doesn't matter if thats local or gitlab or github or another git service

Look, I know I’m rooting my own horn here AND that this is 6 years old, but just read through this, as it still largely applies.  

It will tell you how the entire thing is architected from top to bottom:

<http://questy.org/blog/2017/04/17/scaling-puppet-enterprise/|http://questy.org/blog/2017/04/17/scaling-puppet-enterprise/>

puppetlabs/peadm has also some good documentation about it (and is the new standard for setting up a PE environment)

See the `puppetlabs-peadm` module at <https://forge.puppet.com/puppetlabs/peadm?src=slack&amp;channel=puppet>

I guess your classes lookup complains about the missing roles as there is no roles defined at that moment. With strict variables puppet will fail. Not sure if it’s your case

No, i guess i was just a little bit dumb :smile: so the role did not work because it just was not in the data dir. however, the notice is still there, i will check how to get rid of it. thanks!

Silly question, did you restart puppetserver after gem install ?
Also do you have path to those gems in your config ? Something like that
```    gem-path: [
      ${jruby-puppet.gem-home},
      "/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems",
      "/opt/puppetlabs/puppet/lib/ruby/vendor_gems"
    ]```

I roughly remember someone was stuck with activerecord-jdbc too.. but I don’t remember who exactly..

and the place to file a ticket suppose to be <https://_puppet_._atlassian_.net>, but it stuck in never ending migration

<@U125ETYQP> Hi! It seems you had this issue with your Hiera backend, right?

<https://www.linen.dev/s/puppetcommunity/t/10403655/hi-i-have-something-strange-i-m-working-on-a-hiera-backend-t#1a330421-040d-4500-b03a-4dafd641fe89>

also <https://www.linen.dev/s/puppetcommunity/t/10390951/hi-i-have-something-strange-i-m-working-on-a-hiera-backend-t>

Yes I had this issues with active record on Jruby in combination with Puppetserver too

I did not solve the original issue, but worked around it.

Totally different code on the Jruby side as on MRI :disappointed:

We are refactoring now to a http get, where the query logic is on the server and the hiera backed just fetches a specified url

Thank you for the update :bow::skin-tone-2:

Thanks. For the information.
So, bottom line is: I can not use sqlite3 as a source of information for Puppet7 server custom functions now.

ouch. alright. That is interesting because I am migrating from Puppet5 (standalone installation, each server in our infra getting puppet code by r10k and running `puppet apply`)  which is using `activerecord` as custom function (not JDBC, but still).

Probably standalone is a different story…

as long as you’re going to read data from sqlite it should be ok.. though if you’d like to write data from puppetserver to a sqlite DB I’d start looking for another solution instead :slightly_smiling_face:

I am using it for read only, but for master-agent installation I have no option but have to use activerecord-jdbc-adapter which does not work

I’ll look through the links provided above

maybe you should try to convert this to ENC instead

i.e. supply the data from sqlite as part of ENC parameters

(I’m just guessing you’re fetching some data from sqlite based on the target host name or so)

Was thinking about it, not an option for us. As I said it was used successfully with puppet5 as such we have big list of modules which uses this custom function (+some other similar) in the verity of places. Moving to ENC would require refactoring all of that modules.

well.. then try to find some other way to query sqlite from ruby.. other than using activerecord

sqlite3 is a part of Ruby on Rails application which is used for configuration (templates) generations. Thanks. Will think about it.

use REST API e.g. :slightly_smiling_face:

That was what I was looking to avoid ))) Actually we have eliminated need of the RoR App running when migrated from Puppet2 to Puppet5 by introducing this custom function.
And now getting back to be dependent on RoR App in the configuration system - not a good approach.

I managed to get my stuff working with regular JDBC. That might work for you too.

I use the rails app to generate the query’s and used them with JDBC

which (J)Ruby gem did you use for this? So I’ll start dig into this direction

```  def connect_database
    if defined? JRUBY_VERSION
      require 'ostruct'
      require 'connection_pool'

      $db_pool ||= ConnectionPool.new(size: 2, timeout: 5) do
        require 'jdbc/postgres'
        Jdbc::Postgres.load_driver

        props = java.util.Properties.new
        props.put('user', db_config['username'])
        connection = org.postgresql.Driver.new.connect(jdbc_url, props)
        #
        # When this process stops. Close the jdbc connection. Don't signal an error when the close fails.
        #
        at_exit do
          connection.close
        rescue StandardError
        end
        connection
      end

    else
      # RAILS stiff here
    end
  end```

```  def jdbc_find(query, *args)
    give_feeback_if_error do
      $db_pool.with do |connection|
        value = OpenStruct.new
        begin
          statement = connection.prepare_statement(query)
          args.each_with_index { |val, index| statement.set_string(index + 1, val) }
          result_set = statement.execute_query
          meta_data = result_set.get_meta_data
          return nil unless result_set.next

          meta_data.get_column_count.times do |index|
            name = meta_data.get_column_name(index + 1)
            db_value = result_set.get_object(index + 1)
            value.send("#{name}=", db_value)
          end
        ensure
          statement.close
        end
        value
      end
    end
  end```

``` def environment_settings
    if defined? JRUBY_VERSION
      jdbc_find 'select * from environments where envname = ?', environment
    else
      @environment_settings ||= Environment.find_by(envname: environment)
    end
  end```

I hope this gives you an idea where to start

I have a strange issue, not sure how to debug. Once in a blue moon, some of the servers 'disappear' from puppetdb, only to re-appear back next agent run.
```  $servers_query = 'resources[certname]{ type = "Class" and title = "Some::Profile" }'
  $servers = puppetdb_query($servers_query).map |$value| { $value['certname'] }```
Server classification is static, so not sure, why would that be the case. Doesn't happen often, like 4-5 times a month, but enough to trigger my inner-sheldon

did you check if puppetdb deleted the node?

By the time I see the change, it's 'self-corrected', lasts only one agent run

like for whatever reason node decides it's not classified anymore

Only 'schedule' and 'filebucket' resources included

was the environment code not present on the compiler?

that would affect all other servers and only one was at the time. Nope, classification on these is static