https://www.puppet.com/community logo
Join Slack
Powered by
This message was deleted.
# puppet
s
This message was deleted.
v
I think there is one gotcha there. dashboard will show if catalog failed to apply.
but if catalog failed to compile - it's a different story
b
mhm yeah
v
I don't think that one shows
b
@runlolarun ^
v
I just splunk puppetserver.log for failures like that
r
hmm then what’s the point of puppetboard’s failures… Sorry…
i wonder if nagios can monitor it.. i’ll look into it later on..
b
I'm happy to review your patches for puppetboard 🙂
v
I am with you on that one
PE added very unnoticeable "intended catalog failure"
I call intended failure a sabotage 🙂
r
sorry i didn’t mean to sound annoyed. Well i am, but you’re right.
v
But is it your case though?
r
i just wish i knew this from the beginning.. Sucks that i wasn’t the one who noticed it.. that’s all. It’s my responsibility.
l
so, what we do is monitor not failures, but last success
r
how?
l
if last success is > 2x the normal difference in time, alert
so, there's a puppetdb field that tracks it, but our way of doing it was a tiny report plugin that produced a zabbix_send from the puppetserver to the zabbix server
ken's script below has the basic idea
r
fair enough. Okay, thank you.
sorry guys, but can you please explain to me again why puppetboard didn’t show this node as failed? There was an error 
Could not retrieve catalog; skipping run
. Why that didn’t trigger the change and put the node under failed nodes?
l
could not retrieve catalog means that building the catalog failed, so there's no run for puppet agent to send to the db
r
but how come other failures were “noted”? Like i have other nodes that have failures, but catalog succeeds. What is different in this case..?
It seems like at this point it’s a rather puppet question and not a puppetboard one.
Actually i take it back. In both cases I see :“applied catalog” message…
One is notified by puppetboard and another one is not…
so my question stands. I’m still confused …
l
🤷 I look for "sent report"
r
where?
b
@runlolarun those are two different things. puppetboards lists puppet that failed to apply their catalog, it doesn't show you nodes that couldn't retrieve a catalog. to identify that it's possible to check when the last run was. for example via puppetdb or the state file
r
But right now, i have a node in failed state and it has “Applied catalog in 15.29 seconds”
b
but that's an older catalog?
or a cached one?
usecacheonfailure
defaults to true. if a node fails to retrieve a new catalog it will apply an older, cached, catalog
r
I don’t think so.. as it added some things..
b
can you show the whole puppet agent output?
r
I’ll DM you.. sorry, it has too much stuff.
i removed some stuff…
it has a lot more failures, all expected….
b
okay, so in that case it received a catalog
r
right, and it’s under Failures right now. I will show you from the one that wasn’t showing..
v
that should be reported as a failure
are you looking under right environment ?
l
good, that should be in puppetdb
v
master is not a standard one in dashboard
r
it is in my setup
this is from the logs
and wasn’t showing in Failures under puppetboard
l
yea, so 500 error means that it doesn't ever get a catalog so it can't tell puppetdb that a run occured
v
try this
Copy code
puppet config set usecacheonfailure false
and run agent again
r
$ puppet config print |grep usecache usecacheonfailure = true
so set it to false?
v
yep
r
do i need to restart puppet server or it will be automatic?
v
automatic
and this is on agent
not on server
r
oh run this is on all agents?
v
you can add it to a manifest
if that's the behavior you want
because puppet did apply a catalog, just not the latest 🙂
r
I see… I’ll read up on this to understand better. I understand now what happened. Thank you, all.
👍 1
seriously. All of you. thank you for taking your time to respond to me. I really appreciate all your help.
l
no problem, thanks for the good question
r
oh anytime! I have a lot of those 🙂 sometimes good, sometimes not so.. 🙂
2 Views
Open in Slack
PreviousNext
Powered by