This message was deleted.
# puppet-enterprise
s
This message was deleted.
b
identify why it times out
large IO/IOWait on the disk?
http timeout reaching git server/forge?
and therw is somewhere a logback config file where you can configure the verbosity
c
Sounds familiar that it is having problem auth against git server
c
Tim's got it right: pulling together the status of what is in File Sync storage took longer than a few tens of seconds. Likely due to I/O constraints. An upgrade could help as there have been a few optimizations in the 2019.8.x timeline that either reduced the number of walks done on the filesystem, or avoided pulling detailed information when the status endpoint was not queried with
?level=debug
.
i
I have one compiler that is having problems (according to
puppet-infrastructure status
)
Copy code
Compiler: compiler22
  File Sync Client Service: Error, checked via <https://compiler22:8140/status/v1/services>
  Puppet Server: Running, checked via <https://compiler22:8140/status/v1/services>
I rebooted it, I restarted the services
not sure what its problem is
b
jup. upgrade to 2019.9.10 is usually a nobrainer. and running iostat 1 in another terminal is quite easy to check iowait
check iowait :)
i
upgrading in a huge production env is not a no-brainer...
I have 12 other compilers that are not having issue with sync in the same env, so...
c
Well patching a prod environment is probably something that should be accounted for... 2019.8.5 to 2019.8.10 or 2019.8.11 should be something you can do especially for security reasons
i
yes, its on the roadmap but isnt doable to fix one compiler
c
i still would check iowait.. is it a vm? local? vmware? cloud? instance size/disk any different from this one vs the other 11?
sounds like there is a disk issue... maybe the SAN disk is degraded for this one compiler
i
they are all identical
yes vmware
c
so the underlying disk can be on a bad lun... degraded... etc
vmhost having issues this node is on
i
other thing is that I removed the bad one from the pe-compiler nodegroup, stopped it services, and ran puppet on the mom it took the bad one out for a few minutes but its back
though its still not in the nodegroup, strange
c
your compilers are behind a vip right?
since dns entry for all agents?
i
oh, it was also in the pe-masters group
I was finally able to get that broken compiler out of the mix