we had idle timeout at 5 minutes and keep alive at 30 minutes. idle timout was increased today to 16 minutes (to be higher than the timeout in the broker). I will check if this changed anything tomorrow morning

and we had no sticky connections. that got enabled today as well, based on src ip. but we still saw errors afterwards (but maybe less)

lowering keep-alive shouldn't be required since the pxp-agent sends that on it's own

Are you sure?

not in my experience

which statement do you mean? the pxp-agent own keep alive?

yes

https://github.com/puppetlabs/pxp-agent/blob/main/CHANGELOG.md#150

The default ping-interval is now 2 minutes.

I want to verify that with tcpdump tomorrow (but that requires approval from security team...)

The agent won’t retry its connection until several (maybe 3?) consecutive pings are missed. So if some network device drops the connection, the agent won’t notice for a while and will just be offline

Ahhh I just figured out what this is referring to! “Timed out waiting for provisional response”, that means orchestrator sent the request to run a task but never got back a response.

And that implies the broker thinks the agent is connected (otherwise it would just fail immediately and tell you the agent isn’t connected). That’s still more evidence suggesting the connection is being silently dropped. Each side thinks they’re still connected but they’re not.

ahh

so the pxp-agent receives a job with the correct id and executes it. so it also thinks it's still connected I guess

I don't have the logfile by hand at the moment, I think it even logged that it reported a successful status back

Actually that means the connection isn’t totally dropped, it’s just dropping packets in one direction

Or it’s delivering those packets very slowly

https://support.f5.com/csp/article/K14814 potentially?? That’s out of my area of expertise, and quite an old article/issue

https://support.f5.com/csp/article/K14814 potentially?? That’s out of my area of expertise, and quite an old article/issue

now if I would know anything about f5, that would help a lot

or having at least read only access to the config, that would be a good start

mhm I think it's a tcp profile in our case

right, you need to see what type of load balancing they created

if we don't break up tls it's tcp-profile?

I use Performance (Layer 4), so it's something related to it

reassembled-fastL4, I think

let me check, I think I've screenshot from the f5 config somewhere

yes it's a tcp profile

There are quite a few websocket-related bugs on the F5 website 😐 I wonder whether any of those cause problems, but I don’t have a good way to test it

Though it looks like they’re all problems when using an HTTP profile, which isn’t possible with pxp