yep. most of them have a delay of a few seconds, just a few are around 59

okay, so the questions is, where is the delay coming from 🤔

Can you correlate a “late” pong with the pcp-broker logs to determine whether the broker a) received the ping in a timely manner, and/or b) sent the ping in a timely manner?

It looks like when things are functioning properly, pongs come back in a couple of milliseconds. My assumption is that the packets are being delivered late in one direction or the other, so either the broker receives the ping ~59 seconds after it was supposed to, sends back a response immediately and that response is delivered immediately (very late), or the broker receives the ping immediately, sends back the response immediately, but the response is delivered ~59 seconds later.

I’m unclear what the buffering behavior of f5 is, but I also wonder if some of this traffic is being buffered somewhere.

I can tell for sure, no 🙂

my problem is that that the broker doesnt log from where it received a ping / to where it sends it

and I have around 200 agents connected to each broker and I cannot isolate a broker. I the logback config be adjusted so it logs the remote host ame/certname?

Ah yeah it looks like all the ping behavior is handled by jetty directly, hmm

Turning on debug logging for

org.eclipse.jetty.websocket.common

might show it. I need to test that locally though

Well, that does show it, but it also logs way more than just that

Is there any correlation between these slow pongs and when you are actually running task jobs? That is, does this only happen when the broker is actually under some amount of load?

no. I see the timeouts all the time. I didnt find a pattern yet. it varies from 5 to 100 timeouts per day per node

and primary/compiler VMs are kinda beefy, and we rarely run more than two tasks at the same time (with 1 to 150 targets maybe)

The only way I could imagine your environment creating substantial load is if you’re running tasks that return really large results (mbs)

mhm

we have some envs where every class does a notify to print the class name

and when a task runs puppet there is a lot of output. maybe a few hundred lines. but should be several MB

has the broker or orchestrator any metric I should take a look at?

I would expect that to only cause a problem when that task is actually running though 😕

and we didnt tune any of the default pcp message size options

mhm yeah

There are KB articles like this https://support.f5.com/csp/article/K87173503 but I assume that would be a problem for all TCP traffic through the load balancer, not just PXP

And if the problem is simply network congestion/latency, why don’t other applications experience problems?

It’s hard to imagine, though not impossible, network conditions where the same request/response might take any of: 3 milliseconds, 7 seconds, and 59 seconds.

maybe other applications are not that sensitive to latency

This message was deleted.

This message was deleted.

This message was deleted.

@Gareth McGrillan has left the channel