Hi, I am trying to use mirrord for my project and I am facing this issue. When I run:

mirrord exec --target pod/llm-7cb6646d5b-cmz45 --agent-namespace healthtech --target-namespace healthtech echo 123

or

mirrord exec --target deployment/llm --agent-namespace myns --target-namespace myns echo 123

I get following output:

mirrord exec
   ✓ running on latest (3.140.0)!
 ⠒ preparing to launch process
     ✓ layer extracted
⠉ mirrord exec
   ✓ running on latest (3.140.0)!
 ⠤ preparing to launch process
 ✓ layer extracted
 x mirrord exec
   ✓ running on latest (3.140.0)!
   x preparing to launch process
     ✓ layer extracted
     ✓ operator not found
     ✓ agent pod healthtech/mirrord-agent-bquydkyrt2-bbm2q created
     ✓ pod is ready
     Error:   × Failed to communicate with the agent: timeout

I've searched everywhere but couldn't find any solution. Even in this channel there are 4 threads with same issue but still could not find any solution. Can someone help me with this? The pod starts and is in running state for several seconds.

mirrord-agent-5q1rdfasxl-vrqvf   1/1     Running   0             5s

The only log I was able to find in the pod before it disappeared is:

agent ready - version 3.140.0

hey, can you run with this prepended to the command line?

MIRRORD_PROGRESS_MODE=off RUST_LOG=mirrord=trace

This is the last part:

2025-04-24T14:03:16.511980Z TRACE mirrord_kube::api::kubernetes::portforwarder: port-forward to pod AgentKubernetesConnectInfo { pod_name: "mirrord-agent-tejadlq1bt-pff4m", pod_namespace: "healthtech", agent_port: 39136 }
    at mirrord/kube/src/api/kubernetes/portforwarder.rs:92

  2025-04-24T14:03:26.511245Z DEBUG mirrord::execution: error: Failed to communicate with the agent: timeout
    at mirrord/cli/src/execution.rs:180
    in mirrord::execution::start_internal

  2025-04-24T14:03:26.513113Z TRACE mirrord_kube::api: No more client messages, disconnecting
    at mirrord/kube/src/api.rs:42
    in mirrord_kube::api::mirrord_protocol_connection_wrapper_task
    in mirrord::connection::create_and_connect
    in mirrord::execution::start_internal

do you have any network policy or something that might block prot forward from working?

Not that I know of

can you try using vanilla port forward?

I have telepresence and I can successfully connect to my cluster

mmm can you try to remove traffic-manager? maybe it's blocking it

yeah it seems like something in the network is blocking.. I'll try to investigate my network more, will let you know the result

if you'd like you can schedule time with me to debug https://calendar.app.google/Rtj7Ujs7UyGCAc1u8