Hi, I'm trying to run mirrord extension from vscode and it fails. while running it from cli works without issues. it doesn't let me pick a pod and then complains on not being able to get pods from kubectl https://cdn.discordapp.com/attachments/950020257156833340/1369964721691951114/image.png?ex=681dc692&is=681c7512&hm=63d6061e0ea84b527a6f6c78fc8b0221d4424dbac8df9c317fbf30e2dcb81bad&

Hi, when using included ENV, mirrord didn't "take over" the target pod for example this case: "env": { "include": ["OTEL_NODE_DISABLED_INSTRUMENTATIONS"] }

Intelij IDE plugin

Anyone tried running mirrord with RustRover? For me even when mirrord is enabled, the application runs locally instead of mirroring

Hi, I am building a small stateful application using the ETCD raft protocol. I am wanting to use mirrord to speed up the development but am unable to do so due to file locks being created by my application. The application indicates that the wal log is already locked when trying to take over a pod. Is there maybe a way on how I could work around this or maybe restart the pod when trying to start the application?

message has been deleted

I am experimenting with mirrord, and I thought i understood a lot of how it works. Where I couldn't figure out is: How to use Workload Identity with this setup? I can see the workload identity token being injected into agent pod. Though the connection to other services i.e. Azure Storage Queues fail with Client Assertion. Wondering if there is anyone who has done setup where Workload Identity/ Service Account" have been used. { "target": { "path": "pod/app-dev-charts-api-95874f8d5-b72ws", "namespace": "app-dev" }, "feature": { "env": true, "fs": "read", "network": true }, "agent": { "log_level": "info", "labels": { "app": "workload", "azure.workload.identity/use": "true" }, "namespace": "app-dev", "service_account": "sa-app-dev" }, "kube_context": "au-aks-ctspoke-npd" }

Hi i'm try to use the vscode exetions for mirrord and get the following error 2025-05-20T08:24:03.524216Z ERROR mirrord::connection: error: mirrord operator API failed: creating a websocket connection failed with ApiError: Cannot find namespace

default

for targetless run.: 400 Bad Request (ErrorResponse { status: "Failure", message: "Cannot find namespace

default

for targetless run.", reason: "400 Bad Request", code: 400 }) at mirrord/cli/src/connection.rs:112 Error: × mirrord operator API failed: creating a websocket connection failed with │ ApiError: Cannot find namespace

default

for targetless run.: 400 Bad │ Request (ErrorResponse { status: "Failure", message: "Cannot find │ namespace

default

for targetless run.", reason: "400 Bad Request", code: │ 400 }) but i gave it a namespace this is my mirrord config { "feature": { "network": { "incoming": "mirror", "outgoing": true }, "fs": "read", "env": true }, "operator": false, "agent": { "namespace": "dev-amir-prompt" }, "target": { "namespace": "dev-amir-prompt" } }

Hello everyone. I'm trying to run mirrord with --steal towards a deployment with 2 replicas, but for some reason I only get some of the calls I make forwarded to the local process. In other words, If I run a sequence of

curl

requests to the remote endpoint, I only get about 1/2 of them in my local service. Can it be because the deployment has 2 replicas? This is how I run mirrord

mirrord exec --steal --no-remote-dns --no-outgoing --agent-log-level debug -n <my namespace> -a <my namespace> --target deployment/<my deployment>/container/<my container> -- <my command here>

I am not able to make it work on rustrover. The intelij plugin does not show the prompt to select the pod.

How to fix it ?

Is there any way to silence the output from mirrord? Or only extract the actual command output?

Hi there, we’re facing an issue where some developers (that are using a mac m3 with 16GB of RAM) are stuck in the project start process when running debug with Mirrod. If we disable the Mirrod extension, the issue goes away. Do you have any ideas on how to resolve this?

Hi, I have been trying to resolve some issues where connection drops. I have had limited success by doing this - https://metalbear.co/mirrord/docs/faq/common-issues#agent-connection-fails-or-drops-when-using-an-ephemeral-agent-with-a-service-mesh The first time it works, but the second time I get this.

{
  "timestamp": "2025-07-18T05:54:45.794723Z",
  "level": "ERROR",
  "fields": {
    "error": "io error: Address already in use (os error 98)"
  },
  "target": "mirrord_agent::entrypoint",
  "span": {
    "args": "Args { mode: Ephemeral { mesh: None }, communicate_port: 51234, communication_timeout: 360, network_interface: None, metrics: None, test_error: false, operator_tls_cert_pem: None, is_mesh: true, ipv6: false }",
    "name": "start_agent"
  },
  "spans": [
    {
      "args": "Args { mode: Ephemeral { mesh: None }, communicate_port: 51234, communication_timeout: 360, network_interface: None, metrics: None, test_error: false, operator_tls_cert_pem: None, is_mesh: true, ipv6: false }",
      "name": "start_agent"
    }
  ],
  "threadId": "ThreadId(1)"
}

I don't have

ttl

set, but I do have

communication_timeout: 360
      startup_timeout: 360

Will these keep the agent alive? Or is there a recommended way to make this work? Thanks

Hey, me again. I don't think this is the same issue but similar to above. The first time a pod boots up, mirrord works. After the second time I try to connect, it logs something about IP tables, I'll post in thread. There is still another agent Running in the pod according to k8. However said running agent has logged several close messages, so I'm confused why it's still running.

Hello 👋 ! How mirrord handle statefulset if I have local file system that I need to access ? Or mounted volume ? Do I need to set any particular configuration to make it available through my local process ? Thanks in advance ! 🙏

Hello again 👋 ! Anyone already tried to use OTEL autoinstrumentation with mirrord ? It seems to not work since Rider with mirrord plugin does not launch the dotnet process using the

instrument.sh

and I am wondering if there is any workaround to make it work. Thanks in advance 🙏

Hey everyone, my mirrord proxy stopped working today. When spinning up my local debug session, the API returned a temporary redirect, so I investigated further. I found the following entries in the mirrord operator logs:

shell
2025-11-20T16:38:20.099316Z ERROR ThreadId(02) operator::shared_session_watcher: 108: mirrordclustersessions watch stream encountered an error error=watch stream failed: ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): storage is (re)initializing: TooManyRequests

2025-11-20T16:39:10.078007Z  WARN ThreadId(02) operator::mutated_pods_cleanup_controller: 205: pods watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof

2025-11-20T16:39:10.095030Z  WARN ThreadId(02) operator::workload_patch_request_controller: 415: mirrordclusterworkloadpatchrequests watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof

2025-11-20T16:39:10.096672Z  WARN ThreadId(02) operator::copied_pods_controller: 172: pods watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof

2025-11-20T16:39:10.107676Z  WARN ThreadId(02) operator::agent_pods_controller: 178: pods watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof

2025-11-20T16:39:10.111854Z  WARN ThreadId(02) operator::workload_patch_controller: 245: mirrordclusterworkloadpatches watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof

I completely uninstalled the operator from the cluster and reinstalled it, but that didn’t resolve the issue. It looks like some kind of “too many requests” error. Could this be coming from the Kubernetes API server? Technical information: • Kubernetes: 1.31.4 • Mirrord-Operator: 3.130.0 • Mirrord: 3.173.2

We use a paid Team License if that is of any matter 🙂

Hey y'all, looks like some of the links are broken on the db-branching docs page: https://metalbear.com/mirrord/docs/using-mirrord/db-branching#whats-next https://cdn.discordapp.com/attachments/950020257156833340/1445865976263348224/image.png?ex=697b125f&is=6979c0df&hm=c50c7b94b9bc3ce188e72c6f194ea5812aac2f7c85118b6c5c6fc67d73351578&

Hey all, trying to get the operator up and running for teams and running into some issues. The Operator is up and deployed and has the license configured properly (checked via

mirrord operator status

), and trying to start a new session continuously fails with a 403 on upgrading connection to be over Websockets:

mirrord operator API failed: creating a websocket connection failed with failed to upgrade to a WebSocket connection: failed to switch protocol: 403 Forbidden

I'm using my admin role so i can port-forward/create-pods/etc. Any tips on what this could be?

Hey guys, since trying a new cloud vendor we got this error log in the operator

2026-01-13T10:06:45.161892Z ERROR ThreadId(02) operator::workload_patch_request_controller: 494: mirrordclusterworkloadpatchrequests watch stream encountered an error error=failed to perform initial object list: ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): storage is (re)initializing: TooManyRequests
2026-01-13T10:06:45.198122Z ERROR ThreadId(02) operator::shared_session_watcher: 262: mirrordclustersessions watch stream encountered an error error=watch stream failed: ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): storage is (re)initializing: TooManyRequests
2026-01-13T10:06:45.216294Z  INFO ThreadId(01) operator_service: 726: Shutting down probe and OpenAPI server to start full API
2026-01-13T10:06:45.217852Z ERROR ThreadId(02) operator::external_resources_controller: 292: mirrordclusterexternalresources watch stream encountered an error error=watch stream failed: ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): storage is (re)initializing: TooManyRequests
2026-01-13T10:06:45.283038Z  INFO ThreadId(01) operator_service: 737: Starting full API server
2026-01-13T10:06:45.988932Z ERROR ThreadId(02) operator::workload_patch_request_controller: 494: mirrordclusterworkloadpatchrequests watch stream encountered an error error=failed to perform initial object list: ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): storage is (re)initializing: TooManyRequests
2026-01-13T10:06:47.624663Z ERROR ThreadId(02) operator::shared_session_watcher: 262: mirrordclustersessions watch stream encountered an error error=failed to perform initial object list: ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): storage is (re)initializing: TooManyRequests

Is this related to our cluster or indicates a problem in our setup ?

also this could be interesting

2026-01-13T10:07:12.674016Z  WARN ThreadId(02) operator_kube_utils::list_retry: 28: Failed to list mirrordpolicies, retrying... error=ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 })
2026-01-13T10:07:12.880635Z  WARN ThreadId(02) operator_kube_utils::list_retry: 28: Failed to list mirrordclusterpolicies, retrying... error=ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 })
2026-01-13T10:07:13.205928Z  WARN ThreadId(02) operator_kube_utils::list_retry: 28: Failed to list mirrordtlsstealconfigs, retrying... error=ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 })
2026-01-13T10:07:13.443006Z  WARN ThreadId(02) operator_kube_utils::list_retry: 28: Failed to list mirrordclustertlsstealconfigs, retrying... error=ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 })

Here are the specs Kubernetes Version: v1.34.3 Mirrod Agent version: 3.180.0 Operator Helm chart: 1.43.0

Hello 👋, Is it mandatory to have operator (and Team license) to remote debug a statefulset ? Or it's compatible with OSS version ? Thanks in advance !

Hello 👋, We are trying to setup Mirrord using WSL with Ubuntu to run a Rider project. The plug-in seems to not work when trying to launch the application with Debug and we observed the following behavior : 1. Clicking on "Select Active config does nothing" 2. Click on "Settings" open a mirrord.json that does not seems to be inside the project .mirrord/json that seems to be under Windows file system on a weird location (see screenshot). Message.txt contains attached Rider logs with a stack trace for : Index is not created for

mirrordConfig

Any idea ? Thanks in advance ! https://cdn.discordapp.com/attachments/950020257156833340/1466467437594546308/message.txt?ex=697cd9ba&is=697b883a&hm=0da207c0194da16c8229a53aa5085f30f55103dfae7c5f261e71946e916774cd& https://cdn.discordapp.com/attachments/950020257156833340/1466467438106378280/image.png?ex=697cd9ba&is=697b883a&hm=5ebe3987357012e439f5d43be11d98238bff9b85c2b833d8dd9a47dd96426c68&

How do one get the Intellij plugin to download the newest mirrord version? It seems to insist on downloading 3.180.0 only

Would one expect inotify/fsnotify events and kubernetes secrets mounted volumes to play nicely with mirrord? I've put some debugging on my app which has a hot reload function when the mounted secret changes via the projected volume (no subpath just the whole directory) and when using mirrord the application is seeing constant "created" fsnotify events.

All of a sudden I started getting

mirrord error: Failed to execute authentication command specified in kubeconfig: No such file or directory (os error 2)

(VSCode) Haven't changed a bit of my kubeconfig. Any ideas?

seems like mirrord didn't like the command

aws

being alone in the config. I had to change it to

command: /usr/local/bin/aws

. That's strange because I know that wasn't touched for ages

Yet another problem that I keep on hitting these days. The logs are not very helpful, but once I reboot seems to work fine. And why does the VSCode plugin constantly trying to download the binary every single time I try to run something through mirrord? https://cdn.discordapp.com/attachments/950020257156833340/1494623617491796038/image.png?ex=69e34837&is=69e1f6b7&hm=db22a03d591b9e27393b9649f265296a355755c6639031d15104ab4968973db0&