Hello Team, while i am not new to mirrord, wanted to ask, i have a local environment mac, and a aws cluster(mirrord installed) what do i need in local to get connected or basically step by step what next should i do. Thanks in advance.

Hey Guys, we recently started using mirrord in our company. i get this error when staling traffic from a pod

2025-04-02T08:33:52.103630Z  WARN operator_context::target::watcher: Watcher stream encountered an error and will resume with backoff, stream: SingleWatchTask { field_selector: None, label_selector: Some("app.kubernetes.io/instance=notification-api,app.kubernetes.io/name=notification-api,operator.metalbear.co/owner!=mirrord-operator"), target: Multiple { labels: FormattedLabels("app.kubernetes.io/instance=notification-api,app.kubernetes.io/name=notification-api,operator.metalbear.co/owner!=mirrord-operator"), namespace: "assisto" }, resource_url: "/api/v1/namespaces/assisto/pods" }, error: watch stream failed: Error reading events stream: ServiceError: error reading a body from connection
    at operator/context/src/target/watcher.rs:169 on ThreadId(2)

Does anyone has a clue how to debug this ?

I have a Teams License, agent version 3.137.0 and operator helm chart version 1.22.0

Hey team! I'm seeing some weird behavior on recent versions of mirrord paired with elixir/erlang. It appears that under mirrord, i'm unable to access cacert files? Working on a better repro but at minimum you can:

$ mirrord exec -- iex
iex(1)> :public_key.cacerts_get
** (EXIT from #PID<0.108.0>) shell process exited with reason: an exception was raised:
    ** (ArgumentError) construction of binary failed: segment 2 of type 'binary': expected a binary but got: [70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 108, 111, 97, 100, 32, 99, 97, 99, 101, 114, 116, 115, 58, 32, ~c"operation failed with status 137"]
        (elixir 1.17.3) lib/exception.ex:2456: ErlangError.error_info/3
        (elixir 1.17.3) lib/exception.ex:2413: ErlangError.normalize/2
        (elixir 1.17.3) lib/exception.ex:192: Exception.blame/3
        (iex 1.17.3) lib/iex/evaluator.ex:392: IEx.Evaluator.print_error/3
        (iex 1.17.3) lib/iex/evaluator.ex:298: IEx.Evaluator.parse_eval_inspect/4
        (iex 1.17.3) lib/iex/evaluator.ex:187: IEx.Evaluator.loop/1
        (iex 1.17.3) lib/iex/evaluator.ex:32: IEx.Evaluator.init/5
        (stdlib 6.1.1) proc_lib.erl:329: :proc_lib.init_p_do_apply/3

In a non-mirrord iex session you'll get a non-error back. Is anyone aware of anything that may have changed?

Hello, when the mirrord config targets a deployment, and if I have two pod replicas, does the incoming traffic come from both pods?

Does the Mirrord plugin (Jetbrains) support AWS EKS environments that require an MFA token? If so, how/where do we configure that? It seems no matter what I try, I get:

Failed to create mirrord-agent: ApiError: Unauthorized: Unauthorized (ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 })

when trying to run anything via the plugin.

Hi , i updated today to 3.137 and it seems mirrord breaks the pod's health my health check route is 'v1/health/internal' so I added an http_filter for that like following

{
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "path_filter": "^(?!/v1/health/internal)"
        }
      },
      "outgoing": true
    }
  }
}

but it seems i'm not getting logs anymore ? Is that correct ?

$| mirrord exec --context=zc-carsharing-dev mirrord exec -a scott -n scott -t deployment/member-web/container/member-web  yarn start
When targeting multi-pod deployments, mirrord impersonates the first pod in the deployment.
Support for multi-pod impersonation requires the mirrord operator, which is part of mirrord for Teams.
You can get started with mirrord for Teams at this link: https://mirrord.dev/docs/overview/teams/?utm_source=multipodwarn&utm_medium=cli
* Running command: mirrord exec yarn start
* mirrord will target: deployment/member-web/container/member-web, no configuration file was loaded
* mirrord will run without the mirrord Operator
* env: all remote environment variables will be fetched
* fs: file operations will default to read from the remote
* incoming: incoming traffic will be be mirrored
* outgoing: forwarding is enabled on TCP and UDP
* dns: DNS will be resolved remotely
* internal proxy: logs will be written to /var/folders/gs/8hpnmxmn2vgfdblcp8hbyqg00000gp/T/mirrord-intproxy-1744396438-w8BSItk.log
⠤ mirrord exec
    ✓ running on latest (3.137.0)!
    ✓ ready to launch process
      ✓ layer extracted
      ✓ operator not found
      ✓ agent pod created
      ✓ pod is ready
                                                   2025-04-11T18:34:08.215605Z ERROR mirrord::connection: error: Failed to create mirrord-agent: ApiError: jobs.batch is forbidden: User "system:serviceaccount:scott:default" cannot create resource "jobs" in API group "batch" in the namespace "scott": Forbidden (ErrorResponse { status: "Failure", message: "jobs.batch is forbidden: User \"system:serviceaccount:scott:default\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"scott\"", reason: "Forbidden", code: 403 })

am i doing something obvious wrong or does mirrord require you to add perms to the default service account? creating a container like so works

mirrord exec  -a scott telnet cheetah-postgresql.scott 5432

If I do something like this

mirrord exec -a scott --steal -t deployment/member-web/container/member-web sleep 400

i would think all the requests to the remote deployment would fail? but in my testing they dont

mirrord only steals bound ports

you can run a bash script that runs nc then sleeps, that should achieve what you want

thanks thats very helpful. i added it to our instructions to actually do that first, so you can verify traffic stealing works, before trying to run your local server which IME is helpful for debugging

getting the following region error in while using sqs splitting

Hey folks, anyone have an example of running mirrord exec and starting a springboot appllication with gradle bootRun?

https://metalbear.co/guides/how-to-debug-a-java-microservice/ closest we have I think

Thanks for sharing mate. I was hitting some issues with the gradle daemon when executing via cli but starting the application from the IDE worked fine. 🙂

oh, you probably want to use

network.outgoing.ignore_localhost: true

so gradle can communicate safely

Hi all, we are hitting a weird issue where the command

mirrord exec -- /bin/bash

works in one namespace but not another, why is that? the error:

mirrord exec -- /bin/bash
New mirrord version available: 3.138.0. To update, run: `"brew upgrade metalbear-co/mirrord/mirrord"`.
To disable version checks, set env variable MIRRORD_CHECK_VERSION to 'false'.
* Running binary "/var/folders/xj/4xt3127542728_rxnc_z5b6r0000gn/T/mirrord-bin-ghu3278mz/bin/bash" with arguments: [].
* mirrord will run without a target, no configuration file was loaded
* operator: the operator will be used if possible
* env: all environment variables will be fetched
* fs: file operations will default to read only from the remote
* incoming: incoming traffic will be mirrored
* outgoing: forwarding is enabled on TCP and UDP
* dns: DNS will be resolved remotely
⠉ mirrord exec
    ✓ update to 3.138.0 available
  x mirrord exec
    ✓ update to 3.138.0 available
    ✓ ready to launch process
      ✓ layer extracted
      ✓ operator not found
      ✓ agent pod created
      ✓ pod is ready
      ✓ arm64 layer library extracted
    ✓ config summary                                                       Error:   × Failed to execute binary `/var/folders/
  │ xj/4xt3127542728_rxnc_z5b6r0000gn/T/mirrord-bin-ghu3278mz/bin/bash`
  │ with args ["/bin/bash"]

I can see the agent pod starts as expected with the pod log being "agent is ready"

amazing!

Folks, I got a nice working example with a hello-world type service. My next thing to understand is how to make this work in the context of a multi service call chain. Is this something mirrord supports?

Hi. I want usage mirrord in my cluster for demo to my command, but our security policy doesn’t allow pull image from different registry. I moved image to our registry, but I don’t see possibility to configure new image path. Is it possible?

I encountered the error: "mirrord process timed out" when trying to run the mirrord extension on IntelliJ. With the same setup, my colleague can run it without issues, but I get the above error. Could someone please help me with how to debug when using mirrord? Thank you!

hello mirrord people. I have a question: I have a windows 11 machine with wsl install with ubuntu 20.4 (kubectl available in both systems) in Goland from Intellij I have everyting setup to run in the windows environment. (not using wsl host machine) does mirrord works with this setup or I need to setup the "run environment" in wsl? from the IDE I have the error "can't use on Windows without WSL"

Hi, I am trying to use mirrord for my project and I am facing this issue. When I run:

mirrord exec --target pod/llm-7cb6646d5b-cmz45 --agent-namespace healthtech --target-namespace healthtech echo 123

or

mirrord exec --target deployment/llm --agent-namespace myns --target-namespace myns echo 123

I get following output:

mirrord exec
   ✓ running on latest (3.140.0)!
 ⠒ preparing to launch process
     ✓ layer extracted
⠉ mirrord exec
   ✓ running on latest (3.140.0)!
 ⠤ preparing to launch process
 ✓ layer extracted
 x mirrord exec
   ✓ running on latest (3.140.0)!
   x preparing to launch process
     ✓ layer extracted
     ✓ operator not found
     ✓ agent pod healthtech/mirrord-agent-bquydkyrt2-bbm2q created
     ✓ pod is ready
     Error:   × Failed to communicate with the agent: timeout

I've searched everywhere but couldn't find any solution. Even in this channel there are 4 threads with same issue but still could not find any solution. Can someone help me with this? The pod starts and is in running state for several seconds.

mirrord-agent-5q1rdfasxl-vrqvf   1/1     Running   0             5s

The only log I was able to find in the pod before it disappeared is:

agent ready - version 3.140.0

Hello, is it possible to both use a local container and IDE debugging or is it only for local processes ? Or even better debug a remote copy of the pod

Hi, I'm having this issue, attached the full log and the config file node:internal/process/promises:394 triggerUncaughtException(err, true /* fromPromise */); ^ Error: connect Unknown system error -111 127.0.0.1:4200 - Local (0.0.0.0:0) at internalConnect (node:net:1115:16) at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18) at node:net:1353:9 at process.processTicksAndRejections (node:internal/process/task_queues:85:11) { errno: -111, code: 'Unknown system error -111', syscall: 'connect', address: '127.0.0.1', port: 4200 } Any ideas on how to solve it? thanks. https://cdn.discordapp.com/attachments/950020257156833340/1367210380962693231/mirrord.json?ex=6813c164&is=68126fe4&hm=61adc1614c50197d00ccd4ca12faa0b37c6faafdef147525cb0c66a2d4206ebc& https://cdn.discordapp.com/attachments/950020257156833340/1367210381369409536/agent.log?ex=6813c164&is=68126fe4&hm=d5e221f096b2d5d9b42771137fc6f0b1e4f725e4c9ffe75057ae7c7618081f6d&

Has anyone had any luck with NextJS in VScode? Mirrord starts but it never reaches

npm dev -p 3001

* Running command: ./node_modules/.bin/next dev -p 3001
* mirrord will target: service/app-dev-service-dc961645, the configuration file was loaded from xxx
* mirrord will run without the mirrord Operator
* env: all remote environment variables will be fetched
* fs: file operations will default to read from the remote
* incoming: incoming traffic will be stolen
* outgoing: forwarding is enabled on TCP and UDP
* dns: DNS will be resolved remotely
* internal proxy: logs will be written to /var/folders/9w/mn_jvst936z08pnj1xsx0by40000gn/T/mirrord-intproxy-1746180472-zLPxJMa.log
⠄ mirrord exec
    ✓ running on latest (3.141.0)!
    ✓ ready to launch process
      ✓ layer extracted
      ✓ operator not found
      ✓ agent pod app-dev/mirrord-agent-kznolrogb1-hnvxt created
      ✓ pod is ready
      ✓ arm64 layer library extracted
    ✓ config summary

launch.json

{
  "type": "node",
  "request": "launch",
  "name": "Next.js: Debug with mirrord",
  "runtimeExecutable": "mirrord",
  "runtimeArgs": [
    "exec",
    "--config-file",
    "${workspaceFolder}/.mirrord/mirrord.json",
    "--"
  ],
  "program": "${workspaceFolder}/node_modules/.bin/next",
  "args": [
    "dev",
    "-p",
    "3001"
  ],
  "cwd": "${workspaceFolder}",
  "console": "integratedTerminal"
}

Hi, I'm trying to run mirrord extension from vscode and it fails. while running it from cli works without issues. it doesn't let me pick a pod and then complains on not being able to get pods from kubectl https://cdn.discordapp.com/attachments/950020257156833340/1369964721691951114/image.png?ex=681dc692&is=681c7512&hm=63d6061e0ea84b527a6f6c78fc8b0221d4424dbac8df9c317fbf30e2dcb81bad&