I've noticed some odd behaviour in MirrorD I'm using the latest MirrorD binary, and IntelliJ MirrorD plugin 3.60,1 When stealing GRPC traffic, either with a HTTP filter or without, one of 3 things happen and it's almost random which will happen: * The traffic is stoled as expected and my laptop intercepts the calls made to the remote Pod * The client making the call starts to hang and eventually throws a 504 error, neither the remote pod nor my local app receives the traffic. I know MirrorD is causing this because as soon as I stop stealing traffic, the client is able to reach the remote pod again successfully * The remote pod receives all the traffic, and MirrorD is being ignored I've witnessed all 3 behaviours within the span of 30 mins, with no changes made in my MirrorD config. I'm not entirely sure how to debug this, because in all 3 scenarios I have enabled DEBUG logging on the agent and could not find any logs to suggest the packets were dropped, or traffic was being ignored from stealing. I suspect Istio might be interfering with the mirrorD agent, but I have no way of proving it. Am I correct that packets received by the remote pod, will first go to the Istio Sidecar, then when the sidecar forwards packets to the receiving remote pod, only then does mirrord steal traffic from my remote pod and sends to my laptop?

hello, i'm getting some odd behaviour when trying to mirror a pod in GKE:

Create agent failed. InvalidResourceState( “Node default/gke......... is in invalid state: too full with 118 pods”, )

Could someone assist why this is happening, i can't see why too many pods would be an issue but happy to be proven wrong.

Hi there!

Why traffic intercept is not working?

Hi! Is it possible to run a job using mirrord? I tried to do this for a one-off job from a cronjob but couldn’t because I couldn’t get a completed pod in the ready state

hey for some reason i can't get mirrord to steal connections. i see multiple errors in the agent

Hey guys, I am failing to run MirrorD within IntelliJ, after running my Java Spring boot service with MirrorD enabled I see the following error Failed to create Kubernetes API client: Kube failed: auth error: unable to run auth exec: No such file or directory (os error 2) I can confirm that my kubernetes permissions are correct, as running

kubectl get pods

and

kubectl apply -f ....

work on my terminal, I am authenticated in the cluster with admin permissions so MirrorD should have no difficulty spawning the MirrorD agent

Hi The mirrord extension tried to intercept my test runs thinking its a "mirrord" session in GoLand. Perhaps you should identify its a test run and not try to intercept

hi guys, I'm encountering some problems while trying to use mirrord - I believe it is trying to use my local fs although I configured it to use the pod's fs. this is my mirrord.json

{
  "target": {
    "path": "deployment/opencost/container/opencost",
    "namespace": "opencost"
  },
  "feature": {
    "fs": {
      "mode": "write"
    },
    "env": {
      "include": "*"
    }
  }
}

that's my error:

2024-07-11T17:10:10.701429+03:00 INF starting download of "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/eu-west-2/index.json", which is quite large ...
Proxy error, connectivity issue or a bug. 
You may report it to us on https://github.com/metalbear-co/mirrord/issues/new?assignees=&labels=bug&projects=&template=bug_report.yml 
io failed: Resource temporarily unavailable (os error 35)
2024-07-11T17:10:29.109697+03:00 ERR Storage Stat Error: Failed to stat file: stat /models/aws.json: invalid argument

how can I debug this?

Hello! I'm trying to use mirrord with dapr and it seems my local process cannot connect to the dapr sidecar, I get a connection refused error 😢

Hi everyone, I'm having trouble setting up

mirrord

to debug my Kubernetes pod, and I could really use some help. Here's the issue I'm facing: I am trying to use

mirrord

to run my Go application and target a Kubernetes pod, but I keep getting the following error:

2024-07-31T12:52:49.253+0200    ERROR   controller-runtime.source.EventHandler  source/kind.go:76       failed to get informer from cache       {"error": "failed to get restmapping: failed to get server groups: Get \"https://127.0.0.1:6443/api\": dial tcp 127.0.0.1:6443: connect: connection refused"}

Additionally, I have the following

mirrord

configuration file: mirrord.json:

json
{
  "accept_invalid_certificates": false,
  "skip_processes": "ide-debugger",
  "kubeconfig": "~/.kube/config",
  "kube_context": "default",
  "target": {
    "path": "pod/operator-controller-manager-86bb6b4c79-t5dsl",
    "namespace": "c8yedge"
  },
    "feature": {
        "network": {
            "incoming": "mirror",
            "outgoing": true
        },
        "fs": "read",
        "env": true
    }
}

shell
❯ kgp
NAME                                                   READY   STATUS    RESTARTS   AGE
operator-controller-manager-86bb6b4c79-t5dsl   1/1     Running   0          40m
mirrord-agent-zeoqsitccd-cbc7t                         1/1     Running   0          5m13s

❯ kl mirrord-agent-zeoqsitccd-cbc7t
agent ready - version 3.112.0
2024-07-31T10:52:11.782957Z  WARN ThreadId(07) mirrord_agent::outgoing::udp: interceptor_task -> no messages left

Any insights on why this might be happening or how to resolve it would be greatly appreciated! Thanks in advance for your help!

Hi guys trying to steal some traffic, but getting an error(didn't change anything from yesterday when it was working)

mirrord exec -c --steal --target pod/my-pod --config-file kubie-config.yaml  ./.mirrord/mirrord.json

  x mirrord exec
    ✓ running on latest (3.112.0)!                                                                                                                                                                                                                                                                                 Error:   × Failed to verify mirrord config: mirrord-config: `unknown field `clusters`, expected one of `accept_invalid_certificates`, `skip_processes`, `skip_build_tools`, `connect_tcp`, `operator`, `kubeconfig`, `sip_binaries`, `target`, `agent`, `feature`, `telemetry`, `kube_context`, `internal_proxy`,
  │ `use_proxy`, `experimental` at line 2 column 1`!
  ├─▶ mirrord-config: `unknown field `clusters`, expected one of `accept_invalid_certificates`, `skip_processes`, `skip_build_tools`, `connect_tcp`, `operator`, `kubeconfig`, `sip_binaries`, `target`, `agent`, `feature`, `telemetry`, `kube_context`, `internal_proxy`, `use_proxy`, `experimental` at line 2
  │   column 1`!
  ╰─▶ unknown field `clusters`, expected one of `accept_invalid_certificates`, `skip_processes`, `skip_build_tools`, `connect_tcp`, `operator`, `kubeconfig`, `sip_binaries`, `target`, `agent`, `feature`, `telemetry`, `kube_context`, `internal_proxy`, `use_proxy`, `experimental` at line 2 column 1
  help: Inspect your config file and arguments provided.



{
    "kubeconfig": "kubie-configf.yaml",
    "feature": {
        "network": {
            "incoming": {
                "mode": "steal",
                "ignore_ports": [
                    8081
                ],
                "http_filter": {
                    "header_filter": "DEBUG"
                }
            }
        },
        "fs": "read",
        "env": true
    }
}

Hi, I've seen some comments here about FastAPI and uvicorn. I have a similar issue where I cannot get the traffic from the cluster when running locally. IP and port are the same as in the cluster. Any tips ? Thanks

Hi all, I have a kafka consumer pod running on my kubernetes cluster, can mirrord do stealing of specific kafka messages? I did something like this for http request, is there a similar way for kafka consumer?

{
  "feature": {
    "env": true,
    "fs": {
      "mode": "read"
    },
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "header_filter": "X-Mirror: testing",
          "ports": [
            80,
            8080
          ]
        }
      }
    }
  }
}

Hey guys, I've been reading into the new Queue splitting feature (can't wait for it's release 😁 ) I was wondering how this config would work:

feature:
  split_queues:
    my-first-queue:
      queue_type: SQS
      message_filter:
        wows: "so wows"

Does this mean that if a message is added to the SQS with property

wow

set to

so wows

then the remote pod can still consume it, but if a local service is running with this config, then the local service will have priority over this message?

Hi all, I'm thinking how to use mirrord with a CronJob, should I just use Targetless mode ?

Hi everyone I'm back with another question. I'm trying to figure out how to debug a Docker image locally using mirrord. I came across a comment on this [GitHub issue](https://github.com/metalbear-co/mirrord/issues/1658) suggesting to > work in a dev container and run mirrord from there I haven't tried this approach yet, but I was wondering if there's any documentation available on it? Any guidance would be appreciated! Thanks

Hi! Thank you for such a great piece of software. I really love how easy it has made my development via configuration. I have a few questions. 1. I am stealing traffic via headers but it seems sometimes it works, sometimes it doesnt. What I have observed is if I wait for sometime for the first request after container is up then it seems to work fine. Here is my cofiguration:

"network": {
  "incoming": {
    "mode": "steal",
      "http_filter": {
        "header_filter": "^User-Agent: Mozilla\/5.0 \\(Macintosh; Intel Mac OS X 10\\.15; rv\\:130.0\\) Gecko\/20100101 Firefox\/130\\.0",
        "ports": [3000]
      }
    }
  }
}

I am stealing traffic from my system/browser specifically(is there a better way to do this?)

2. Second question is I have been secretly using mirrord and want to bring my entire organization onboard but I am afraid might experiment may interfere with the cluster and weaken my case. Is this possible? I am using ephemeral containers. Here is my configuration again:

"agent": {
  "ephemeral": true
}

The reason I mention this is I am using VS Code extension and I have noticed that the proper way is to click that button next to mirrord in the bottom toolbar to make everything work properly. I killed processes with ctrl+c a few times and that resulted in errors in the cluster(containers not coming up). Today, when I sshed into a container, I got this as output:

> kubectl exec --namespace="allex-accounts" --stdin --tty accounts-96fc6966c-2mk2k -- sh
Defaulted container "accounts" out of: accounts, mirrord-agent-srtm0at7ti (ephem), mirrord-agent-5ds4u0rgio (ephem), mirrord-agent-vkc3cxzcli (ephem), mirrord-agent-dluhpgynee (ephem), mirrord-agent-7y1prbmfl9 (ephem), mirrord-agent-v3xpaqejfw (ephem), mirrord-agent-lmyjkv4ma7 (ephem), mirrord-agent-iq2hv8wb8o (ephem), mirrord-agent-ebk6ecmru8 (ephem), mirrord-agent-vj56mq0yll (ephem), mirrord-agent-oc1yrlxobl (ephem), mirrord-agent-naqh6vmumk (ephem), mirrord-agent-1c3vc8ubpa (ephem), mirrord-agent-th2mjxxloo (ephem), mirrord-agent-2ytfjofd2c (ephem), mirrord-agent-z2n4hm0sbn (ephem), mirrord-agent-c5efsk1bs5 (ephem), mirrord-agent-0zmadv6ov9 (ephem), mirrord-agent-ff97gfch9x (ephem), mirrord-agent-sdtpoz4s2z (ephem), mirrord-agent-fe4e6z0vuh (ephem), mirrord-agent-fro5i3odhp (ephem), mirrord-agent-irviob5pxw (ephem), mirrord-agent-ugqbpvsdm5 (ephem), mirrord-agent-nk9zn0mrqm (ephem), mirrord-agent-bhtqiphy4v (ephem), mirrord-agent-dm2prnlef6 (ephem), mirrord-agent-8coyd3uu2r (ephem), mirrord-agent-d1erwnuq1h (ephem)

Sorry if I am not making much sense but I just want to use it correctly so that I can do a demo for everyone.

Hi all, I had a problem with kubernetes RBAC

Hello! Small question! Does mirrord work with telepresence? If some of your team mates are using telepresence and you are using mirrord, do they both interfere with eachother in any way?

we have a pycharm/python use case where platform-specific (amd64) dependencies come from a git-based file share. with telepresence we can intercept the pod and then debug from an arm64/m3 laptop as amd64 using local docker. in mirrord this almost seems to work, but dns name resolution doesn't work. when we exec to the running docker container,

dig

fails to reach the nameserver from

etc/resolv.conf

. we've tried

--networking=host

and

--networking=bridge

. (it seems like a networking problem, but mirrord network model is a bit opaque so we can't tell if the

ifconfig

makes sense or not) any docs or pointers on how mirrord can intercept a local docker container debug from pycharm/intellij? thanks!

Hi all, I'm curious to try the local container feature (https://mirrord.dev/docs/using-mirrord/local-container/) with VSCode. Is the IDE extension support available yet, even as an alpha version ? Thanks

I'm having some issues getting mirrord to function. More specifically getting my local application to connect to my cloud sql through mirrord. The connection just times out. Are there any good debug methods that I can use?

Hey, we are working with this setup: { "target": { "path": "deployment/service", "namespace": "something" }, "feature": { "network": { "incoming": "mirror", "outgoing": true }, "fs": "read", "env": true }, "operator": false, "kube_context": "env7" } and for some reason we are getting 401 now for not bieng able to contact with mirrord-agent. there is no mirrord-agent because we are not working with operator and im with full admin access so i shouldnt get 401 for anything... thats the error from the IDE: 2025-02-03T12:58:54.212023Z ERROR mirrord_kube::api::kubernetes: error=ApiError: Unauthorized: Unauthorized (ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 }) Error: {"message": "Failed to create mirrord-agent: ApiError: Unauthorized: Unauthorized (ErrorResponse { status: \"Failure\", message: \"Unauthorized\", reason: \"Unauthorized\", code: 401 })","severity": "error","causes": [],"help": "1. Please check the status of the agent pod, using

kubectl get pods

in the relevant namespace.\n 2. If you don't see any

mirrord-agent-[...]

pods, then try running

kubectl get jobs

, and

kubectl describe mirrord-agent-[...]

.\n Make sure it is able to fetch the agent image, it didn't fail due to lack of resources, etc.\n\n- If you're still stuck:\n\n>> Please open a new bug report at https://github.com/metalbear-co/mirrord/issues/new/choose\n\n>> Or join our Discord https://discord.gg/metalbear and request help in #950020257156833340\n\n>> Or email us at hi@metalbear.co\n\n","labels": [],"related": []}

P.S. it used to work a while back

Hey folks, I'm facing an issue when I run with steal mode compared normal mode. When I run my server with "incoming": "off", "outgoing": true. The egress going from my local server to k8s cluster is quite fast - I don't hit any timeouts during startup process. When I run with config "network": { "incoming": { "mode": "steal", "ports": [9281], "http_filter": { "path_filter": "^(?!/sf/healthcheck).*$", "ports": [9282] } }, "outgoing": true } The startup of my service become very slow. For startup of our service we make a call to a grpc service that downloads over 2mb data. does all of that goes through the network traffic passed to my network? In that case How to filter out data coming from a specific port?

Hi guys, I'm trying to run mirrord in my corporate environment. The plugin runs in IntelliJ. But when I try to debug my GKE container mirrord seems to fail to install its agent. Error:

Failed to create mirrord-agent: ServiceError: client error (Connect)

. I've uploaded the image to the Artifactory instance from which GKE pulls all images successfully. I can even start a container manually like so:

kd run mirrord-agent --image="artifactory.local:5200/metalbear-co/mirrord:latest"

But it seems like mirrord doesn't respect the corresponding setting in its mirrord.json:

"agent": { "image": "artifactory.local:5200/metalbear-co/mirrord:latest" }

Hi guys, has something changed recently regarding config schema validation, more specifically the connect_tcp field? Intelij complains that connect_tcp field is not allowed anymore Removing the field works. here is the full config.json

{
  "accept_invalid_certificates": false,
  "skip_processes": "ide-debugger",
  "kube_context": "{{ get_env(name="MIRRORD_KUBE_CONTEXT", default="undefined") }}",
  "target": {
    "path": "{{ get_env(name="MIRRORD_TARGET", default="undefined") }}",
    "namespace": "{{ get_env(name="MIRRORD_NAMESPACE", default="undefined") }}"
  },
  "connect_tcp": null,
  "agent": {
    "log_level": "mirrord=debug",
    "namespace": "{{ get_env(name="MIRRORD_NAMESPACE", default="undefined") }}",
    "image": "ghcr.io/metalbear-co/mirrord:latest",
    "image_pull_policy": "IfNotPresent",
    "ttl": 30,
    "ephemeral": false,
    "communication_timeout": 30,
    "startup_timeout": 360,
    "network_interface": "eth0"
  },
  "feature": {
    "env": { },
    "fs": {
      "mode": "write"
    },
    "network": {
      "incoming": "steal"
    }
  }
}

https://cdn.discordapp.com/attachments/1001193615474827264/1358805343970394233/image.png?ex=67f52d97&is=67f3dc17&hm=449373546a44319240a1581e0a6ca32047cc3e4ee655e486386253fc5b1dcaa2&

Looking through the docs I saw that for intellij based IDEs you can set MIRRORD_ACTIVE to have it always launch for certain configurations. But I can't get it to work with Rider. (In the end I would like to use the config templating, but I figured if I can't get this one to work then probably the templating also won't work). I have added the envvar to my launchSettings.json envvars, which from my understanding is the closest thing to the described "run configuration". At least rider picks it up for the config. Am I missing something here or does this not work for a .NET/Rider setup? https://metalbear.co/mirrord/docs/using-mirrord/intellij-plugin/#enablingdisabling-mirrord-for-a-specific-run-configuration https://cdn.discordapp.com/attachments/1001193615474827264/1362706268279537837/image.png?ex=68035e9c&is=68020d1c&hm=2f38d3c4c37bc57593e61eb3df244e0c30bf1593f163528632be71b193519529& https://cdn.discordapp.com/attachments/1001193615474827264/1362706269470982164/image.png?ex=68035e9c&is=68020d1c&hm=4956ab05523f59772cb9cc4139dcd05607756074cf40827e54ea5e509a863fac&