https://linen.dev logo
Powered by
Hey everyone,
# mirrord-help
k
Hey everyone, my mirrord proxy stopped working today. When spinning up my local debug session, the API returned a temporary redirect, so I investigated further. I found the following entries in the mirrord operator logs:
Copy code
shell
2025-11-20T16:38:20.099316Z ERROR ThreadId(02) operator::shared_session_watcher: 108: mirrordclustersessions watch stream encountered an error error=watch stream failed: ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): storage is (re)initializing: TooManyRequests

2025-11-20T16:39:10.078007Z  WARN ThreadId(02) operator::mutated_pods_cleanup_controller: 205: pods watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof

2025-11-20T16:39:10.095030Z  WARN ThreadId(02) operator::workload_patch_request_controller: 415: mirrordclusterworkloadpatchrequests watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof

2025-11-20T16:39:10.096672Z  WARN ThreadId(02) operator::copied_pods_controller: 172: pods watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof

2025-11-20T16:39:10.107676Z  WARN ThreadId(02) operator::agent_pods_controller: 178: pods watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof

2025-11-20T16:39:10.111854Z  WARN ThreadId(02) operator::workload_patch_controller: 245: mirrordclusterworkloadpatches watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof
I completely uninstalled the operator from the cluster and reinstalled it, but that didn’t resolve the issue. It looks like some kind of “too many requests” error. Could this be coming from the Kubernetes API server? Technical information: • Kubernetes: 1.31.4 • Mirrord-Operator: 3.130.0 • Mirrord: 3.173.2
a
Hey, this usually happens when k8s apiserver is not feeling happy
Any chance your control plane is upgrading?
k
I can look into it
Thank you for the suggestion!
My last resort would be to restart the k8 cluster completely, but i will first try to look into the control plane
I restarted the proxy and the main error did go away indeed but still there are some warnings
Copy code
shell
2025-11-20T16:59:32.132760Z  INFO ThreadId(02) operator_session::logs: 110: port=8080 http_filter=path:^(?!/health/) client_user="system:serviceaccount:admin-namespace:andrei-ziminov" client_name="Elliot Alderson" client_hostname="Elliots-MacBook-Pro.local" client_id="00qyA2AU2e5B1qXkmdixAY7oVZY8gxuOuT0MYDV8gEU" session_id="c4c4239f12c30fb3" target_name="portmanager-backend" target_namespace="portmanager-application" target_type="deployment" Port Steal
2025-11-20T16:59:32.617375Z  WARN ThreadId(02) operator::mutated_pods_cleanup_controller: 205: pods watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof
2025-11-20T16:59:32.618585Z  WARN ThreadId(02) operator::workload_patch_controller: 245: mirrordclusterworkloadpatches watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof
2025-11-20T16:59:32.619208Z  WARN ThreadId(02) operator::workload_patch_request_controller: 415: mirrordclusterworkloadpatchrequests watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof
2025-11-20T16:59:32.640540Z  WARN ThreadId(02) operator::copied_pods_controller: 172: pods watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof
2025-11-20T17:00:04.419327Z  WARN ThreadId(02) operator::target_watch: 171: pods watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof
I think I will restart the complete cluster
a
I'd try to restart operator as last thing before that
k
Quick update: It looks like there is a deeper issue with the control plane. I’ve opened a support ticket, since this is a managed cluster and I don’t have access to the control-plane components. Thank you for your help! In the future, I’ll know to check the control plane first when I see something similar. I’ll post an update here once the issue is resolved.
Hey @Aviram Hassan , we fixed the control plane issue 🙂 but still the proxy has some issues:
Copy code
bash
2025-11-22T19:19:50.163821Z ERROR ThreadId(02) operator_proxy::layer: 106: error=WebSocket protocol error: Connection reset without closing handshake
2025-11-22T19:19:56.710172Z  WARN ThreadId(02) operator::mutated_pods_cleanup_controller: 205: pods watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof
2025-11-22T19:19:56.710501Z  WARN ThreadId(02) operator::workload_patch_controller: 245: mirrordclusterworkloadpatches watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof
2025-11-22T19:19:56.711735Z  WARN ThreadId(02) operator::workload_patch_request_controller: 415: mirrordclusterworkloadpatchrequests watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof
2025-11-22T19:19:56.727725Z  WARN ThreadId(02) operator::copied_pods_controller: 172: pods watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof
2025-11-22T19:20:04.546744Z  INFO ThreadId(02) operator_session::logs: 110: port=8080 http_filter=path:^(?!/health/) client_user="system:serviceaccount:admin-namespace:andrei-ziminov" client_name="xxxxxxxxxx" client_hostname="Elliots-MacBook-Pro.local" client_id="xxxxxxxxxx" session_id="xxxxxxxxxx" target_name="portmanager-backend" target_namespace="portmanager-application" target_type="deployment" Port Steal
2025-11-22T19:20:37.927329Z  WARN ThreadId(02) operator::target_watch: 171: pods watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof
2025-11-22T19
this is my mirrord.json btw
Copy code
json
{
  "agent": {
    "image": "ghcr.io/metalbear-co/mirrord:3.173.2"
  },
  "feature": {
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "path_filter": "^(?!/health/)"
        },
        "port_mapping": [
          [
            5392,
            80
          ]
        ]
      },
      "outgoing": true
    },
    "fs": "read",
    "env": true
  },
  "telemetry": false,
  "kubeconfig": "~/.kube/config",
  "target": {
    "path": "deploy/portmanager-backend/container/portmanager-backend",
    "namespace": "portmanager-application"
  }
}
And again when the mirrord-agent spins up and proxies the traffic i get a temporary redirect.
This one pops up a lot, ```bash pods watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof ```bash
a
What operator version?
Also not sure I understand what is the current symptom?
k
The operator version is on 3.130.0
The symptom is that since upgrading to version 3.130.0, remote debugging has completely stopped working. Even rolling back does not help anymore. The mirrord agent starts and everything looks fine at first. We are debugging a .NET Web API, and normally when starting mirrord, the traffic is correctly redirected to my local machine. However, now when mirrord starts, it effectively chokes off my service, causing the API to return a temporary redirect (I’m not completely sure whether this is just Kubernetes acting up). As soon as I run the service without mirrord, everything becomes available instantly again. What’s odd is that I haven’t changed anything except the operator version.
a
can you try downgrading the operator please?
k
Yeah tried that already didnt help 🙁
Its almost like something broke during the upgrade
for exampe thats something postman shows when the mirrord agent spinns up
a
could it be something with your local app?
mirrord is stateless so that's weird 😮
do you see any logs/errors in the agent spawned?
k
Wait I will take a quick look
a
can you add
Copy code
"internal_proxy": {
        "log_destination": "/tmp/mirrord.log",
        "log_level": "mirrord=trace"
    }
to your mirrord.json then attach 
/tmp/mirrord.log
?
k
sure!
And no in the agent there is no error
only the operator throwing this one everytime I start a session
2025-11-23T16:09:02.385670Z ERROR ThreadId(02) operator_proxy::layer: 106: error=WebSocket protocol error: Connection reset without closing handshake
An the above warnings
But I will try this one quickly
a
seems there's dns errors
k
really ?
like my local dns ?
ahh it seems like he cant wire up the db which lies in another namespace
a
loki-write.monitoring.svc.cluster.local
the remote pod is able to access it?
maybe different settings between local/remote?
k
yeah it is, but i found the same thing for the database. funny thos is that the app is launching successfully
if that would be off the app would crash on startup
the service sits behind a traefik btw
a
does health checks fail?
k
nop and trfk is also not showing any errors
i wonder why the service returns a redirect anyway that doesnt come from us or the application
a
sidecar?
and you said no agent errors?
k
the service does not have sidecars in this setup secifically. but mirrord usually worked even with consul and consul messes the netwoking up pretty much
And no there ar no agent errors
a
wanna share screen?
k
sure
should i call you directly or ypu mean like a sceen shot ? 🙂
a
k
a
Copy code
"target":"mirrord_intproxy::proxies::incoming","filename":"mirrord/intproxy/src/proxies/incoming.rs","line_number":882,"span":{"id":"HttpGatewayId { connection_id: 21, request_id: 0, port: 8080, version: HTTP/1.1 }","is_steal":true,"update":"Message(Http(ResponseFramed(HttpResponse { port: 8080, connection_id: 21, request_id: 0, internal_response: InternalHttpResponse { status: 307, version: HTTP/1.1, headers: 4, body: InternalHttpBody(1 frames) } })))","name":"handle_http_gateway_update"},"spans":[{"name":"incoming_proxy_main_loop"},{"id":"HttpGatewayId { connection_id: 21, request_id: 0, port: 8080, version: HTTP/1.1 }","is_steal":true,"update":"Message(Http(ResponseFramed(HttpResponse { port: 8080, connection_id: 21, request_id: 0, internal_response: InternalHttpResponse { status: 307, version: HTTP/1.1, headers: 4, body: InternalHttpBody(1 frames) } })))","name":"handle_http_gateway_update"}]}
Copy code
{"timestamp":"2025-11-23T16:24:04.622621Z","level":"INFO","fields":{"message":"Received an HTTP response from an HttpGatewayTask","full_headers":"{\"content-type\": \"text/html; charset=utf-8\", \"location\": \"https://api.rest.portmanager.dev.teleconsult.zone35.de/v1/vehicles/applications/all?search=bv-1&searchIn=all&page=1&pageSize=10&sortBy=createdAt&order=desc&include=active&expand=editorMetadata\", \"date\": \"Sun, 23 Nov 2025 16:24:04 GMT\", \"content-length\": \"256\"}","response":"HttpResponse { port: 8080, connection_id: 0, request_id: 0, internal_response: InternalHttpResponse { status: 307, version: HTTP/1.1, headers: 4, body: InternalHttpBody(1 frames) }
k
@Aviram Hassan You’ll probably laugh at this. I had to restart my Mac. It had been running for over 36 days, and when I dug deeper using netcat locally, I was getting strange behavior — mirrord wasn’t showing up and everything felt off. I didn’t really expect a simple reboot to solve it, but I tried it anyway… and of course, problem solved. 😄 A classic Layer 8 issue. Thanks a lot for your support! Honestly, I would’ve been banging my head against this for weeks if you hadn’t pointed out that the request was the issue.
a
you're welcome 🙂 glad it got resolved
PreviousNext
Powered by