https://linen.dev logo
Powered by
Hey guys, since trying a new cloud
# mirrord-help
k
Hey guys, since trying a new cloud vendor we got this error log in the operator
Copy code
2026-01-13T10:06:45.161892Z ERROR ThreadId(02) operator::workload_patch_request_controller: 494: mirrordclusterworkloadpatchrequests watch stream encountered an error error=failed to perform initial object list: ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): storage is (re)initializing: TooManyRequests
2026-01-13T10:06:45.198122Z ERROR ThreadId(02) operator::shared_session_watcher: 262: mirrordclustersessions watch stream encountered an error error=watch stream failed: ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): storage is (re)initializing: TooManyRequests
2026-01-13T10:06:45.216294Z  INFO ThreadId(01) operator_service: 726: Shutting down probe and OpenAPI server to start full API
2026-01-13T10:06:45.217852Z ERROR ThreadId(02) operator::external_resources_controller: 292: mirrordclusterexternalresources watch stream encountered an error error=watch stream failed: ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): storage is (re)initializing: TooManyRequests
2026-01-13T10:06:45.283038Z  INFO ThreadId(01) operator_service: 737: Starting full API server
2026-01-13T10:06:45.988932Z ERROR ThreadId(02) operator::workload_patch_request_controller: 494: mirrordclusterworkloadpatchrequests watch stream encountered an error error=failed to perform initial object list: ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): storage is (re)initializing: TooManyRequests
2026-01-13T10:06:47.624663Z ERROR ThreadId(02) operator::shared_session_watcher: 262: mirrordclustersessions watch stream encountered an error error=failed to perform initial object list: ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }): storage is (re)initializing: TooManyRequests
Is this related to our cluster or indicates a problem in our setup ?
a
hey, this is usually interminnent issue that gets resolved after few seconds
it happens when the k8s control plane is down
k
Ahh okey, I will try to restart the cluster then. This log happens when starting connectiong the agent.
What was the jason part again where i can dump the agent log again ? ๐Ÿ™‚
Copy code
json
{
  "agent": {
    "image": "ghcr.io/metalbear-co/mirrord:3.180.0"
  },
  "feature": {
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "path_filter": "^(?!/health/)"
        },
        "port_mapping": [
          [
            5392,
            8080
          ]
        ]
      },
      "outgoing": true
    },
    "fs": "read",
    "env": true
  },
  "telemetry": false,
  "kubeconfig": "~/.kube/config",
  "target": {
    "path": "deploy/aao-backend/container/aao-backend",
    "namespace": "aao-application"
  }
}
`
I think i have to dig deeper. The agent dies right after the start with this log
Copy code
bash
2026-01-13T10:20:13.905918Z  INFO ThreadId(02) operator_session::logs: 110: client_user="system:serviceaccount:admin-namespace:andrei-ziminov" client_name="Elliot Alderson" client_hostname="Elliots-MacBook-Pro.local" client_id="SVIVGClHdnPP+VMPrzaJggoaW9U3/RjHY64omLvbyns" session_id="52dbc83e224cf9c1" target_name="aao-backend" target_namespace="aao-application" target_type="deployment" Session Start
2026-01-13T10:20:16.553786Z ERROR ThreadId(02) operator_proxy::layer: 108: error=WebSocket protocol error: Connection reset without closing handshake
But befor annoying you guys maybe I can find the issue myself ^^
a
what's your new loud provider?
k
STACKIT
German private cloud
a
maybe it has some default security limitiations?
k
Like network policies ?
a
yes
or something is deleting the pod
k
very weird indeed the agent looks healthly as well, no error logs
But i will dig deeper ๐Ÿ™‚
At least i found out why the agent is dissapearing. its because the app creshe wih this log.
Copy code
bash
Unhandled exception. Npgsql.NpgsqlException (0x80004005): Failed to connect to 100.82.65.20:5432
 ---> System.Net.Sockets.SocketException (51): Network is unreachable
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
   at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
   at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Migrations.HistoryRepository.GetAppliedMigrationsAsync(CancellationToken cancellationToken)
   at Npgsql.EntityFrameworkCore.PostgreSQL.Migrations.Internal.NpgsqlHistoryRepository.GetAppliedMigrationsAsync(CancellationToken cancellationToken)
   at Npgsql.EntityFrameworkCore.PostgreSQL.Migrations.Internal.NpgsqlMigrator.MigrateAsync(String targetMigration, CancellationToken cancellationToken)
   at Program.<Main>$(String[] args) in /Volumes/T9/Development/Namespaces/ILS-Karlsruhe/Projects/AAO-Portal/aao-portal-backend/src/AAOPortal.Api/Program.cs:line 48
   at Program.<Main>$(String[] args) in /Volumes/T9/Development/Namespaces/ILS-Karlsruhe/Projects/AAO-Portal/aao-portal-backend/src/AAOPortal.Api/Program.cs:line 50
   at Program.<Main>(String[] args)
seems that it resolves the ip from the connection string but has troubles to connect to it
a
oh
can you add agent.extraConfig.privileged: true
maybe host is more protected ๐Ÿ™‚
k
Copy code
json
{
  "agent": {
    "image": "ghcr.io/metalbear-co/mirrord:3.180.0",
    "extraConfig": {
      "privileged": true
    }
  },
  "feature": {
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "path_filter": "^(?!/health/)"
        },
        "port_mapping": [
          [
            5392,
            8080
          ]
        ]
      },
      "outgoing": true
    },
    "fs": "read",
    "env": true
  },
  "telemetry": false,
  "kubeconfig": "~/.kube/config",
  "target": {
    "path": "deploy/aao-backend/container/aao-backend",
    "namespace": "aao-application"
  }
}
like this? its telling me Property 'extraConfig' is not allowed
a
no, in the operator helm chart values
k
ahh
sure
ouh now i get thsi one ๐Ÿ˜ฎ
Copy code
bash
2026-01-13T12:33:26.377574Z  WARN ThreadId(02) operator_session: 148: error=failed to create the session resource: ApiError: "404 page not found\n": Failed to parse error data (ErrorResponse { status: "404 Not Found", message: "\"404 page not found\\n\"", reason: "Failed to parse error data", code: 404 })
2026-01-13T12:33:54.199446Z ERROR ThreadId(02) operator::shared_session_watcher: 262: mirrordclustersessions watch stream encountered an error error=watch stream failed: Error deserializing response: invalid type: integer `404`, expected adjacently tagged enum WatchEvent at line 1 column 3: Error deserializing response: invalid type: integer `404`, expected adjacently tagged enum WatchEvent at line 1 column 3: invalid type: integer `404`, expected adjacently tagged enum WatchEvent at line 1 column 3
2026-01-13T12:34:01.817035Z  WARN ThreadId(02) operator::mutated_pods_cleanup_controller: 207: pods watch stream encountered an error error=watch stream failed: Error reading events stream: ServiceError: error reading a body from connection: Error reading events stream: ServiceError: error reading a body from connection: ServiceError: error reading a body from connection: error reading a body from connection: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof
20
this one comes also up in the Jetbrains Mirrord plugin "WARN ThreadId(02) operator_session: 148: error=failed to create the session resource: ApiError: "404 page not found\n": Failed to parse error data (ErrorResponse { status: "404 Not Found", message: "\"404 page not found\\n\"", reason: "Failed to parse error data", code: 404 })"
a
are you on old cli?
k
whats is that ? ๐Ÿ™‚
a
nvm
it happens everytime, this error?
k
Yes
since i added the extra config
Copy code
yaml
  - name: mirrord-operator
    namespace: mirrord
    chart: metalbear/mirrord-operator
    version: 1.43.0
    values:
      - agent:
          extraConfig:
            privileged: true
      - createNamespace: false
      - roleNamespaces:
          - aao-application
          - aao-infrastructure
      - license:
          key: "REDACTED"
The error comes also when running the mirrord agent per command
Copy code
bash
MIRRORD_INTPROXY_LOG_LEVEL="mirrord=trace" mirrord exec -n aao-application --target deploy/aao-backend -- dotnet run --project src/AAOPortal.Api/AAOPortal.Api.csproj
Copy code
bash
 mirrord exec (3.181.0)
    โœ“ running on latest!
  โ ฒ preparing to launch process
      โœ“ layer extracted
    โ ฒ checking operator
        โœ“ operator license valid
        โœ“ user credentials prepared
      โ ‚ starting session
        โ ’ connecting to the target                                                                                                                                                                                       2026-01-13T12:48:58.297610Z ERROR kube_client::client::builder: failed with status 500 Internal Server Error
โ   mirrord exec (3.181.0)
    โœ“ running on latest!
  โ ฒ preparing to launch process
      โœ“ layer extracted
    โ ฒ checking operator
        โœ“ operator license valid
        โœ“ user credentials prepared
โ   mirrord exec (3.181.0)
    โœ“ running on latest!
  โ ฒ preparing to launch process
      โœ“ layer extracted
      x checking operator
        โœ“ operator license valid
        โœ“ user credentials prepared
  x mirrord exec (3.181.0)
    โœ“ running on latest!
    x preparing to launch process
      โœ“ layer extracted
      x checking operator
        โœ“ operator license valid
        โœ“ user credentials prepared
        x starting session
          x connecting to the target                                                                                                                                                                                   Error:   ร— mirrord operator API failed: creating a websocket connection failed with ApiError: Failed to acquire the session: failed to create the session resource: ApiError: "404 page not found\n": Failed to parse error
  โ”‚ data (ErrorResponse { status: "404 Not Found", message: "\"404 page not found\\n\"", reason: "Failed to parse error data", code: 404 }): 500 Internal Server Error (ErrorResponse { status: "Failure", message:
  โ”‚ "Failed to acquire the session: failed to create the session resource: ApiError: \"404 page not found\\n\": Failed to parse error data (ErrorResponse { status: \"404 Not Found\", message: \"\\\"404 page not
  โ”‚ found\\\\n\\\"\", reason: \"Failed to parse error data\", code: 404 })", reason: "500 Internal Server Error", code: 500 })
  help: Please check the following:
            1. The operator is running and the logs are not showing any errors.
            2. You have sufficient permissions to port forward to the operator.
        
            If you want to run without the operator, please set `"operator": false` in the mirrord configuration file.
        
            Please remember that some features are supported only when using mirrord operator (https://metalbear.com/mirrord/docs/overview/teams?utm_source=erropfailed&utm_medium=cli#supported-features).
        
        - If you're still stuck:
        
        >> Please open a new bug report at https://github.com/metalbear-co/mirrord/issues/new/choose
        
        >> Or join our Slack https://metalbear.com/slack and request help in #mirrord-help
        
        >> Or email us at hi@metalbear.com
a
do you see the agent pod being created?
perhaps it's again admission?
k
No this time it straight up crashes without creating the agent
a
can you share logs from the operator?
k
s
This works Plain Python socket connect succeeds every time:
Copy code
bash
mirrord exec -n aao-application --target deploy/aao-backend -- \
  bash -lc 'python3 - <<PY
import socket
print("resolving...")
addr = socket.getaddrinfo("postgresql-primary.aao-infrastructure.svc.cluster.local", 5432)[0][4]
print("addr:", addr)
s=socket.socket()
s.settimeout(3)
print("connecting...")
s.connect(addr)
print("CONNECTED OK")
PY'
Output:
Copy code
bash
addr: ('100.82.65.20', 5432)
CONNECTED OK
So: โ€ข DNS works โ€ข TCP connectivity from the agentโ€™s network context works โ€ข The DB is reachable โธป โŒ This does not work nc (and also .NET / Npgsql) fail with Network is unreachable:
Copy code
bash
mirrord exec -n aao-application --target deploy/aao-backend -- \
  bash -lc 'nc -vz postgresql-primary.aao-infrastructure.svc.cluster.local 5432'
Result:
Copy code
bash
nc: connectx to postgresql-primary.aao-infrastructure.svc.cluster.local port 5432 (tcp) failed: Network is unreachable
Same error when running a local .NET 10 app (Npgsql) via mirrord. โธป ๐Ÿ” Key observations โ€ข The agent is not being killed โ€” it shuts down because the local app exits after the DB connection failure. โ€ข Pythonโ€™s socket.connect() succeeds, but: โ€ข nc โ€ข .NET (Npgsql) fail consistently. โ€ข This strongly suggests a syscall / socket path difference (e.g. connectx, non-blocking connect, happy-eyeballs, etc.). โ€ข On macOS, tools like nc and .NET often use connectx, whereas Python uses a simpler connect() path. โ€ข When mirrord doesnโ€™t intercept that path, the connection falls back to the local host network โ€” which obviously canโ€™t route to 100.82.x.x. โธป ๐Ÿง  Working assumption mirrord on macOS does not fully intercept certain outbound TCP paths (likely connectx / nonblocking connect), especially in non-privileged mode. That would explain: โ€ข Why Python works โ€ข Why nc + .NET fail โ€ข Why DNS resolution still succeeds โ€ข Why the behavior is consistent and reproducible โธป ๐Ÿงช Things we tried / confirmed โ€ข No NetworkPolicies blocking traffic โ€ข Same mirrord config works on another cluster โ€ข Agent pod can reach DB (verified) โ€ข Problem reproduces even with:
Copy code
json
"feature": {
  "network": {
    "outgoing": false
  }
}
โ€ข mirrord internal proxy logs show clean startup, no denial errors โ€” just no interception for the failing path โธป โ“ Questions 1. Is connectx fully supported on macOS ARM in non-privileged mode? 2. Is this a known limitation / documented caveat? 3. Is there a recommended workaround for .NET / nc users (flags, env vars, config)? 4. Would forcing blocking connect() be expected to fix this?
I polished my text with chatgpt so no technical spec gets missed ๐Ÿ™‚
a
but you're showing that it fails before the process runs
in the cases chatgpt generated the process runs
hence agent/operator suceeeds
k
Teh above problem is the previos one without
Copy code
yaml
      - agent:
          extraConfig:
            privileged: true
i will send you the operator logs in a second
This is now with
Copy code
yaml
      - agent:
          extraConfig:
            privileged: true
No i created the cli command i just used chat gpt to polish the text ๐Ÿ™‚
The debugging worked on .NET 9 and still works. The only difference in this project is that we jumped to .NET 10 and for triubleshooting reasons i tried a simple connection with python
Because without this part
Copy code
yaml
      - agent:
          extraConfig:
            privileged: true
The agent spins up but gets shut down because the app cant connect to the database and shuts down. I assume the IDE plugin then keills the agent
the last time i fixed a similiar issue by restarting my mac & cluster. I did this this time right from the beginning and tested it on my linux machine as well and get the same error
r
Can you try a couple things please: 1. 
kubectl get mirrordclustersessions -o yaml
2. 
mirrord exec -n aao-application --target deploy/aao-backend -- bash -lc 'nc -vz 100.82.65.20 5432'
k
yes sure!
Copy code
bash
kubectl get mirrordclustersessions -o yaml
apiVersion: v1
items:
- apiVersion: mirrord.metalbear.co/v1alpha
  kind: MirrordClusterSession
  metadata:
    annotations:
      operator.metalbear.co/instance_id: 2e7f90d3-ad69-48c1-9ec6-60231004d38f
    creationTimestamp: "2026-01-13T13:17:31Z"
    finalizers:
    - operator.metalbear.co/sessions-controller
    - operator.metalbear.co/sessions-ttl-controller
    generation: 1
    labels:
      operator.metalbear.co/owner: mirrord-operator
    name: cc7c10a48c49b04c
    resourceVersion: "22310871"
    uid: 0c552ae1-3f9c-4f26-9dbe-c5cd5ae99d17
  spec:
    jiraMetrics:
      branchName: dev
    namespace: aao-application
    owner:
      hostname: Elliots-MacBook-Pro.local
      k8sUsername: system:serviceaccount:admin-namespace:andrei-ziminov
      userId: SVIVGClHdnPP+VMPrzaJggoaW9U3/RjHY64omLvbyns
      username: Elliot Alderson
    target:
      apiVersion: apps/v1
      container: aao-backend
      kind: Deployment
      name: aao-backend
  status:
    connectedTimestamp: "2026-01-13T13:23:13.166615Z"
kind: List
metadata:
  resourceVersion: ""
Copy code
bash
mirrord exec -n aao-application --target deploy/aao-backend -- bash -lc 'nc -vz 100.82.65.20 5432'
* Running command: bash -lc nc -vz 100.82.65.20 5432
* mirrord will target: deployment/aao-backend, no configuration file was loaded
* mirrord will run with the mirrord Operator
* env: all remote environment variables will be fetched
* fs: file operations will default to read from the remote
* incoming: incoming traffic will be mirrored
* outgoing: forwarding is enabled on TCP and UDP
* dns: DNS will be resolved remotely
* internal proxy: logs will be written to /var/folders/dz/z4gwwfvd6y94p_plpwcj4lbc0000gn/T/mirrord-intproxy-1768310635-xjXd9qX.log
* key: b49f5234-59bd-478d-8d4d-bf80671139d1
โ  mirrord exec (3.181.0)
    โœ“ running on latest!
    โœ“ ready to launch process
      โœ“ layer extracted
      โœ“ using operator
        โœ“ operator license valid
        โœ“ user credentials prepared
        โœ“ session started
          โœ“ connected to the target
      โœ“ arm64 layer library extracted
    โœ“ config summary
    โœ“ Ready!                                                                                                                                                                                                                                                              /Users/elliot/.profile: line 6: /Users/elliot-mac/.rover/env: No such file or directory
/Users/elliot/.bash_profile: line 16: /Users/elliot-mac/.rover/env: No such file or directory
nc: connectx to 100.82.65.20 port 5432 (tcp) failed: Network is unreachable
I think thtas the core issue
but the pod/container itself can communicate with the dabase
Local Machine -> Mirrord -> My Contianer -x-> Databse
The IP 100.82.65.20 gets resolved from this connection string Host=postgresql-primary.aao-infrastructure.svc.cluster.local;Port=5432;Database=aao;Username=postgres;Password=REDACTED;
Copy code
bash
kubectl describe svc/postgresql-primary -n aao-infrastructure
Name:                     postgresql-primary
Namespace:                aao-infrastructure
Labels:                   app.kubernetes.io/component=primary
                          app.kubernetes.io/instance=postgresql
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=postgresql
                          app.kubernetes.io/version=17.6.0
                          helm.sh/chart=postgresql-16.7.27
Annotations:              meta.helm.sh/release-name: postgresql
                          meta.helm.sh/release-namespace: aao-infrastructure
Selector:                 app.kubernetes.io/component=primary,app.kubernetes.io/instance=postgresql,app.kubernetes.io/name=postgresql
Type:                     ClusterIP
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       100.82.65.20
IPs:                      100.82.65.20
Port:                     tcp-postgresql  5432/TCP
TargetPort:               tcp-postgresql/TCP
Endpoints:                100.64.0.14:5432
Session Affinity:         None
Internal Traffic Policy:  Cluster
Events:                   <none>
r
Ok, can you try with intproxy logging now? You'd need to use this snippet as mirrord config - 
mirrord exec -n aao-application --target deploy/aao-backend -f ./path/to/file/with/snippet.json -- bash -lc 'nc -vz 100.82.65.20 5432'
Copy code
{
  "internal_proxy": {
    "log_level": "mirrord=trace",
    "log_destination": "/path/to/some/log/directory"
  }
}
k
sure
r
And I'll need that logfile ๐Ÿ˜…
Thanks!
Huh, what is your 
nc
version?
k
ehh let me look that up
one sec
a
that's with the privileged?
k
no
with is without
i can post a log with privileged
second
a
@razz4780 I think node is more "secure" so need privleged agent (we get blocked on access to /etc/hosts)
r
In the logfile I see DNS success though ๐Ÿ‘€
k
Yeah that works !
Thts the part whihc confuses me the most ๐Ÿ˜„
@Aviram Hassan When running with priviled access it doesnt generate a log file
Copy code
bash
aao-portal-backend git:(dev) โœ— mirrord exec -n aao-application --target deploy/aao-backend -f ./snippet.json -- bash -lc 'nc -vz 100.82.65.20 5432'
โ „ mirrord exec (3.181.0)
    โœ“ running on latest!
  โ ’ preparing to launch process
      โœ“ layer extracted
    โ ‚ checking operator
        โœ“ operator license valid
        โœ“ user credentials prepared
      โ ‚ starting session
        โ š connecting to the target                                                                                                                                                                                       2026-01-13T13:43:34.446219Z ERROR kube_client::client::builder: failed with status 500 Internal Server Error
โ ค mirrord exec (3.181.0)
    โœ“ running on latest!
  โ ’ preparing to launch process
      โœ“ layer extracted
    โ ’ checking operator
        โœ“ operator license valid
        โœ“ user credentials prepared
โ ค mirrord exec (3.181.0)
    โœ“ running on latest!
  โ ’ preparing to launch process
      โœ“ layer extracted
      x checking operator
        โœ“ operator license valid
        โœ“ user credentials prepared
  x mirrord exec (3.181.0)
    โœ“ running on latest!
    x preparing to launch process
      โœ“ layer extracted
      x checking operator
        โœ“ operator license valid
        โœ“ user credentials prepared
        x starting session
          x connecting to the target                                                                                                                                                                                   Error:   ร— mirrord operator API failed: creating a websocket connection failed with ApiError: Failed to acquire the session: failed to create the session resource: ApiError: "404 page not found\n": Failed to parse error
  โ”‚ data (ErrorResponse { status: "404 Not Found", message: "\"404 page not found\\n\"", reason: "Failed to parse error data", code: 404 }): 500 Internal Server Error (ErrorResponse { status: "Failure", message:
  โ”‚ "Failed to acquire the session: failed to create the session resource: ApiError: \"404 page not found\\n\": Failed to parse error data (ErrorResponse { status: \"404 Not Found\", message: \"\\\"404 page not
  โ”‚ found\\\\n\\\"\", reason: \"Failed to parse error data\", code: 404 })", reason: "500 Internal Server Error", code: 500 })
  help: Please check the following:
            1. The operator is running and the logs are not showing any errors.
            2. You have sufficient permissions to port forward to the operator.
        
            If you want to run without the operator, please set `"operator": false` in the mirrord configuration file.
        
            Please remember that some features are supported only when using mirrord operator (https://metalbear.com/mirrord/docs/overview/teams?utm_source=erropfailed&utm_medium=cli#supported-features).
        
        - If you're still stuck:
        
        >> Please open a new bug report at https://github.com/metalbear-co/mirrord/issues/new/choose
        
        >> Or join our Slack https://metalbear.com/slack and request help in #mirrord-help
        
        >> Or email us at hi@metalbear.com
it crashes right away without creating the agent at all
r
This is a very weird error. Can you do 
kubectl get mirrordclustersessions
again?
k
yup
kubectl get mirrordclustersessions Error from server (NotFound): Unable to list "mirrord.metalbear.co/v1alpha, Resource=mirrordclustersessions": the server could not find the requested resource (get mirrordclustersessions.mirrord.metalbear.co)
Thats with priviliged: true
r
How do you switch between privileged and unprivileged?
k
Copy code
yaml
  - name: mirrord-operator
    namespace: mirrord
    chart: metalbear/mirrord-operator
    version: 1.43.0
    values:
      - agent:
          extraConfig:
            privileged: true
      - createNamespace: false
      - roleNamespaces:
          - aao-application
          - aao-infrastructure
      - license:
          key: "REDACTED"
I reinstall the operator
r
Interesting. Looks like your cluster needs some time to sync custom resource definitions. 
kubectl get mirrordclustersessions
should eventually succeed ๐Ÿ‘€
k
helm uninstall mirrord-operator -n mirrord then comment in
Copy code
yaml
      - agent:
          extraConfig:
            privileged: true
and then helmfile -e prod -l name=mirrord-operator sync -f ./debugging.helmfile.gotmpl
let me reinstall it with some time between it. Sometimes kubernetes has race conditions when doing it to fast ๐Ÿ˜„
kubectl get mirrordclustersessions NAME USER ID USERNAME HOSTNAME K8S USER NAMESPACE TARGET STARTED AT CLOSED AT CLOSE REASON cb9aec822ea29dac SVIVGClHdnPP+VMPrzaJggoaW9U3/RjHY64omLvbyns Elliot Alderson Elliots-MacBook-Pro.local system:serviceaccount:admin-namespace:andrei-ziminov aao-application {"apiVersion":"apps/v1","container":"aao-backend","kind":"Deployment","name":"aao-backend"} 14s
Copy code
bash
kubectl get mirrordclustersessions                                           
NAME               USER ID                                       USERNAME          HOSTNAME                    K8S USER                                               NAMESPACE         TARGET                                                                                        STARTED AT   CLOSED AT   CLOSE REASON
cb9aec822ea29dac   SVIVGClHdnPP+VMPrzaJggoaW9U3/RjHY64omLvbyns   Elliot Alderson   Elliots-MacBook-Pro.local   system:serviceaccount:admin-namespace:andrei-ziminov   aao-application   {"apiVersion":"apps/v1","container":"aao-backend","kind":"Deployment","name":"aao-backend"}   14s
@Aviram Hassan Now the 404 error is gone
Copy code
bash
mirrord exec -n aao-application --target deploy/aao-backend -f ./snippet.json -- bash -lc 'nc -vz 100.82.65.20 5432'
* Running command: bash -lc nc -vz 100.82.65.20 5432
* mirrord will target: deployment/aao-backend, the configuration file was loaded from ./snippet.json
* mirrord will run with the mirrord Operator
* env: all remote environment variables will be fetched
* fs: file operations will default to read from the remote
* incoming: incoming traffic will be mirrored
* outgoing: forwarding is enabled on TCP and UDP
* dns: DNS will be resolved remotely
* internal proxy: logs will be written to ./mirrord-intproxy-1768312303-fCRs7Un.log
* key: f971c350-ca20-4546-a773-89f2352a61ca
โ š mirrord exec (3.181.0)
    โœ“ running on latest!
    โœ“ ready to launch process
      โœ“ layer extracted
      โœ“ using operator
        โœ“ operator license valid
        โœ“ user credentials prepared
        โœ“ session started
          โœ“ connected to the target
      โœ“ arm64 layer library extracted
    โœ“ config summary
    โœ“ Ready!                                                                                                                                                                                                           /Users/elliot/.profile: line 6: /Users/elliot-mac/.rover/env: No such file or directory
/Users/elliot/.bash_profile: line 16: /Users/elliot-mac/.rover/env: No such file or directory
nc: connectx to 100.82.65.20 port 5432 (tcp) failed: Network is unreachable
And here is the log ๐Ÿ™‚
r
Nothing significant changed, the issue is not related to the incluster part of mirrord. It must be something local. I'll check the source code and let you know
k
Thank you. You guys rock! @Aviram Hassan @razz4780
Idk if that helps but this works @razz4780
Copy code
bash
mirrord exec -n aao-application --target deploy/aao-backend -- \                                                    
  bash -lc 'python3 - <<PY                                                                                                                                
import socket
print("resolving...")
addr = socket.getaddrinfo("postgresql-primary.aao-infrastructure.svc.cluster.local", 5432)[0][4]
print("addr:", addr)
s=socket.socket()
s.settimeout(3)
print("connecting...")
s.connect(addr)
print("CONNECTED OK")
PY'
Copy code
bash
โžœ  aao-portal-backend git:(dev) โœ— mirrord exec -n aao-application --target deploy/aao-backend -f ./snippet.json -- \
  bash -lc 'python3 - <<PY
import socket
print("resolving...")
addr = socket.getaddrinfo("postgresql-primary.aao-infrastructure.svc.cluster.local", 5432)[0][4]
print("addr:", addr)
s=socket.socket()
s.settimeout(3)
print("connecting...")
s.connect(addr)
print("CONNECTED OK")
PY'
* Running command: bash -lc python3 - <<PY
import socket
print("resolving...")
addr = socket.getaddrinfo("postgresql-primary.aao-infrastructure.svc.cluster.local", 5432)[0][4]
print("addr:", addr)
s=socket.socket()
s.settimeout(3)
print("connecting...")
s.connect(addr)
print("CONNECTED OK")
PY
* mirrord will target: deployment/aao-backend, the configuration file was loaded from ./snippet.json
* mirrord will run with the mirrord Operator
* env: all remote environment variables will be fetched
* fs: file operations will default to read from the remote
* incoming: incoming traffic will be mirrored
* outgoing: forwarding is enabled on TCP and UDP
* dns: DNS will be resolved remotely
* internal proxy: logs will be written to ./mirrord-intproxy-1768313105-cqb4DGx.log
* key: 6fb5f1a0-0b48-44e3-b00c-89701719f7e0
โ ’ mirrord exec (3.181.0)
    โœ“ running on latest!
    โœ“ ready to launch process
      โœ“ layer extracted
      โœ“ using operator
        โœ“ operator license valid
        โœ“ user credentials prepared
        โœ“ session started
          โœ“ connected to the target
      โœ“ arm64 layer library extracted
    โœ“ config summary
    โœ“ Ready!                                                                                                                                                                                                                       /Users/elliot/.profile: line 6: /Users/elliot-mac/.rover/env: No such file or directory
/Users/elliot/.bash_profile: line 16: /Users/elliot-mac/.rover/env: No such file or directory
resolving...
addr: ('100.82.65.20', 5432)
connecting...
CONNECTED OK
@razz4780 i got somenews
Copy code
bash
mirrord exec -n aao-application --target deploy/aao-backend -f ./snippet.json -- bash -lc 'nc -4 -O -vz 100.82.65.20 5432'
* Running command: bash -lc nc -4 -O -vz 100.82.65.20 5432
* mirrord will target: deployment/aao-backend, the configuration file was loaded from ./snippet.json
* mirrord will run with the mirrord Operator
* env: all remote environment variables will be fetched
* fs: file operations will default to read from the remote
* incoming: incoming traffic will be mirrored
* outgoing: forwarding is enabled on TCP and UDP
* dns: DNS will be resolved remotely
* internal proxy: logs will be written to ./mirrord-intproxy-1768314588-koReedJ.log
* key: 689cc6c6-ad47-4c7b-8b47-c1b693a0b3e8
โ ‹ mirrord exec (3.181.0)
    โœ“ running on latest!
    โœ“ ready to launch process
      โœ“ layer extracted
      โœ“ using operator
        โœ“ operator license valid
        โœ“ user credentials prepared
        โœ“ session started
          โœ“ connected to the target
      โœ“ arm64 layer library extracted
    โœ“ config summary
    โœ“ Ready!                                                                                                                                                                                                                       /Users/elliot/.profile: line 6: /Users/elliot-mac/.rover/env: No such file or directory
/Users/elliot/.bash_profile: line 16: /Users/elliot-mac/.rover/env: No such file or directory
Connection to 100.82.65.20 port 5432 [tcp/postgresql] succeeded!
it works with those options
-4 -O
that is what chatgpt gave me. Because nc on macOS was not actually using the normal connect() syscall before โ€” it was using Appleโ€™s newer connectx() path, and mirrordโ€™s macOS layer wasnโ€™t catching (or wasnโ€™t able to fully reroute) that call. What changed with your working command: 1) -O forces โ€œold-style connectโ€ On Appleโ€™s nc, -O means: donโ€™t use connectx, use the classic connect(). โ€ข Without -O: nc โ†’ connectx() โ†’ bypasses mirrord interception โ†’ your Mac tries to route 100.82.65.20 locally โ†’ Network is unreachable โ€ข With -O: nc โ†’ connect() โ†’ mirrord intercepts โ†’ tunnels through the agent โ†’ succeeds This matches your earlier observation: โ€ข Python socket connect worked (uses connect()) โ€ข nc failed (used connectx()) 2) -4 removes IPv6 ambiguity Even if your DNS resolves to IPv4, some clients still try IPv6 first or create dual-stack sockets. -4 forces IPv4-only, which removes another macOS networking path that might not be hooked the same way. 3) You connected to the IP directly In the working command you used 100.82.65.20 directly. That avoids any extra resolver behavior and makes the test purely โ€œcan we intercept and route this TCP connectโ€. โธป So the โ€œwhyโ€ in one sentence mirrord on macOS reliably hooks classic connect() calls, but Appleโ€™s connectx() (used by default by /usr/bin/nc and often by .NET/Npgsql) can bypass the hook โ€” so forcing old-style connect (-O) makes mirrord able to proxy the traffic. โธป What this implies for your .NET/Npgsql issue Your Npgsql failure is likely the same root cause: .NET on macOS is probably using connectx() internally, so the DB connect bypasses mirrord and fails locally. Practical next steps โ€ข Easiest workaround: run the debug session in an environment that doesnโ€™t use connectx() (Linux container / devcontainer). โ€ข Or: see if mirrord has a config/flag/workaround for macOS connectx hooking (this is exactly what to tell their devs). If you paste your Discord draft, Iโ€™ll tighten it into a crisp โ€œrepro + expected/actual + hypothesisโ€ message.
r
Huh! Nice, this is very helpful ^^ Thank you!
k
I will also dig deeper ๐Ÿ™‚
r
You can try with just 
-O
, it should work
We're indeed missing a hook for 
connectx()
, we'll need to add it
k
Yup worked!
Okey, glad that we found the issue ๐Ÿ˜„
That explians pretty much why it works with .NET9 and not with .NET10 anymore
So then I will look for a workaround. What release cycle do you guys have so I can makr it in my calender ๐Ÿ˜„
Until this is fixed upstream, I found a workaround that others on macOS can use as well (tested with .NET 10, but this should apply to other runtimes using the same networking behavior). The idea is based on this doc: https://metalbear.com/mirrord/docs/using-mirrord/local-container Instead of running the app directly on macOS, we run it inside a Linux container and let mirrord proxy the containerโ€™s network traffic. Helper shell script
Copy code
bash
#!/usr/bin/env bash
set -euo pipefail

IMAGE_NAME="aao-backend:debug"
DOCKERFILE="src/AAOPortal.Api/Dockerfile.debug"
CONTAINER_NAME="aao-backend"

MIRRORD_CONFIG="src/AAOPortal.Api/mirrord.json"
NAMESPACE="aao-application"
TARGET="deploy/aao-backend"

PORT="5392"

echo "โ–ถ Building image: ${IMAGE_NAME}"
docker build -t "${IMAGE_NAME}" -f "${DOCKERFILE}" .

echo "โ–ถ Starting container via mirrord (port ${PORT})"
exec mirrord container \
  -f "${MIRRORD_CONFIG}" \
  -n "${NAMESPACE}" \
  --target "${TARGET}" \
  -- \
  docker run --rm \
    --name "${CONTAINER_NAME}" \
    "${IMAGE_NAME}"
Minimal Dockerfile.debug
Copy code
Dockerfile
FROM mcr.microsoft.com/dotnet/sdk:10.0.101-azurelinux3.0
WORKDIR /src

COPY . ./

ENV ASPNETCORE_URLS=http://0.0.0.0:5392
ENV ASPNETCORE_ENVIRONMENT=debug

EXPOSE 5392

CMD ["dotnet", "run", "--project", "src/AAOPortal.Api/AAOPortal.Api.csproj", "-c", "Debug", "--no-launch-profile"]
How this is used 1. Run the shell script 2. The app starts inside a Linux container via mirrord container 3. Attach your local Docker debugger (Rider in my case) to the running container 4. Debug as usual Why this works On macOS, newer runtimes (including .NET 10) rely heavily on connectx() and other Apple-specific networking APIs. as stated by @razz4780 mirrord currently has a missing / incomplete hook for connectx, which causes outbound connections to fail or the agent to shut down. By running the app inside a Linux container, the runtime uses standard Linux syscalls (connect, socket, etc.) instead of macOS networking primitives. mirrord can fully intercept and proxy those calls โ€” so networking behaves correctly again.
Even with all that said: this is a fairly dirty workaround ๐Ÿ˜„ But until the connectx hook is fixed, itโ€™s a practical way to keep moving.
y
Hi @K3RN3L_P4N1C , I have added 
connectx()
detour in mirrord which works in my test using 
nc
and a C test program. It is released in 3.183.0. Can you try it and let me know if the issue is resolved?
k
Hey @Yuan I will look directly into it ๐Ÿ™‚
Hey Guys, sorry for the late response. We had a painful release....
@Yuan Unfortunately it doesnt work on mirrord:3.183.0 either...
I will scrap some proxy logs for you
Slowly i am getting also a little bit preassure from my team cause i introduced mirrord first and we have theis problem now on all developer machines
Around 15 people have the same problem
y
> Unhandled exception. Npgsql.NpgsqlException (0x80004005): Failed to connect to 100.82.65.20:5432 You shared this error before. With >= 3.183.0, is the app fail with the same error? I will look into Npgsql more today to see what's missing. Sorry for the trouble, will try to unblock you asap.
k
@Yuan But your changes made indeed an impact
Copy code
bash
โžœ  aao-portal-backend git:(userstory/344-file-management) โœ— mirrord exec -n aao-application --target deploy/aao-backend -f ./snippet.json -- bash -lc 'nc -vz 100.82.65.20 5432' 
* Running command: bash -lc nc -vz 100.82.65.20 5432
* mirrord will target: deployment/aao-backend, the configuration file was loaded from ./snippet.json
* mirrord will run with the mirrord Operator
* env: all remote environment variables will be fetched
* fs: file operations will default to read from the remote
* incoming: incoming traffic will be mirrored
* outgoing: forwarding is enabled on TCP and UDP
* dns: DNS will be resolved remotely
* internal proxy: logs will be written to ./mirrord-intproxy-1769619330-nFO7UhE.log
* key: f08037c2-0e57-4951-a4ae-ed18dbdc8957
โ ˆ mirrord exec (3.183.0)
    โœ“ running on latest!
    โœ“ ready to launch process
      โœ“ layer extracted
      โœ“ using operator
        โœ“ operator license valid
        โœ“ user credentials prepared
        โœ“ session started
          โœ“ connected to the target
      โœ“ arm64 layer library extracted
    โœ“ config summary
    โœ“ Ready!                                                                                                                                                                                                                       /Users/elliot/.profile: line 6: /Users/elliot-mac/.rover/env: No such file or directory
/Users/elliot/.bash_profile: line 16: /Users/elliot-mac/.rover/env: No such file or directory
Connection to 100.82.65.20 port 5432 [tcp/postgresql] succeeded!
nc now works. we have now a differrent kind of issue here i thinkg
a
can I give you a binary to test?
k
A binary of the .NET application ?
a
nvm - we reproduced and we're on it.
k
Thank you! If I can be of any help just ping me ๐Ÿ™‚
@Yuan @Aviram Hassan Hey guys, I have another piece of information that might be helpful. A colleague of mine confirmed that it definitely works with .NET 9 and earlier versions. .NET 10, however, does not work.
a
yes, we know, thank you
we're debugging it
k
Alright, did not wanted to rush ๐Ÿ™‚ Just thought it could help
a
no all good thank you
y
Root cause found: https://github.com/dotnet/runtime/pull/117330 .NET 10 use 
$NOCANCEL
variants. We are working on adding missing hooks.
k
Amazing, thank you for the update @Yuan !
y
The fix was merged, and we should be able to release for you to test it early next week ๐Ÿ™
k
You guys rock! Thank you, iI will check it out instantly when it is released!
y
Hi @K3RN3L_P4N1C , we just released mirrord 3.185.0. I did some tests and verified fetching remote env, FS operation, outgoing network with Npgsql all function normally with .NET 10. Can you give it a shot when you have a chance please?
k
Hey @Yuan i will check it out today ! Thank you for the update ๐Ÿ™‚
I will also test stuff like AMQP & GRPC and will give you a feed back
4 Views
PreviousNext
Powered by