https://linen.dev logo
b
Hi, I have been trying to resolve some issues where connection drops. I have had limited success by doing this - https://metalbear.co/mirrord/docs/faq/common-issues#agent-connection-fails-or-drops-when-using-an-ephemeral-agent-with-a-service-mesh The first time it works, but the second time I get this.
Copy code
{
  "timestamp": "2025-07-18T05:54:45.794723Z",
  "level": "ERROR",
  "fields": {
    "error": "io error: Address already in use (os error 98)"
  },
  "target": "mirrord_agent::entrypoint",
  "span": {
    "args": "Args { mode: Ephemeral { mesh: None }, communicate_port: 51234, communication_timeout: 360, network_interface: None, metrics: None, test_error: false, operator_tls_cert_pem: None, is_mesh: true, ipv6: false }",
    "name": "start_agent"
  },
  "spans": [
    {
      "args": "Args { mode: Ephemeral { mesh: None }, communicate_port: 51234, communication_timeout: 360, network_interface: None, metrics: None, test_error: false, operator_tls_cert_pem: None, is_mesh: true, ipv6: false }",
      "name": "start_agent"
    }
  ],
  "threadId": "ThreadId(1)"
}
I don't have
ttl
set, but I do have
Copy code
communication_timeout: 360
      startup_timeout: 360
Will these keep the agent alive? Or is there a recommended way to make this work? Thanks
a
Hi Brad, It seems that you have a lingering agent on the target Can you make sure you don’t have any local mirrord processes running before trying to use mirrord?
b
The first ephemeral container (the working one) has logged this
Copy code
│ {"timestamp":"2025-07-18T06:01:46.460798Z","level":"TRACE","fields":{"message":"start_agent -> All clients finished, exitin │
│ g main agent loop"},"target":"mirrord_agent::entrypoint","span":{"args":"Args { mode: Ephemeral { mesh: None }, communicate │
│ _port: 51234, communication_timeout: 360, network_interface: None, metrics: None, test_error: false, operator_tls_cert_pem: │
│  None, is_mesh: true, ipv6: false }","name":"start_agent"},"spans":[{"args":"Args { mode: Ephemeral { mesh: None }, communi │
│ cate_port: 51234, communication_timeout: 360, network_interface: None, metrics: None, test_error: false, operator_tls_cert_ │
│ pem: None, is_mesh: true, ipv6: false }","name":"start_agent"}],"threadId":"ThreadId(1)"}                                   │
│ {"timestamp":"2025-07-18T06:01:46.460804Z","level":"TRACE","fields":{"message":"start_agent -> Agent shutting down, droppin │
│ g cancellation token for background tasks"},"target":"mirrord_agent::entrypoint","span":{"args":"Args { mode: Ephemeral { m │
│ esh: None }, communicate_port: 51234, communication_timeout: 360, network_interface: None, metrics: None, test_error: false │
│ , operator_tls_cert_pem: None, is_mesh: true, ipv6: false }","name":"start_agent"},"spans":[{"args":"Args { mode: Ephemeral │
│  { mesh: None }, communicate_port: 51234, communication_timeout: 360, network_interface: None, metrics: None, test_error: f │
│ alse, operator_tls_cert_pem: None, is_mesh: true, ipv6: false }","name":"start_agent"}],"threadId":"ThreadId(1)"}
So I thought that it meant that there were no lingering processes
a
Makes sense
Can you share full mirrord.json?
Seems you’re using your own port instead of letting mirrord assign - it could be that port wasn’t freed yet
Just linux quirks
b
Copy code
{
  "target": {
    "path": "{{ get_env(name='MD_DEPLOYMENT', default='deployment/playmaker-grpc/container/grpc') }}",
    "namespace": "{{ get_env(name='MD_NAMESPACE', default='') }}"
  },
  "feature": {
    "network": {
      "incoming": {
        "mode": "{{ get_env(name='MD_MODE', default='off') }}",
        "ignore_ports": [
          8081
        ],
        "port_mapping": [
          [
            50000,
            5000
          ]
        ]
      },
      "outgoing": true
    },
    "fs": {
      "mode": "read",
      "local": [
        ".+\\.js",
        ".+\\.mjs",
        ".+\\.ts",
        ".+\\.go",
        "dist/*"
      ]
    },
    "env": {
      "override": {
        "GRPC_PORT": "{{ get_env(name='MD_GRPC_PORT', default='50000') }}",
        "GRPC_REFLECTION": "true"
      }
    }
  },
  "operator": true,
  "skip_processes": "tsc",
  "kube_context": "{{ get_env(name='MD_CLUSTER', default='dev') }}",
  "internal_proxy": {
    "log_destination": "/tmp/mirrord.log",
    "log_level": "mirrord=info"
  }
}
Yes, have the agent to a static port as per https://metalbear.co/mirrord/docs/faq/common-issues#agent-connection-fails-or-drops-when-using-an-ephemeral-agent-with-a-service-mesh So we can exclude it from istio
Copy code
traffic.sidecar.istio.io/excludeInboundPorts: 'xxxx
'
a
Oh you don’t need to. There’s a flag we added to auto resolve that, one sec
Hey, can upgrade to chart mirrord-operator-1.27.3 to enable new flag, add in values.yaml: agent: extraConfig: exclude_from_mesh: trueif you’re using internal registry for agent make sure to pull latest version (3.144.0)
Sorry I’m on mobile so couldn’t edit nicely but if you’re not using the Kelley’s that goes under agent config minus the extra config
So agent.exclude_from_mesh: true
b
Thankyou! I'll give it a shot
I think that has worked, will keep testing
2 Views