Can I get :eyes: on those 2 PRs that enable safe c...
# proj-agent
s
Can I get 👀 on those 2 PRs that enable safe concurrent Codex credentials refresh? • OpenHands github.com/OpenHands/OpenHands/pull/15287 • SDK github.com/OpenHands/software-agent-sdk/pull/4124 @limited-student-10747 @calm-artist-7206 @quick-window-91705 (sorry rohit its because tim is on pto 😅 not sure how to ask to in cloud team)
Why so complex? I tried reducing the scope but concurrent safe Codex auth refresh seems to be inherently complicated. Hence the table in the openhands PR description
Untitled.md
c
Sorry, I’d like to understand the desired behavior here: • in my understanding, we are about to keep the
app-server
and make it a supported backend for
agent-canvas
. Ref: Isolated containers for each conversation ◦ (we have two PRs for it) • the OpenHands PR seems to be Saas-only • afaik, ACP is supported (?) in
app-server
backend, or isn’t it? • it seems to me that this concurrency bug would happen there too? Please tell me what I’m missing!
Maybe we could solve the multiple sandboxes issue in any other of those options in the linked issue. But idk, Robert seemed to have settled on
app-server
. And Christopher has worked on fixing WS for it. So this is a fair question, it seems to me: if we are maintaining the
app-server
as a backend, then, well, we cannot solve bugs only on the Saas, right… But, I’m sorry for not keeping tabs on this, maybe we should start with the beginning: is ACP supported on the
app-server
alone? (without
enterprise
code)
l
Hi Engel, we are at long last discussing what to do with repo restructuring, including what to do about supporting
app-server
(probably renamed to
sandbox-server
). Let me write up a summary of the current thinking in a moment
gratitude thank you 1
c
FWIW, I have taken a stab at detaching a minimal
app-server
, in the wild hope that it’s maybe not a lot on top of sandbox orchestration, but it turns out that it is quite a lot: https://github.com/enyst/app-server Because then it needs to manage settings and all stuff that is shared between sandboxes… doesn’t look quite “minimal” 😭
l
ah, great, let me move this discussion to a new thing on #C0ACTL5GAQ3
Hey @strong-ice-84145 going back to the original PR list, approved the SDK PR and left a comment on the openhands PR.
🙌 1
c
Just for clarity, according to your comment, Simon, since Codex ACP creds are in separate directories even for a single
agent-server
case, it means the same concurrency issue exists in the Open Source project, even if we consider it unsupported for the legacy
app-server
Could we make an issue on it? https://github.com/OpenHands/OpenHands/pull/15287#discussion_r3603163578
👀 1
Now that I think of it, is it really necessary to save duplicates and triplicates etc of codex auth credentials?
I think we have avoided doing that with other credentials: a single place, which is not conversation-specific, should be enough
(instead of duplicating credential files in each conversation directory) I think maybe it’s worth thinking about 🤔
s
@calm-artist-7206 I am investigating if its safe to point all local-Docker conversations to a same CODEX_HOME If it is then the code will be much simpler
👍 1
@calm-artist-7206 btw I updated the PR hours ago so it works with locak-Docker multi conversations About having a single CODEX_HOME, it does not seem possible
Copy code
Conclusion: do not point concurrent local-Docker conversations at one writable CODEX_HOME for ChatGPT auth.

  Why:

  - Every conversation launches a separate Codex app-server process. Codex's refresh semaphore is process-local, so it cannot coordinate conversations.
  - Codex 0.144.6 reads auth.json directly and saves it using truncate(true) followed by write_all, without an inter-process lock or atomic replacement. Exact pinned source
    (<https://github.com/openai/codex/blob/rust-v0.144.6/codex-rs/login/src/auth/storage.rs#L191-L223>)

  - The OpenHands broker correctly serializes upstream token rotation, but it cannot serialize the subsequent writes performed independently by each Codex process.
  - With synthetic ChatGPT OAuth credentials and eight concurrent Codex 0.144.6 processes, I observed the shared auth.json becoming empty during refresh writes. A separate stress run left it persistently malformed.
  - OpenAI documents that local clients share cached login state, but it does not promise concurrent file-write safety. Normal interactive CLI usage has much less contention than an agent server running many simultaneous conversations.
    Authentication documentation (<https://developers.openai.com/codex/auth>)

  The best design for #15287 remains:

  - One authoritative brokered credential per user.
  - One disposable, writable CODEX_HOME working copy per conversation.
  - Propagate rotations back through the broker with CAS, as the PR now does.

  Those duplicate files are therefore isolation copies, not duplicate sources of truth. A shared home would only become defensible after Codex adds an inter-process transaction lock around load/modify/save/delete plus atomic file replacement—or
  if OpenHands multiplexes every conversation through one shared Codex app-server, which is a substantially larger architectural change.
c
This sounds totally weird. Codex has it locally in ~/.codex/, and it’s a single credential
sorry I’m dense, if that works with multiple CLIs, and it does, and with multiple codex app threads, I’m not sure why do we have to have multiple?
s
Copy code
It is one Codex process versus several Codex processes.

  Direct Codex app-server:

  Conversation A ─┐
  Conversation B ─┼─> one Codex app-server
  Conversation C ─┘       └─ one AuthManager
                             └─ one refresh lock
                                 └─ auth.json

  All conversations use the same in-memory authentication manager. If A refreshes, B waits; afterward B reloads the new credentials and skips its own refresh. Only one process writes auth.json.

  Local Docker with OpenHands ACP:

  Conversation A -> codex-acp A -> Codex app-server A -> refresh lock A ┐
  Conversation B -> codex-acp B -> Codex app-server B -> refresh lock B ├-> shared auth.json
  Conversation C -> codex-acp C -> Codex app-server C -> refresh lock C ┘

  Each OpenHands conversation launches a separate codex-acp, and each adapter launches its own codex app-server. The locks are therefore independent. codex-acp process creation
  (<https://github.com/agentclientprotocol/codex-acp/blob/v1.1.2/src/CodexJsonRpcConnection.ts#L15-L42>)

  Even with the OpenHands broker:

  1. A and B both load refresh token r0.
  2. The broker safely rotates r0 → r1 only once.
  3. Both Codex processes receive r1.
  4. Both independently truncate and rewrite the same auth.json.
  5. One process can read the file while it is empty or partially written.

  The broker protects the logical token rotation, but not the subsequent filesystem writes. Codex currently performs those writes without an inter-process lock or atomic replacement. Codex file-auth implementation
  (<https://github.com/openai/codex/blob/rust-v0.144.6/codex-rs/login/src/auth/storage.rs#L191-L223>)

  One important qualification: if by "backend agent-server directly" you mean an OpenHands agent-server started natively that still creates one ACP subprocess per conversation, then it is not safer. It has the same problem. Sharing is safe only
  when one Codex app-server process owns the home and multiplexes all conversations—and no separate CLI/login/logout process writes there concurrently.
@calm-artist-7206 investigating further there's something fishy I agree
❤️ 1
c
Ah, but are we saying that codex cli in multiple tabs use only one?
s
Ok so the conclusion is: local agent-server is also unsafe because multiple acp processes are not meant to work with same codex_home
there is a way to use acp with one process + multiple tabs But this is not what we are doing: each conversation is one acp process and that does not work well with a single codex home
c
is this because of acp or because of codex
normally, is it possible in e.g. Zed to have multiple CLIs running and accessible via ACP?
s
AI
Copy code
Current topology:

  Default local Docker
    conversation A container
      OpenHands agent-server → ACPAgent A → codex-acp A → Codex app-server A
    conversation B container
      OpenHands agent-server → ACPAgent B → codex-acp B → Codex app-server B

  Direct OpenHands agent-server
    one OpenHands agent-server
      ACPAgent A → codex-acp A → Codex app-server A
      ACPAgent B → codex-acp B → Codex app-server B

  Local Docker defaults to NO_GROUPING, which deliberately creates a new sandbox/container for every conversation. The setting defaults to NO_GROUPING
  (<https://github.com/OpenHands/OpenHands/blob/7db8bd4d92d6db33816cb534799b0bb618bbb7de/openhands/app_server/settings/settings_models.py#L605-L693>), and the Docker sandbox service then starts a separate container
  (<https://github.com/OpenHands/OpenHands/blob/7db8bd4d92d6db33816cb534799b0bb618bbb7de/openhands/app_server/sandbox/docker_sandbox_service.py#L385-L495>).

  A directly run OpenHands agent-server can host multiple conversations, but that does not mean one Codex process. Every conversation gets its own ACPAgent, and each ACPAgent owns and spawns its own ACP subprocess
  (<https://github.com/OpenHands/software-agent-sdk/blob/68aa583ebf07efefbb9219b63859c1aacecaf7b3/openhands-sdk/openhands/sdk/agent/acp_agent.py#L1668-L1674>). The launch happens once per agent instance
  (<https://github.com/OpenHands/software-agent-sdk/blob/68aa583ebf07efefbb9219b63859c1aacecaf7b3/openhands-sdk/openhands/sdk/agent/acp_agent.py#L2333-L2470>). Therefore, sharing ~/.codex from a direct agent-server is also unsafe when multiple
  Codex conversations run concurrently.

  Could Docker use one Codex process? Yes, architecturally. The upstream adapter already supports it:

  - One codex-acp process starts exactly one Codex app-server (<https://github.com/agentclientprotocol/codex-acp/blob/v1.1.2/src/index.ts#L71-L127>).
  - It maintains multiple session states keyed by session ID (<https://github.com/agentclientprotocol/codex-acp/blob/v1.1.2/src/CodexAcpServer.ts#L145-L173>).
  - It explicitly tests independent sessions and concurrent prompts (<https://github.com/agentclientprotocol/codex-acp/blob/v1.1.2/src/__tests__/CodexACPAgent/CodexAcpClient.test.ts#L917-L1022>).

  But simply enabling sandbox grouping or pointing every conversation at one CODEX_HOME does not accomplish that. OpenHands would need a sandbox-owned Codex/ACP manager instead of conversation-owned subprocesses. It must also route events,
  approvals, cancellation, masking and teardown by session. More importantly, OpenHands currently injects each conversation's entire secret registry into that conversation's subprocess environment. A shared process would mix conversation-
  specific secrets unless that mechanism were redesigned.

  Default Docker isolation creates another constraint: a Codex process inside container A cannot operate on container B's filesystem and runtime. A shared process therefore requires grouped conversations in the same sandbox, or a much larger
  remote-execution design.

  There is also a subtle Codex 0.144.6 caveat: one app-server shares its main AuthManager across threads and serializes its refreshes, but it also constructs a separate auth manager for its background cloud-config loader. Meanwhile, file
  storage rewrites auth.json using truncate-and-write without a file lock (<https://github.com/openai/codex/blob/rust-v0.144.6/codex-rs/login/src/auth/storage.rs#L191-L218>). So one process substantially reduces the conversation-driven race, but
  it is not yet a complete storage-safety guarantee.

  My conclusion:

  - Do not point current local-Docker conversations at a shared CODEX_HOME.
  - Do not consider direct OpenHands agent-server sharing safe either; it currently launches multiple Codex processes.
  - Keep the per-conversation homes and broker work for the current PR.
  - A future simplification should be a sandbox-scoped, multi-session Codex process for grouped, same-user conversations—with session-scoped secrets and lifecycle routing. That is a separate architectural change, not a small replacement for the
    broker.
c
The agent mixes up app-server with agent-server
secrets are sent per conversation already
What they call session here seems to be the same with conversation for us, or should be 🤔
If my understanding is correct, I think we are adding too much complexity, when apparently codex-acp was already designed for multi-conversation use
s
Yes that's possible. But then if another ACP process does not support multi-conversation use the refactor will be painful
c
I mean, the complexity is already there, and now we’re attempting to fix on top of it for a problem that wouldn’t exist
s
We can try to see how to do multi-conversation use acp/codex multi-conversation use but that might be non trivial
c
On the other side, purely local, do we use one place or multiple places for creds?
I ask because purely local means we don’t encrypt
s
one place, same as regular codex CLI
👍 1
and there's still a concurrency issue there
c
Ah. Because of this architecture where each codex cli is in its own process?
s
Yes. This needs. further investigation, I thought the issue was only for saas tbh
but it's not clear to be if the local concurrency is "real" tbh because it's still one human on one local machine handling conversations on canvas so
c
it's simply that multiple conversations are started and run, they don’t pause while one is running
well… and on a remote VM, there might more more conversations and even more client machines
👍 1
remote VM is “local” if people don’t run the old app-server, I think
s
correct
c
so multiple conversations in the same time
s
yes, correct 😅
c
I am tempted to suggest us to investigate and find a good solution as soon as we can, because if it means it’s bugged, we cannot fix it only on the saas. Or take an even deeper look, and consider if so many bugs are due to attempting to adapt a protocol meant for editors, into browser views? For editors I would have expected a single running conversation TBH. Idk, most if not all VSCode extensions were running in a sidebar one at a time, if I recall this right. But in openhands we need multiple conversations, as canvas is in a browser, and locally - specially on a remote VM. cc @wooden-umbrella-84098
s
Yes, I agree. Using ACP multi-tab codex feature seems impossible: it would be a complete redesign of how openhands uses ACP today. Fixing this by having one ground truth somewhere and a credential broker handling sync with several processes , whether its local / docker / saas, seems like the only way forward
l
I’m a little bit confused — is it not possible to have two codex CLIs open at the same time through the same authentication? I’ve done this before and it seems to work fine
s
it's possible but if both reauthenticate at the same time it's not safe
Copy code
## Why a shared CODEX_HOME is unsafe

  Codex 0.144.6 writes auth.json by opening it with truncate(true) and writing directly. It does not use atomic replacement or an interprocess file lock. Exact Codex source
  (<https://github.com/openai/codex/blob/rust-v0.144.6/codex-rs/login/src/auth/storage.rs#L202-L218>).

  Codex has a refresh semaphore, but it belongs to one AuthManager inside one Codex process. It does not coordinate the separate Codex app-server processes launched by separate OpenHands conversations. Refresh locking source
  (<https://github.com/openai/codex/blob/rust-v0.144.6/codex-rs/login/src/auth/manager.rs#L2362-L2413>).
c
Fixing this by having one ground truth somewhere and a credential broker handling sync with several processes , whether its local / docker / saas, seems like the only way forward
This sounds to me like, we already added a lot of complexity, the only way forward is to add more ?
I’m a little bit confused — is it not possible to have two codex CLIs open at the same time through the same authentication? I’ve done this before and it seems to work fine
Yes… Maybe this is a problem in theory, but the probability is very low?
LLMs LOVE to call things “unsafe”
s
This sounds to me like, we already added a lot of complexity, the only way forward is to add more ?
If this fits with this github.com/OpenHands/OpenHands/issues/15298 we could be OK
Copy code
But Codex 0.144.6 has a much better mechanism we could technically use: external ChatGPT auth.

  The flow would be:

  1. Start Codex with no auth.json.
  2. Initialize app-server with experimentalApi: true.
  3. Call account/login/start with type: "chatgptAuthTokens", supplying only the access token, account ID, and plan.
  4. When Codex receives a 401, it calls the host via account/chatgptAuthTokens/refresh.
  5. Our broker rotates the authoritative refresh token and returns only the new access token.

  Codex keeps these externally supplied tokens in memory rather than persisting them. After login, it also replaces the cloud-config loader with the primary auth manager, so the runtime and cloud-config path use the same manager. Protocol
  (<https://github.com/openai/codex/blob/rust-v0.144.6/codex-rs/app-server-protocol/src/protocol/v2/account.rs#L90-L106>), in-memory auth mode
  (<https://github.com/openai/codex/blob/rust-v0.144.6/codex-rs/app-server-protocol/src/protocol/common.rs#L26-L33>), manager rebinding
  (<https://github.com/openai/codex/blob/rust-v0.144.6/codex-rs/app-server/src/request_processors/account_processor.rs#L658-L674>).

  That would remove most of our current complexity:

  - No duplicated auth.json.
  - No refresh tokens inside conversation containers.
  - No file PUT/CAS reconciliation or final credential synchronization.
  - No reason to share CODEX_HOME for authentication.
  - The broker still owns encryption, capability lifecycle, and cross-conversation refresh serialization.

  The blocker is codex-acp. Our pinned 1.1.2 initializes Codex without experimental capabilities and only implements normal file/browser ChatGPT auth; it does not handle the refresh callback. codex-acp 1.1.2 source
  (<https://github.com/agentclientprotocol/codex-acp/blob/8aff492d4b033ff2c02ad3b9d591994d57617463/src/CodexAcpClient.ts#L70-L104>). We would need an upstream change or fork.

  The other serious caveat is that Codex explicitly labels this API "FOR OPENAI INTERNAL USE ONLY — DO NOT USE."
FOR OPENAI INTERNAL USE ONLY — DO NOT USE.
huh
😅 1
this could work but codex-acp doesn't quite enforce it yet
c
What if we ask the agent to describe how we can use codex-acp for a single agent-server, multiple conversations, to display it in a browser, without your previous conversation context?… Maybe ignoring the existing ACP in agent-server completely if possible 🤔
or maybe, not even tell it about agent-server
just display multiple codex sessions, in a browser… so we can see what would be needed
(though this is probably just me being dense, but I got confused)
Simon, how often did you see the apparent concurrency bug that you were fixing here?
s
It hasn't been seen yet 🙂 What we have seen is saas/docker credentials getting stale (mainly saas)
perhaps what we need is a happy recovery path if one day this concurrency bug happens?
c
The token is refreshed, right?
> This sounds to me like, we already added a lot of complexity, the only way forward is to add more ?
If this fits with this github.com/OpenHands/OpenHands/issues/15298 we could be OK
Multiple codex auth.json files in conversations logs… doesn’t fit with the goal of 15298 🤔 I think maybe we are making this too complicated. If this works:
one place, same as regular codex CLI
then it should work the same • inside docker (slightly different “home”) • on a remote vm • in isolated cloud sandboxes (same as docker) • in shared configurations (one per each box) What if we clean up the per-conversation credential locations?
perhaps what we need is a happy recovery path if one day this concurrency bug happens?
Maybe like • catch the exception, poke codex to refresh... if that's possible • or, not do anything, we can cross that bridge when we come to it?
s
Multiple codex auth.json files in conversations logs… doesn’t fit with the goal of 15298 🤔
It does. Ground truth current codex auth secret can be stored in a Cred Store that safely allows concurrent secret rewrite. Codex ACP still owns the detection of new secret in the auth.json + the responsability of rewriting each auth.json for each conversation Still thinking of simplest fix to go forward
@calm-artist-7206 The bug we have actually observed is sequential staleness: Codex refreshes R0 → R1 and writes R1 to its runtime auth.json, but R1 never reaches the canonical credential store. A later conversation receives R0 again. Asking that conversation to refresh cannot recover because R0 may already be invalid and it has no way to discover R1. The simultaneous-refresh race is different. We haven’t observed it, so we should not build an OAuth proxy, Redis refresh locks, or refresh ownership now. If it happens, retrying a new conversation after the winning process has persisted R1 may already be sufficient. We can add telemetry and cross that bridge with evidence. I’ve reduced the staleness fix to two PRs: - SDK: one small load / conditional replace interface, the Codex auth.json write-through lifecycle, and a local implementation for Agent Server/Canvas. - OpenHands: expose that same interface over HTTP using the existing SaaS and OSS credential stores. There remains one canonical credential location per deployment. Each Codex process only gets an ephemeral writable working copy because Codex itself owns and rewrites auth.json; those copies stay outside persisted conversation state and logs. No generic secrets rearchitecture, refresh proxy, Redis locking, or ACP multi-session redesign in this phase. The detailed pared-down plan is #4177 (github.com/OpenHands/software-agent-sdk/issues/4177).
PRs incoming
c
Thank you! I think that makes it more clear… It sounds like there should be a single auth.json per box - because we let codex do its thing, and it can work with one > The bug we have actually observed is sequential staleness: Codex refreshes R0 → R1 and writes R1 to its runtime auth.json, but R1 never reaches the canonical credential store. If I understand correctly, we have two cases here: • if the runtime auth.json is the same per sandbox/machine, then R1 will be accessible to later conversations in two cases: either the later convo reads the same auth.json, as codex does; or codex reads from a credential store, but then we need to be notified of the refresh and pick up the value? • if the runtime auth.json is one per convo, so many inside the same machine/box… well, we can also get notified Do we make the credential store read/receive R1 and store it?
ahhh sorry I just read the detailed plan - sounds great to me
❤️ 1