https://docs.all-hands.dev/ logo
Join Slack
Powered by
Are there any plans in progress to optimize agent ...
# proj-agent
m
Are there any plans in progress to optimize agent startup time? I think LiteLLM is the main culprit
l
We could certainly take a look, do you have an example of a command that takes a long time?
m
Let me create some isolated scenarios - I think this is related to https://openhands-ai.slack.com/archives/C0A6XE64XK4/p1776382333516889 as well
m
Are you referring to the agent-server in particular, or to the CLI?
m
agent-server - I don't utilize the CLI It's mainly noticable when operating in a binary where startup time is involved
👍 1
l
Another quick question: is this the first time you run it on a machine, or every time?
m
Everytime - Still working on isolating some tests
👍 1
Ok, it's not fully litellm -
Copy code
SDK import time: 2.33s (3767 modules)

  lmnr (observability)                56 modules
  grpc (via lmnr)                     21 modules
  google.protobuf (via otel)          30 modules
  opentelemetry exporters             20 modules
  tokenizers (via litellm)            14 modules
  starlette (via fastmcp)             23 modules
  uvicorn (via fastmcp)               18 modules
Copy code
1. LAZY-IMPORT lmnr in observability/laminar.py
   Move `from lmnr import ...` inside the functions that use it (observe(),
   maybe_init_laminar(), etc.). This is the single highest-impact change:
   eliminates 53 MB of mandatory transitive deps (grpc + protobuf + otel
   exporters) for consumers who don't use Laminar tracing.

2. REMOVE fakeredis from hard dependencies
   It's declared as `fakeredis[lua]>=2.32.1` but never imported anywhere
   in the SDK source tree. Removing it drops redis + lupa from the tree.

3. LAZY-IMPORT litellm in llm.py (or ask litellm to lazy-import tokenizers)
   litellm → tokenizers is 8.3 MB of native binary. If litellm won't fix
   it upstream, the SDK could lazy-import litellm inside LLM.__init__()
   instead of at module level.
l
ah, thanks we'll take a look
🙌 1
c
@microscopic-night-41309 thank you! Robert from Laminar here, will take a look and resolve asap
🙌 1
qq: how did you measure the import impact?
m
Copy code
python3 test.py
---
import sys
import time

t0 = time.perf_counter()
from openhands.sdk import LLM, Agent, Conversation

elapsed = time.perf_counter() - t0

print(f"SDK import time: {elapsed:.2f}s ({len(sys.modules)} modules)")
print()

checks = {
    "lmnr (observability)": "lmnr",
    "grpc (via lmnr)": "grpc",
    "google.protobuf (via otel)": "google.protobuf",
    "opentelemetry exporters": "opentelemetry.exporter",
    "tokenizers (via litellm)": "tokenizers",
    "starlette (via fastmcp)": "starlette",
    "uvicorn (via fastmcp)": "uvicorn",
}

for label, prefix in checks.items():
    mods = [k for k in sys.modules if k.startswith(prefix)]
    status = f"{len(mods)} modules" if mods else "NOT LOADED"
    print(f"  {label:35s} {status}")
c
cc @proud-analyst-89204
👀 1
m
hey @microscopic-night-41309 I also ran a benchmark independently and got the following results (for startup time with 10 iterations):
Copy code
import_s               p50=  1.55s  p90=  1.63s  max=  2.06s
  create_app_s           p50=  0.03s  p90=  0.03s  max=  0.03s
  import_to_ready_s      p50=  1.61s  p90=  1.69s  max=  2.13s
  total_s                p50=  1.63s  p90=  1.71s  max=  2.15s
Ideally, it would be nice to get the p90 under 1s. Is this consistent with what you’re seeing? Or is the startup time even worse in your case? I’m asking because, for a private project, I converted the agent-server into a binary and ran it that way. The startup time was worse than what I’m seeing here, but that was due to how I handled the conversion.
m
@modern-fireman-99755
Copy code
Running 10 iterations...
  [1/10] import=2.22s  create=0.002s  total=2.22s
  [2/10] import=1.75s  create=0.001s  total=1.75s
  [3/10] import=1.74s  create=0.001s  total=1.74s
  [4/10] import=1.87s  create=0.002s  total=1.88s
  [5/10] import=1.73s  create=0.001s  total=1.74s
  [6/10] import=1.75s  create=0.001s  total=1.75s
  [7/10] import=1.78s  create=0.002s  total=1.79s
  [8/10] import=1.80s  create=0.001s  total=1.80s
  [9/10] import=1.79s  create=0.001s  total=1.79s
  [10/10] import=1.80s  create=0.001s  total=1.80s

Results (10 iterations):

  import_s               p50=  1.79s  p90=  2.22s  max=  2.22s
  create_app_s           p50=  0.00s  p90=  0.00s  max=  0.00s
  import_to_ready_s      p50=  1.79s  p90=  2.22s  max=  2.22s
  total_s                p50=  1.79s  p90=  2.22s  max=  2.22s
m
Hi all, I made a PR to try and load Laminar lazily https://github.com/OpenHands/software-agent-sdk/pull/3007 Would greatly appreciate if you could try your benchmarks on your setups with this potential fix. If things look good to OpenHands team, we could merge this
🙌 1
2 Views
Open in Slack
PreviousNext
Powered by