https://www.boundaryml.com logo
Powered by
🟡 No Content
# questions
v
Experiencing 249s delays on async LLM calls - possible connection pool exhaustion?
Hey team! 👋 We're seeing intermittent long delays (~250s) on some async LLM calls and think it might be related to HTTP connection pooling. Looking for guidance! Setup: - Using BAML async client in Python - Total: 100-1000 concurrent requests to the same provider (OpenRouter) - Each per-document batch uses its own semaphore for concurrency control The Issue: - Random 1-4 chunks (out of 100s of chunks per document) take 240-250 seconds - BAML's function_log.timing.duration_ms shows 249447ms - Low input/output tokens, not atypical workload. - Provider's metrics show these requests completed quickly on their end - Our Python process was alive (lock renewals every 30s confirmed) - Other chunks complete normally in 2-5 seconds What we've ruled out: - ✅ Not provider backend latency (their metrics disagree with BAML's timing) - ✅ Not Python/asyncio blocking (other chunks complete fine concurrently) - ✅ BAML timeouts configured: connect_timeout_ms: 10000, time_to_first_token_timeout_ms: 60000, idle_timeout_ms: 30000 - ✅ Started happening ~1-2 months ago (Nov 2025). My theory is moving to single provider is exhausting the pool. Questions: 1. Does BAML expose HTTP connection pool configuration? I saw some chatter on GH issues. 3. What's the recommended max concurrent calls to the same provider? I think we startd seeing this when we moved the waterfall to single provider (OpenRouter) 4. Any way to log/get visibility into HTTP connection pool utilization? Would love any insights or suggestions! Happy to provide more logs/traces. Thanks! 🙏
* HTTP/2 head-of-line blocking or flow control window exhaustion* ?
I tried creating three identical baml clients to the same provider with round robin, not sure if it helped, but i did get into the same issue and can repro.
If I re-queue the same request that is stuck on the same machine, it quickly gets done while other one is still stuck.
v
Thanks for sharing! This is a high priority request. Will investigate this for my Friday!
v
Yes, are you available at 11?
@iamalsogreg
v
@greg (BAML)
v
@Vaibhav (BAML 🐑)
g
👋
v
#Build a virtualenv using the appropriate Debian release #* Install python3-venv for the built-in Python3 venv module (not installed by default) #* Install gcc libpython3-dev to compile C Python modules #* In the virtualenv: Update pip setuputils and wheel to support building new packages FROM debian:12-slim AS build RUN apt-get update && \ apt-get install --no-install-suggests --no-install-recommends --yes python3-venv build-essential libpython3-dev && \ apt-get upgrade -y liblzma5 && \ python3 -m venv /venv && \ /venv/bin/pip install --upgrade pip setuptools wheel # Build the virtualenv as a separate step: Only re-execute this step when requirements.txt changes FROM build AS build-venv COPY requirements.txt /requirements.txt RUN mkdir /app # For local docker testing without Azure App Config, uncomment the next line: # COPY config/config.json /app/config/config.json RUN /venv/bin/pip install --no-cache-dir --disable-pip-version-check -r /requirements.txt # Copy baml_src (test files excluded via .dockerignore) and regenerate baml_client # This ensures production builds don't include test data (~400MB memory savings) COPY baml_src /app/baml_src RUN cd /app && /venv/bin/baml-cli generate # Copy project files needed for editable install COPY pyproject.toml /app/ COPY app /app/app # Install the project in editable mode so 'app' module is importable RUN cd /app && /venv/bin/pip install -e . # Copy the virtualenv into a distroless image FROM gcr.io/distroless/python3-debian12 COPY --from=build-venv /venv /venv COPY --from=build-venv /app /app WORKDIR /app ENTRYPOINT ["/venv/bin/python3", "app/task.py", "./config/config.json"]
PreviousNext
Powered by