We have flows where we enqueue methods via Django RQ workers and make BAML / LLM calls within those enqueued methods. We've noticed these BAML calls made in the RQ workers are not surfacing in Boundary Studio for observability, but our BAML calls made outside of RQ workers get surfaced fine. I've confirmed the BOUNDARY_API_KEY env var is accessible in all scenarios but can't nail down what's causing this issue. Any guidance here would be much appreciated!

any chance you could make a very simple repro? If not, it may be that you need to

import baml_client

inside the function that is run inside the RQ worker

Here's a repro! https://cdn.discordapp.com/attachments/1457844717457768499/1457883582474227855/BamlRepro.py?ex=695d9f64&is=695c4de4&hm=8c0cdbbf7b66ef6adc5767ff4642d84cd81ef091f9683960d9b0705e30b81463&

ok this is great, thanks for the repro

To your point, we did move the import into the job but still no luck with seeing it in Boundary Studio

the logs also didnt show much right?

you dont see a log line like 'publisher not started'

Hmmm nope not that I'm seeing

we will take a look this friday

FYI @JB , I'm investigating now.

@greg (BAML) I'm also encountering the same issue here. I use Restate When Restate side effect retries cause BAML functions to be called multiple times, Boundary Studio only records the first call - subsequent retry calls are not recorded. ## Reproduction Scenario 1. User sends a message triggering the

message-processing

workflow 2.

ctx.run("process-message")

internally calls

b.NewOrchestrator()

3. Due to long execution time, Restate times out and retries 4. BAML function

NewOrchestrator

is called multiple times ## Expected Behavior Boundary Studio should record all BAML function calls, including: - The first call - Subsequent calls caused by Restate retries ## Actual Behavior Boundary Studio only shows the first call, resulting in: - Unable to trace LLM input/output during retries - Difficult to debug retry-related issues - Inaccurate token consumption statistics ## Log Evidence Restate logs show the same messageId being processed multiple times (iteration restarting from 1), but Boundary Studio only has one record. ## Possible Causes 1. **BAML client caching**: May cache some state at the process level 2. **Trace ID reuse**: Retries may reuse the same trace ID 3. **Boundary Studio deduplication**: Backend may deduplicate identical requests ## Impact - Debugging difficulty: Cannot see actual LLM responses during retries - Cost opacity: Actual token consumption is higher than what Boundary Studio shows - Problem localization difficulty: Cannot determine if it's an LLM output issue or a retry issue

Hello @JB and @vEth! I've been looking in to this. It's definitely an issue with the way our internal infrastructure works in a single-threaded python environment. I don't have a fix from inside BAML yet, but I do have a quick workaround you can do. Whever you call your BAML client function, manually flush the tracing publisher, like this:

from baml_client.async_client import b
from baml_client.tracing import flush

result = await b.MyLLMFunction()

# Force the tracing system to finish publishing.
flush()

My runtime is TypeScript.

When I try to add the flush() call to the

_async_worker_job

method in my repro example linked above, it leads to the method timing out when trying to flush

Oh interesting! And your

_async_worker_job()

function is similar to the one in your repro?

Yeah it's the exact one in the repro