A few Laminar Signals questions: 1. Per-agent tri...
# questions
c
A few Laminar Signals questions: 1. Per-agent triggering. We want a signal to fire on just one agent's traces , but a trigger only filters the traces table with scalar predicates, and nothing there identifies the agent — top_span_name is "conversation" for every agent, trace_type is DEFAULT, tags are empty, metadata.agent_id is an opaque UUID. We confirmed the trigger does match a scalar eq on top_span_name. Is there a way (Laminar SDK/instrumentation config) to name the root span per agent, or surface the agent type as a scalar trace field, so we can trigger on it? (May be an OpenHands-side change — asking both.) 2. Clusters. Can we configure or steer the clustering ourselves (e.g. group a signal's events by our own key/label), or is it fully automatic? 3. Signal chaining. Is there any supported way to trigger one signal after another — signal B running only on traces/events (or clusters) that signal A flagged? We couldn't find a dependency/ordering field on triggers. Is it on the roadmap?
c
hey @clever-window-69122! Robert, co-founder of Laminar here. I will break it down shortly
1. Are you using OpenHands SDK? Also, any chance you can elaborate on what do you mean by agent type, how this type is defined? 2. you can setup signal to produce enum structured field, and then filter signal events on enum values 3. what's your usecase for this? I think it might be achieved with a single signal? I can also invite you to the shared slack channel in Laminar slack for deeper support.
c
Hey Robert, thanks so much, really appreciate you jumping in! 🙏 And yes please, add us to the shared channel ; that'd be great for the deeper back-and-forth. OpenHands SDK? We run our agents on OpenHands (self-hosted - openhands.c3ci.cloud) and the traces come from OpenHands' built-in Laminar auto-instrumentation, we drive it through the OpenHands API (our own client wrapper), so we're not doing a custom Laminar SDK integration ourselves. What "agent type" means: our product is multi-agent — one user request fans out across ~8 distinct agents (a main orchestrator, a Workflow Builder, a DI agent, ML, RAG, etc.), each with its own prompts/skills. "Agent type" = which of those produced a given trace. The problem : nothing at the trace level identifies it : the root span is "conversation" for every agent, trace_type is DEFAULT, tags are empty, and metadata.agent_id is an opaque UUID that doesn't map to the agent type. So we can't tell a trigger "only run on Workflow Builder traces." Use case: we run per-agent rubric signals to catch quality regressions in each agent's behavior. We want each rubric-signal to evaluate only its agent's traces, and to be able to filter/cluster events by agent. Today we work around it by self-gating in the prompt : the signal fires on every trace and returns an empty result if it's not the target agent. It works, but it runs on everything (cost) and muddies clustering. Your enum idea sounds promising. To make sure I follow: one signal whose structured output includes an agent_type enum, and we filter signal_events on that enum value? That would cover our filtering need. Two things I'd love your take on: 1. Does that filtering happen after the signal has already run on every trace, or is there a way to gate at the trigger level so it doesn't run on irrelevant traces at all ? 2. A single signal is appealing but each agent's rubric is fairly different . A few other things we've been trying to do (happy to move these to the channel): - Tags: can we attach tags to a trace programmatically (and does anything from the OpenHands conversation-create payload — e.g. an extra/metadata field — propagate onto the Laminar trace) so we can filter traces by, say, the git commit they ran against or the agent? Filtering by tag in the UI is exactly what we want. - Clusters: can we configure/steer how a signal's events are clustered (group by our own key), or is it fully automatic? - Signal chaining: any way to trigger one signal after another (B only on what A flagged)? Thanks again!
👀 1
m
@clever-window-69122 for question 1 I think you could pass
observability_tags
,
observability_metadata
, and
observability_span_name
to your conversation start endpoint. Not sure you are using the endpoint that supports it, but I think generally this is either already supported or can be supported by OpenHands.
@millions-crayon-32880 @limited-student-10747 can you keep me honest on the above please? (tagging you as the authors of the relevant lines in github.com/OpenHands/software-agent-sdk/blob/…/request.py#…)
c
Hey following up on this if @millions-crayon-32880 could provide some insight
m
Hey @clever-window-69122 top_span_name being conversation is expected, span is set just under conversation. Span triggered signals still work. See github.com/OpenHands/software-agent-sdk/pull/3881 with the evidence.
See the images inside the PR description with "mySpanName" set and triggering signals.
Could you confirm if that example works for you?
c
Thanks @millions-crayon-32880, and yes I went through #3881, that all lines up. Just to confirm I've got it right: the conversation root span stays conversation by design, and the observability_span_name we set is emitted as an immediate child span just under that root. So top_span_name = conversation is expected, not a bug. And since Laminar signals trigger on the child spans rather than only the root, recall isn't affected. That matches what we see on our end: our WB rubric signal is already firing fine on the conversation.arun / conversation.send_message child spans, so signal-based recall works today without us setting anything extra. The one piece still open for us is actually setting the span name in the first place, which looks like an endpoint mismatch rather than a signals issue. We hit OpenHands purely over the REST API: POST /api/v1/app-conversations to create the conversation, then upload skills, send the prompt, and poll events. The sandbox is fully OpenHands-managed so we can't touch the SDK env, only the API from outside. @melodic-furniture-87909 found that observability_metadata / observability_tags / observability_span_name live on POST /api/conversations (request.py L228-246), but we're calling /api/v1/app-conversations. That lines up with our test: passing observability_span_name to app-conversations gave us no named child span. So the question: does /api/v1/app-conversations accept those observability fields?
m
@clever-window-69122 You are correct on both accounts. I'll fix the /api/v1/app-conversations
c
Hey happy monday! Just wanted to check what the status of the fix looks like @millions-crayon-32880
c
@millions-crayon-32880 we (Laminar team) can jump on it too, if you're too busy, just let us know
m
Hey @clever-window-69122, we are working on this I got a fix and should be released soon.
🙌 1
❤️ 1
c
Thanks, let me know when we need to update on our end!
m
@clever-window-69122 this is fixed in github.com/OpenHands/enterprise/pull/130 and released in 1.51.0
🙌 1
c
Hey @clean-address-80622 I am actually unable to access the laminar slack channel, it says it has disconnected and needs some sort of slack upgrade from my end. If you could check the same and let me know. Also if you could let me know about the remote signal managament on the c3ci self hosted version