Hey Robert, thanks so much, really appreciate you jumping in! 🙏 And yes please, add us to the shared channel ; that'd be great for the deeper back-and-forth.
OpenHands SDK? We run our agents on OpenHands (self-hosted - openhands.c3ci.cloud) and the traces come from OpenHands' built-in Laminar auto-instrumentation, we drive it through the OpenHands API (our own client wrapper), so we're not doing a custom Laminar SDK integration ourselves.
What "agent type" means: our product is multi-agent — one user request fans out across ~8 distinct agents (a main orchestrator, a Workflow Builder, a DI agent, ML, RAG, etc.), each with its own prompts/skills. "Agent type" = which of those produced a given trace. The problem : nothing at the trace level identifies it : the root span is "conversation" for every agent, trace_type is DEFAULT, tags are empty, and metadata.agent_id is an opaque UUID that doesn't map to the agent type. So we can't tell a trigger "only run on Workflow Builder traces."
Use case: we run per-agent rubric signals to catch quality regressions in each agent's behavior. We want each rubric-signal to evaluate only its agent's traces, and to be able to filter/cluster events by agent. Today we work around it by self-gating in the prompt : the signal fires on every trace and returns an empty result if it's not the target agent. It works, but it runs on everything (cost) and muddies clustering.
Your enum idea sounds promising. To make sure I follow: one signal whose structured output includes an agent_type enum, and we filter signal_events on that enum value? That would cover our filtering need. Two things I'd love your take on:
1. Does that filtering happen after the signal has already run on every trace, or is there a way to gate at the trigger level so it doesn't run on irrelevant traces at all ?
2. A single signal is appealing but each agent's rubric is fairly different .
A few other things we've been trying to do (happy to move these to the channel):
- Tags: can we attach tags to a trace programmatically (and does anything from the OpenHands conversation-create payload — e.g. an extra/metadata field — propagate onto the Laminar trace) so we can filter traces by, say, the git commit they ran against or the agent? Filtering by tag in the UI is exactly what we want.
- Clusters: can we configure/steer how a signal's events are clustered (group by our own key), or is it fully automatic?
- Signal chaining: any way to trigger one signal after another (B only on what A flagged)?
Thanks again!