Hey team, I just tried out the Datadog Adapter and could use some help understanding the trace results. I'm using the AsyncDriver and passed in the ddog adapter to `.with_adapters`:

datadog_hook = h_ddog.DDOGTracer(root_name="hamilton_dag_trace")

builder = (
    AsyncBuilder()
    .with_config(config)
    .with_modules(*self.included_modules)
    .allow_module_overrides()
    .with_adapters(self.result_builder, datadog_hook)
)
driver = await builder.build()

I see the traces ok in DataDog but trying to figure out what's happening during long idle times in the graph. I have 5 Hamilton functions that fetch from Postgres, each of which uses

extract_columns

to extract total of ~300 nodes. The odd thing is I see all the nodes execute really fast at the beginning of the graph, and then the postgres queries fire after, with no node execution after that. Is this something in the way the AsyncDriver works by creating a bunch of asyncio tasks at once and awaiting asyncio.gather for them to complete, and the datadog spans are only running over task creation, not the function body running?

Yes, this is likely the async driver. We have updated this for the Hamilton UI, can probably apply the same approach to ddog. Basically it measures the execution of the function which just creates a coroutine/promise, computing it runs later. AFK but can take a look at the code later!

Ok thanks! Makes sense

Just to understand how to reproduce that, what is creating the outer trace context?

I'm using

ddtrace-run uvicorn ...

to auto patch modules like FastAPI. I believe they auto wire in middleware that creates a span around the request

I’m not at my keyboard but should be able to take a look, it should be able to work with this

Cool, definitely open to the opentelemetry plugin too if it makes it easier

(it’s on the roadmap to have the Hamilton UI also ingest OTel… )

Know you guys are busy — I put this in an issue for tracking: https://github.com/DAGWorks-Inc/hamilton/issues/1236

Thanks! Will take a look soon and scope out — agreed that this is high value (and we should be able to do this with OTel as well)

Awesome, thanks! I'll see if I get some time and can open a PR. I will test out the OTel version first and see if it works better as-is with async

Awesome!