Hey all! Is there any way to access a node's metadata like run_id from within the node itself? I'd like to use to add it to some manual telemetry.

The run ID for tracking purposes you mean, right? E.G. that is in the UI?

correct, i want to pass it to some API call functions that have mlflow tracking on them

Oh, cool, yeah thatr’s a good use-case. There isn’t an easy way to do this now that I can think of, but it’s definitely doable. High-level: 1. there are two concepts of run IDs — the UI ones (which are numbers) and the internal unique ones (which are UUIDs) 2. The tracker stores a mapping — you’d have to have the tracker object available 3. You’d access it in a hook (as you mentioned) 4. You could either inject it into kwargs or use a context variable. Injecting it in kwargs is a bit dangerous as you have to check whether the node needs it, and there is a compilation step that checks for missing inputs. Can provide pseudocode if this makes sense? Just checking — you’re using async right?

a context manager sounds like what I have in mind!

yep — that could be part of a hook that takes in the tracker object

I've been workin on an async compatible mlflow adapter: https://gist.github.com/dmexs/3f7b697bab806494051b17a76bc4bab7

things are a bit odd with async but it should work

yeah I've noticed, but I think it's worth it

That’s awesome!

sure, to be transparent, I cut much of the original code out that I don't need but if I get this to where I want perhaps I can contribute back

Yep, exactly. You’d use the tracker to get the mapping + run_id passed in in the hook to get the ID

@Emeka Anyanwu you could do this:

def my_function(a: int, b: int, __run_id: str) -> ...

then in your adapter code (actually this is something we could add to the SDK version easily)

def pre_graph_execute(...) :
   ...
   if inputs is not None: # always have to pass in a dict for inputs to execution
       inputs["__run_id"] = run_id

that makes sense, though I think I like the context manager approach, I'm not quite sure what @Elijah Ben Izzy means by use the tracker to get the mapping + run_id though

> means by use the tracker to get the mapping + run_id though I think he means the tracker code shows various IDs — so you can mimic / copy that. Otherwise using threadlocal context could also work like this example - you have the adapter set values, and then your code pulls them (I assume it should work)

Yes Stefan’s is simpler. The problem is that you still need the run ID for the tracker — it’s different than that from the hook.

got it, I haven't used contextvars before, i'm trying to figure out where to set the context var with the task_Id (which I'm using as my mlflow parent span id), such that the node's function can acesss it, in lieu of parameter injection

Yeah I think inputs are easiest here actually — less global context. But to be clear — which ID do you want specifically? The one for the Hamilton UI? Or the task ID?

I'm looking for the task_id passed to my async pre_node_execute hook: https://gist.github.com/dmexs/3f7b697bab806494051b17a76bc4bab7#file-mlflowtrackerasync-py-L55

Ok, sorry I thought you were talking about the hamilton run ID. Then yeah, setting a context var in pre node execute is easiest

@Emeka Anyanwu in case it’s useful: • FutureAdapter could be a fit for your use case if you didn’t need asyncio • Then you might just be able to use the MLFlow adapter? (or are you after tracing specifically?)

I am planning on asyncio, i wanted to try without it but my nodejs habits wouldn't let me lol

Otherwise to ask the question - what do you want to get out of MLFlow? There’s some Hamilton UI overlap - and we do have an async Hamilton Tracker for the Hamilton UI

yes, I saw, I am running on databricks so wanted to leverage mlflow integration as much as possible

Ok so I can send you some sample pseudo code for context vars soon — it’s similar to how we do otel tracing

Yep great feedback thanks. In any case the more adapters the better — and async mlflow one for tracing sounds great to have.

Tried a couple things, I think in order to get contextvars to work I'd need to wrap this line in context.run() ?

That’s how context vars work — they’re available globally. You: 1. Create a global context var object with all the information you want 2. Call

.set()

in

pre_node_execute

3. Call

.get()

from within the function 4. Call

.set(None)

in

post_node_execute

to reset it

alright, let me play around over here, since these functions are async and run out of order, I'm trying to figure out how the function will get the right contextvar

OK, so yeah, you want to use the async

pre_node_execute

and

post_node_execute

— context vars will work if you do (they’re meant to work in async contexts as well

Alright, I have a reproduction of the issue I'm encountering with contextvars, perhaps a misunderstanding of how they're supported to work on my part: https://github.com/dmexs/hamilton-contextvar-repro

To follow up — we discussed during office hours: 1. We have a nice approach for injecting into the application using kwargs — it uses some internal APIs for now, but: 2. Going to scope out adding a

DataflowContext

object that functions can declare or get globally (like burr does) — this is long overdue IMO 3. @Emeka Anyanwu to figure out what works for him, then post with code and we’ll take it over 4. @Elijah Ben Izzy to figure out why the context var approach didn’t work (there’s some weirdness with context/the indirection layer that Hamilton uses for execute — my guess is we need to do this directly for running the fn, not in pre/post node execute)