features.py

import numpy as np


def a(x: float) -> np.ndarray:
    return np.random.normal(size=1)


def b(a: np.ndarray) -> np.ndarray:
    return np.power(a, 2)


def c(b: np.ndarray) -> np.ndarray:
    return np.convolve(b, np.ones(32), 'valid') / 32

build.py

import time
from hamilton import base
from hamilton.driver import Driver

from src import features

dict_builder = base.DictResult()
adapter = base.SimplePythonGraphAdapter(dict_builder)


dr = Driver({}, features, adapter=adapter)

start_time = time.time()
dr.execute(['c'], inputs={'x': 0.})
end_time = time.time()
print(f"DAG executed on run 1 in {end_time - start_time:.9f} seconds")


start_time = time.time()
dr.execute(['c'], inputs={'x': 0.})
end_time = time.time()
print(f"DAG executed on run 2 in {end_time - start_time:.9f} seconds")

from src.timings.features import a, b, c

start_time = time.time()
output = {}
output['a'] = a(0.)
output['b'] = b(output['a'])
output['c'] = c(output['b'])
end_time = time.time()

print(f"DAG executed manually in {end_time - start_time:.9f} seconds")

Gives me the output:

DAG executed in 0.000430107 seconds
DAG executed in 0.000244141 seconds
DAG executed manually in 0.000026941 seconds

for example. I'm also curious as to why execution time halves on the second run. This is obviously pretty contrived, but I am also experiencing slowdown in a more complicated & realistic pipeline -- around 2x

So .execute on the same graph should be thread safe and independent of each other. Technically you should not need to reconstruct the graph. Do you have more detail on: > This is obviously pretty contrived, but I am also experiencing slowdown in a more complicated & realistic pipeline -- around 2x

How big is your graph? Otherwise any overhead during execution should largely be dictionary related — Hamilton memoizes results.

If I add

time.sleep(1)

to each function, it removes all the differences (makes sense, high variance):

DAG executed on run 1 in 3.011426926 seconds
DAG executed on run 2 in 3.014846087 seconds
DAG executed manually in 3.015386105 seconds

When I remove

numpy

, the difference shortens significantly:

DAG executed on run 1 in 0.000269175 seconds
DAG executed on run 2 in 0.000249147 seconds
DAG executed manually in 0.000003099 seconds

Seeing a 20% speedup the second time still, on a few runs. SOme are closer to 30-50%. Very highly variant.

Given that python doesn’t have a jit, it’s not obvious as to what’s going on. Some thoughts: 1. Didn’t find anything at a cursory glance in the driver that keeps tate 2. This is probably some (unintentionally) stateful piece, likely outside of our logic (anything dynamic that gets cached the first time) 3. I’d be curious to see a more “real-life” repro with a signficant slowdown 4. To really dig in we’d have to profile where it’s spending its time 5. The difference between manual and Hamilton execution is likely a fixed-cost overhead. E.G. it goes away when time.sleep is added, just the cost of dict manpulation/whatnot