Hi team, I am running the 'Your First Dataflow' from Get Started, and it seems quite slow - the runs are recorded to take ~2 seconds, and the overall execution time for the script is closer to 4-5 seconds (probably because I am running the UI locally?) I was hoping for much less overhead, how can I check if I'm doing things in the correct way, or measure the framework overhead better? Many thanks in advance.

Hey! So I don’t know for sure but I think this is a plugin issue — something we addressed recently. It autoloads all the plugins but there’s an easy way to disable it. You can do:

export HAMILTON_AUTOLOAD_EXTENSIONS=0

Or follow instructions here: https://hamilton.dagworks.io/en/latest/how-tos/extensions-autoloading/#disable-autoloading

Basically if you have a lot of things in your python environment, Hamilton will try to auto set up things for them. So this is something quick to try and rule out. Disabling turns that off — but then you need to manually trigger the plugins you need.

Thanks for the quick replies! I tried disabling the autoloading of extensions, as per above, seems to be the same still. I was following https://hamilton.dagworks.io/en/latest/get-started/your-first-dataflow/, together with a local UI setup. This is not preventing me from exploring the framework, just a fyi

Oh - yeah so the local hamilton SDK client has a 5 second flush interval IIRC. So that could be it from the macro level (e.g. why 4-5 seconds?). Are using it in a notebook? or script? or?

Using it from a script, the ~2 sec exection still remains even if I remove the HamiltonTracker code and execute the driver without adapters

yeah in my local

from hamilton import driver, base

takes a couple seconds.