Hi, I am migrating my code from the experiment to the new UI. I have some problems, which I can not find anwser from doc: 1. I pass {'work_dir': Path} to driver.with_config, how can I use it like this:

@save_to.file(path=source("exp_dir")/"task.txt")

? 2. How can I delete project from UI? Can I access those by using python API? I also want to apply a new project by using Python API, so if someone can not launch a UI, they can also use the code(it is too hard to launch a docker in HPC, and also, we have a data limit to file number and size). 3. How to execute jobs parallel, if I want to compute something with python code. I want to run a DAG with same inputs, but execute it at the same time. Should I use ray or dask?

Is this a pattern that worked for you before? As far as I know, it shouldn't. You can think of

source()

as a node name. For example

def exp_dir() -> str:
  return "abc"

@save_to.file(path=source("exp_dir"))
def my_artifact() -> bytes:
  return

Would be roughly equivalent to

def exp_dir() -> str:
  return "abc"

def my_artifact() -> bytes:
  return

def my_artifact__save_to_file(
   my_artifact: bytes,
   exp_dir: str
):
   with open(exp_dir, "wb") as f:
     f.write(my_artifact)

Therefore you can't really use it as part of an f-string I believe

Thanks! Since I have a lot of files to save in a specific folder, can I

os.chdir

before

dr.materialize

a task? I think the old Experiment adapter can switch the experiment folder when there is a saver decorator, but I don't know where it is now. For example: I want to create a folder named

exp1-idxxx

, and all the files should be saved under this folder. When I was using the experiment adapter, it would automatically change to the folder save the file and change back. So I only need to pass file name

For background context, the ExperimentTracker and the HamiltonUI were built for separate reasons, but their developement ended up informing each other. When I built the ExperimentTracker, one of the main feature I cared about was the

os.chdir

to different unique subdirectories to store results on my local filesystem The Hamilton UI was built with different set of assumptions and doesn't care about how you store results Before migrating, I advise trying to use both the

ExperimentTracker

and

HamiltonTracker

adapters within your Driver and see if it meets your current needs (let us know if you hit any bug)

clever... Maybe I can manage my own ExperimentTracker since I am building an scientific workflow, and I have some special requests

Ok, so there are many open questions in this thread, I'll try to summarize and number them for comprehension: 1.

@save_to

and

source()

syntax. It's a technical Hamilton question. Was the previous answer useful? 2. programmatically interact with Hamilton UI projects (create, delete). As far as I know, this isn't currently possible. Would be a good feature to add, but we would have to think about an API for it. @Elijah Ben Izzy 3. Parallel execution. It's a Hamilton question and appears unrelated to experiment tracking or the UI, correct? > 4. Changing directory for materialization, like the

ExperimentTracker

does. As per previous answers, it is not a feature of the

HamiltonTracker

, so you could add both adapters to your

Builder().with_adapters()

5. What does the Hamilton UI track. It collects metadata after you execute your Driver and stores that metadata in a postgres instance included in the Docker compose stack I want to highlight that the

ExperimentTracker

and

HamiltonTracker

aren't interchangeable. The HamiltonTracker displays metadata about what you ran such as statistics for a dataframe, but doesn't actually store the result dataframe. The ExperimentTracker is what you need for managing artifacts. Now, if there are some integration between the two that you think would be valuable, we can discuss that further. Also, we can chat about how to extend or customize the ExperimentTracker. Since it's a Python class, you could write your own tracker by subclassing it and overriding the components you care about

1. crystal clear! 2. It is pretty useful!! I really need it! 3. Correct, then I can ask someone to answer it. It does not belong to this thread. 4. Yes, I understand 5. OK, I really like it as a dashboard for all experiments Thanks for your patience, and I think this thread should be closed. I believe you can provide a fantastic UI and API in the future version. Nice job!

Glad it was helpful! I think the ExperimentTracker + HamiltonTracker make a very powerful pair! Next would be to improve our caching support to supercharge that!

So yes, programatic project creation makes a lot of sense and its something we’d like to do (also happy with user contributions 🙂 ) Created this issue here — if you’re interested in helping out I can detail the steps to how one might do it!