https://ploomber.io/ logo
Join Slack
Powered by
This message was deleted.
# ask-anything
s
This message was deleted.
e
can you provide more detail on inputs/outputs? what's the input and how are you passing it? (e.g. it's an output from an earlier task or is it just a path you pass to the task) and what kind of outputs you're storing?
a
Oh I am using the python API and adding the tasks dynamically. Therefore I should check to see if expected product already exists?
Yes it is my responsibility. Simple fix 
if product.exists(): continue
👍 1
i
So can there be a race condition where you’re adding a task before the product is created?
e
no, so ploomber should take care of checking if the product exists, no need to call it yourself. are these tasks generating a single file?
as long as the task generates the file in the same path, the file exists and the source code hasn't changed, ploomber wont run it again
are any of these conditions not met during runs?
a
all are true, but I am using python API to manually add tasks
Copy code
# we build a new DAG to accumulate all masic tasks
    dag = DAG(executor=Parallel(processes=8, print_progress=True))

    # ============================================================
    def add_task(raw_file):
        final_env = set_env(orig_env, raw_file)

        expected_output = calculate_masic_output(raw_file)
        _masic_output_dir = orig_env["masic"]["outputdir"]
        product = File(Path(_masic_output_dir) / Path(expected_output))

        if product.exists():
            print(f"{product} already exists")
            return

        print(f"adding env: {final_env}")
        ShellScript(
            Path("./scripts/run_masic.sh"),
            dag=dag,
            product=product,
            name=raw_file,
            params=final_env,
        )

    # ============================================================

    # walk
    [x for x in map(add_task, raw_files)]
    dag.build(force=True)
e
so you're passing 
force=True
to 
dag.build
that forces execution of all tasks
a
.. yeah or that could be it. how embarrassing
🙌 1
🤣 1
e
no worries, it happens 🙂
Open in Slack
PreviousNext
Powered by