Slackbot
06/08/2022, 4:18 PMEduardo
what section of the documentation is related to the whole task execution management.are you looking for a long-form explanation of how the rerun logic works?
How does ploomber know when to rerun a task other than if the code_source of the task is changed ?ploomber evaluates all tasks in the "render" step and determines if it has to run a task or not if the task being evaluated has changed (source code) or if any of its dependencies has changed. For example if you have A -> B -> C. and A changes, that will trigger execution of A, B,C. If B changes, it will trigger B and C. There are other cases that trigger a run, this happens when the is a "corrupted state" for example, if you run a pipeline, but then delete any of the products, ploomber will rerun it.
Could it be possible that in the product of a task when saving a .ipynb we're able to modify the name of the file. The execution I would like to have is that the output filename receive the date from datetime.now().yes, this is built in it would be something like this:
tasks:
- source: scripts/my-script.py
product:
# {{now}} will be replaced by the current timestamp
nb: 'products/{{now}}/report.html'
data: 'products/{{now}}/data.csv'
I would have to integrate validation step to activate or not specific task inside the .yaml .
Would those needs be met by using hook step likeI need a few more details here. It sounds like you want to customize the rerun logic and conditionally execute tasks based on some external rules? We have an example here Let me know if this solves all your questions!
Julien Roy
06/10/2022, 4:32 PM