Slackbot
08/10/2022, 3:25 PMEduardo
Luke Murray
08/10/2022, 3:34 PMLuke Murray
08/10/2022, 3:34 PMLuke Murray
08/10/2022, 3:35 PMLuke Murray
08/10/2022, 3:35 PMEduardo
Eduardo
ploomber build --force
and it will run everything - even if it hasn't changed
if you're not scheduling, you could implement this logic with some Python code. This is a bit hacky but it'll work. Here we have an example where we load a pipeline into Python and overwrites the task status. In this case, even if the task has changed, we skip it. in your case, you want a different thing. You can do something like:
# if the task does not have any dependencies
dag['your-sql-task']._exec_status = TaskStatus.WaitingExecution
# if it has dependencies
dag['your-sql-task']._exec_status = TaskStatus.WaitingUpstream
let me know if this works for you!
the final thing is a long term feature to prevent the hacky solution above. so let me know what you think
we could allow dynamic parameters in tasks. example:
tasks:
- source: script.sql
params:
my_param: my_params.compute_param
then have a my_params.py
def compute_param():
return 'some-value'
and pass 'some-value' to script.sql
- this would allow you to define some custom logic, since this parameter will be part of the SQL query, then you could define something that changes the value every 5 mins / 24 hours. etc. let me know what you think. we can add it to the backlog