This message was deleted.
# ask-anything
s
This message was deleted.
e
I'll check the docs. Perhaps we're not mentioning it explicitly. It looks at the source code, if it has changed it reruns. Since parameters are part of the source code, parameter changes will trigger a run
l
yea that matches what I gathered from documentation for the ploomber build cli option
kind of out there but I assume there isn't a way to "expire" a query
for instance deciding that a query is only valid for 5 minutes or 24 hours
depending on how frequently you care about having the most recent results
e
interesting use case, haven't thought about it before. let me try something, I'll keep you posted
alright, so there are a couple options here: are you running the pipeline on a schedule? if so, you can run it every 5 minutes /24 hours with
ploomber build --force
and it will run everything - even if it hasn't changed if you're not scheduling, you could implement this logic with some Python code. This is a bit hacky but it'll work. Here we have an example where we load a pipeline into Python and overwrites the task status. In this case, even if the task has changed, we skip it. in your case, you want a different thing. You can do something like:
Copy code
# if the task does not have any dependencies
 dag['your-sql-task']._exec_status = TaskStatus.WaitingExecution

# if it has dependencies
 dag['your-sql-task']._exec_status = TaskStatus.WaitingUpstream
let me know if this works for you! the final thing is a long term feature to prevent the hacky solution above. so let me know what you think we could allow dynamic parameters in tasks. example:
Copy code
tasks:
  - source: script.sql
    params:
       my_param: my_params.compute_param
then have a
my_params.py
Copy code
def compute_param():
    return 'some-value'
and pass 'some-value' to
script.sql
- this would allow you to define some custom logic, since this parameter will be part of the SQL query, then you could define something that changes the value every 5 mins / 24 hours. etc. let me know what you think. we can add it to the backlog