Seth Stokes
02/20/2025, 9:04 PMlast_run is being used and its reflected by the function getting used _with_filter_run_date.
However, for documentation purposes, how would you suggest amending so that all last_run config options are visible in the DAG? Is this possible?
@extract_fields({
"child_process_run_id": int,
"child_process_completed_at": pd.Timestamp,
})
@pipe_input(
step(_filter_run_bom).when(last_run="bom"),
step(_filter_run_latest).when(last_run="last"),
step(_filter_run_date, completion_date=source("completion_date")).when(last_run="completion_date"),
)
def child_process_run_info(
child_process_runs: pd.DataFrame,
child_rec_process_code: str,
completion_date: Optional[str] = None
) -> Dict[int, pd.Timestamp]:
"""Return run id for process from `last_run` logic."""
Slack ConversationStefan Krawczyk
02/20/2025, 11:07 PMStefan Krawczyk
02/20/2025, 11:08 PMStefan Krawczyk
02/20/2025, 11:09 PMSeth Stokes
02/20/2025, 11:11 PMSeth Stokes
02/21/2025, 1:15 AMexecutors.SynchronousLocalTaskExecutor() simply allow for the syntactic sugar of Parallelizable/Collect or does it actually process in parallel? If not, would you recommend the MultiThreadingExecutor() instead?Stefan Krawczyk
02/21/2025, 1:17 AMSeth Stokes
02/21/2025, 1:22 AMStefan Krawczyk
02/21/2025, 6:22 AM