A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

&gt; 3. Any prior art on Airflow integrations, especially on what hooks are available? Ideally, we’d love Airflow to push events on task scheduled, task started and task completion to correctly calculate timeliness WRT SLA and alert our users. Is that something has been done successfully in the past? Meanwhile, if Airflow is only able to push event on task completion, and <https://github.com/treff7es/datahub/blob/master/metadata-models/src/main/pegasus/com/linkedin/dataprocess/DataProcessInstanceRunEvent.pdl#L15|DataProcessInstanceRunEvent> only have one <https://github.com/treff7es/datahub/blob/master/metadata-models/src/main/pegasus/com/linkedin/timeseries/TimeseriesAspectBase.pdl|TimeseriesAspectBase>, how do I persist several timestamps like `execution_time`, `start_time`, and `finish_time`? Should I create another similar aspect, or extend the existing ones? :thread: 

<@UV14447EU> would be the right person to answer this, his recent townhall demo should give you a good idea of how the plugin approach can give us access to these hooks in the tasks lifecycle, but he only covered success and failure events, I'm not sure if we get scheduled, started callbacks. <https://youtu.be/7iwNxHgqxtg?t=915>

For the RunEvent it makes sense IMO to add those other time durations / time values to the aspect, all you have to do is add it and annotate it as a @TimeseriesField

CleanShot 2022-04-30 at 08.53.23@2x.png

Hey,
There are a couple of callbacks you can use in Airflow
<https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/baseoperator/index.html>
We currently use pre_execute and post_execute for capturing run statuses.
I was using pre_execute as well to capture task run starts. I later decided to only capture status when a task finishes otherwise if a task get canceled the state can be stuck on datahub in the running state (post execute callbacks won’t get called in this case, only dag’s on_failure get called).