Hi! How would you model a process reading from a dataset and publishing metrics in DataDog or Prometheus? I was considering DataFlow+DataJob for the process and Dashboard+Chart for the destination. However DataJob output is exclusively restricted to Datasets.

This is an interesting use case! One way to model this is to think of your Prometheus/DataDog storage component as a dataset in and of itself since you are using it to directly store metrics and have the DataJob point to that. Your charts/dashboards can then include that dataset in their info aspects along with the dataset with relevant data to get a full lineage picture.

Indeed that’s an interesting and valid option, thanks for the suggestion! We can abuse the

KEY_VALUE_STORE

platform type or use the

OTHERS

https://github.com/linkedin/datahub/blob/master/metadata-models/src/main/pegasus/com/linkedin/dataplatform/DataPlatformInfo.pdl#L31