Hi, We're considering datahub for our metadata needs and I'm looking into various features we would need: • First of all, is it possible to deploy Datahub with postgres as the backend database? • Secondly, I can see that datahub itself is licensed under Apache 2.0; Can I assume the same applies also to any out-of-the-box connectors like Metabase, ClickHouse etc? • What options does Datahub provide for showing etl job status on a table? The use case is like this: we have ETL jobs that put data into tables. If the job fails or takes longer than expected, we want to be able to show on the table in the datahub UI that the data is stale/wrong/missing and we're working on it -- can we do that in a pretty way?

Hi @rich-sandwich-70692, 1. Yes it is. There is a simple flag for it in the helm chart. Otherwise you can also use an existing Postgres instance or one that is managed differently. We chose Postgres in our deployment and so for it works like a charm. 2. All connectors are part of the datahub codebase which is licensed under the Apache 2.0 license. So yes the same license applies to the connectors 3. If you ingest metadata about your pipelines as well, then this is possible. We have set this up recently on our end and we can easily see if the pipeline run was successfull and how long it took. But I don't really know how easy it would be to get alerts when a pipeline runs longer than expected etc. Hope this helps 🙂

It sure does, thank you!

Haha, well I can image 😅 Here is the documentation for sending Spark metadata to DataHub: https://datahubproject.io/docs/metadata-integration/java/spark-lineage But for demo purposes you can simply ingest/create metadata via the API. Best way to do so is using the Python SDK (Java SDK is also available if preferred). A pipeline is represented by the Dataflow class, a pipeline step or task is represented as DataJob, and an execution of a pipeline as a DataProcessInstance.

I create the mock job, (see screenshot) and attached it to the table via Lineage, however I don't see any option of displaying the job status on the table ui. am I missing something?

Did you already create some executions via

DataProcessInstance

with a link to the corresponding job?

Yes, this is the code I was using

import time
import uuid

from datahub.api.entities.corpgroup.corpgroup import CorpGroup
from datahub.api.entities.corpuser.corpuser import CorpUser
from datahub.api.entities.datajob.dataflow import DataFlow
from datahub.api.entities.datajob.datajob import DataJob
from datahub.api.entities.dataprocess.dataprocess_instance import (
    DataProcessInstance,
    InstanceRunResult,
)
from datahub.emitter.rest_emitter import DatahubRestEmitter

emitter = DatahubRestEmitter("<http://localhost:8080>")

jobFlow = DataFlow(env="prod", orchestrator="airflow", id="flow2")
jobFlow.emit(emitter)

# Flowurn as constructor
dataJob = DataJob(flow_urn=jobFlow.urn, id="job1", name="My Job 1")
dataJob.properties["custom_properties"] = "test"
dataJob.emit(emitter)

jobFlowRun: DataProcessInstance = DataProcessInstance(
    orchestrator="airflow", cluster="prod", id=f"{jobFlow.id}-{uuid.uuid4()}"
)
jobRun1: DataProcessInstance = DataProcessInstance(
    orchestrator="airflow",
    cluster="prod",
    id=f"{jobFlow.id}-{dataJob.id}-{uuid.uuid4()}",
)
jobRun1.parent_instance = jobFlowRun.urn
jobRun1.template_urn = dataJob.urn
jobRun1.emit_process_start(
    emitter=emitter, start_timestamp_millis=int(time.time() * 1000), emit_template=False
)
jobRun1.emit_process_end(
    emitter=emitter,
    end_timestamp_millis=int(time.time() * 1000),
    result=InstanceRunResult.FAILURE,
)

Also attaching the job runs in the UI

Haven't looked at your code in detail but based on the screenshot I'd assume that you haven't specified

inlets

and

outlets

, right?

No, I haven't. Are they a part of

DataProcessInstance

? I didn't find any reference to them here: https://datahubproject.io/docs/generated/metamodel/entities/dataprocessinstance/#dataprocessinstancerunevent-timeseries

Yes they are, see: https://github.com/datahub-project/datahub/blob/ee84464406996c75137894ef2b854b2020[…]on/src/datahub/api/entities/dataprocess/dataprocess_instance.py

Should I do something like

outlets=["urn:li:dataset:(urn:li:dataPlatform:clickhouse,GlobalLicenseServer.GlobalLicenseServer.license_usages,PROD)"]

?

Almost, in- and outlets parameter expects a List of

DatasetUrn

instances. So you could do the following:

outlets=[DatasetUrn.create_from_string("urn:li:dataset:(urn:li:dataPlatform:clickhouse,GlobalLicenseServer.GlobalLicenseServer.license_usages,PROD)")]

Thanks! It's weird how when you pass a list of strings instead of a list of Urns you don't get an exception, it just works normally

No worries, I'm glad to help 🙂 Did you add the lineage manually?

Yeah I added it manually the ui

Okay, let me have a detailed look and I'll try to send you an example of our approach 🙂