numerous-account-62719
03/30/2023, 7:22 AMnumerous-account-62719
03/30/2023, 11:53 AMhundreds-photographer-13496
03/30/2023, 1:04 PMnumerous-account-62719
03/30/2023, 1:05 PMnumerous-account-62719
03/30/2023, 1:05 PMnumerous-account-62719
03/30/2023, 1:05 PMdazzling-judge-80093
03/30/2023, 1:17 PMnumerous-account-62719
03/30/2023, 1:17 PMaloof-gpu-11378
03/30/2023, 1:18 PMnumerous-account-62719
03/30/2023, 1:18 PMhundreds-photographer-13496
03/30/2023, 1:22 PMnumerous-account-62719
03/30/2023, 1:40 PMnumerous-account-62719
03/31/2023, 4:52 AMhundreds-photographer-13496
03/31/2023, 7:23 AMnumerous-account-62719
03/31/2023, 7:26 AMnumerous-account-62719
04/03/2023, 1:33 PMaloof-gpu-11378
04/04/2023, 6:30 AMnumerous-account-62719
04/04/2023, 6:34 AMaloof-gpu-11378
04/04/2023, 6:37 AMnumerous-account-62719
04/04/2023, 6:41 AMaloof-gpu-11378
04/04/2023, 6:59 AMSo I have a lineage from hive to postgres having airflow as orchestration toolcould you share the link to docs pointing to this example ? Without ingestion job, does above lineage edge show up just fine ? Please share screenshots for one of the concerned downstream entities so that we understand this better. 1. Screenshot of lineage viz or tab before the ingestion job (that shows custom lineage edges, you've added) 2. Screenshot lineage viz or tab after the ingestion job (that shows disabled lineage) Also, I would highly encourage upgrading datahub to latest versions in any case, given that your DataHub version is quite old.
numerous-account-62719
04/05/2023, 9:07 AMaloof-gpu-11378
04/05/2023, 10:05 AMnumerous-account-62719
04/05/2023, 10:06 AMaloof-gpu-11378
04/05/2023, 10:32 AMselect min(createdon) from metadata_aspect_v2 table in mysql.numerous-account-62719
04/06/2023, 6:16 AMhundreds-photographer-13496
04/06/2023, 7:38 AMnumerous-account-62719
04/10/2023, 8:57 AMaloof-gpu-11378
04/10/2023, 9:30 AMdataJobInputOutput aspect associated with it, to confirm if still has this lineage set up correctly..numerous-account-62719
04/10/2023, 9:41 AMaloof-gpu-11378
04/10/2023, 10:25 AMnumerous-account-62719
04/10/2023, 10:29 AMnumerous-account-62719
04/11/2023, 5:54 AMaloof-gpu-11378
04/11/2023, 6:33 AMdim_site_master and bbip_route_details ?numerous-account-62719
04/11/2023, 6:51 AMaloof-gpu-11378
04/11/2023, 7:55 AMdim_site_master and bbip_route_details. I would request you to figure out that particular task urn and monitor versions of dataJobInputOutput aspect for that task in RDS (select * from metadata_aspect_v2 where urn=task_urn) to understand what's causing the reset. This appears to be particular to how you have set up lineage and not an issue from DataHub side.
https://datahubproject.io/docs/debugging/#how-can-i-check-if-data-has-been-loaded-into-mysql-properly
Once RDS data looks correct, i.e. the entry corresponding to version=0 has the expected lineage (inputs and outputs). If the UI continues to not show lineage, a simple restore indices should work - https://datahubproject.io/docs/how/restore-indices.numerous-account-62719
04/11/2023, 9:07 AMaloof-gpu-11378
04/11/2023, 9:21 AMnumerous-account-62719
04/11/2023, 9:56 AMnumerous-account-62719
04/11/2023, 9:57 AMfrom typing import List
import datahub.emitter.mce_builder as builder
from datahub.emitter.mcp import MetadataChangeProposalWrapper
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.com.linkedin.pegasus2avro.datajob import DataJobInputOutputClass
from datahub.metadata.schema_classes import ChangeTypeClass
# Construct the DataJobInputOutput aspect.
input_datasets: List[str] = [
builder.make_dataset_urn(platform="postgres", name="inventory_data.public.alarms")
]
output_datasets: List[str] = [
builder.make_dataset_urn(
platform="postgres", name="heatmap.public.bbip_route_details_alarms")
]
datajob_input_output = DataJobInputOutputClass(
inputDatasets=input_datasets,
outputDatasets=output_datasets
)
airflow_dag = "inv_rsnac_uc10"
orches = "airflow"
datajob_input_output_mcp = MetadataChangeProposalWrapper(
entityType="dataJob",
changeType=ChangeTypeClass.UPSERT,
entityUrn=builder.make_data_job_urn(
orchestrator=orches, flow_id=airflow_dag, job_id=airflow_dag
),
aspectName="dataJobInputOutput",
aspect=datajob_input_output,
)
# Create an emitter to the GMS REST API.
emitter = DatahubRestEmitter("****")
# Emit metadata!
emitter.emit_mcp(datajob_input_output_mcp)numerous-account-62719
04/11/2023, 9:57 AMfrom typing import List
import datahub.emitter.mce_builder as builder
from datahub.emitter.mcp import MetadataChangeProposalWrapper
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.com.linkedin.pegasus2avro.datajob import DataJobInputOutputClass
# Construct the DataJobInputOutput aspect.
input_datasets: List[str] = [
builder.make_dataset_urn(platform="postgres", name="inventory_data.public.alarms", env="PROD")
]
output_datasets: List[str] = [
builder.make_dataset_urn(
platform="postgres", name="inventory_data.public.corellated_alarms_updated", env="PROD"
)
]
# input_data_jobs: List[str] = [
# builder.make_data_job_urn(
# orchestrator="airflow", flow_id="flow0_Alarm-data-pipeline-UC-1-2-3", job_id="job0_Alarm-data-pipeline-UC-1-2-3", cluster="PROD"
# )
# ]
datajob_input_output = DataJobInputOutputClass(
inputDatasets=input_datasets,
outputDatasets=output_datasets,
# inputDatajobs=input_data_jobs,
)
# Construct a MetadataChangeProposalWrapper object.
# NOTE: This will overwrite all of the existing lineage information associated with this job.
datajob_input_output_mcp = MetadataChangeProposalWrapper(
entityUrn=builder.make_data_job_urn(
orchestrator="airflow", flow_id="flow1_Alarm-data-pipeline-UC-1-2-3", job_id="job1_Alarm-data-pipeline-UC-1-2-3", cluster="PROD"
),
aspect=datajob_input_output,
)
# Create an emitter to the GMS REST API.
emitter = DatahubRestEmitter("****")
# Emit metadata!
emitter.emit_mcp(datajob_input_output_mcp)numerous-account-62719
04/11/2023, 9:58 AMnumerous-account-62719
04/12/2023, 8:33 AMhundreds-photographer-13496
04/19/2023, 6:34 AM