Hi Team I had applied lineage to a table in datahu...
# troubleshoot
n
Hi Team I had applied lineage to a table in datahub, Now after 1 day when I am opening that table again, I am not able to see the lineage applied. How to resolve this issue??
✅ 1
@hundreds-photographer-13496 @dazzling-judge-80093
h
between which entity types did you set the lineage edge ? When you applied lineage earlier, did it get added successfully or was there any error ? Did you see the lineage yesterday even after refreshing the page ?
n
I was able to see it yesterday but not today
I had set it up between hive and postgres
There was no error when I applied it earlier
d
Which source do you use?
n
source is hive
a
what is your datahub version ? Can you try changing the time filter window on lineage view to "All time" ?
n
0.8.41 How to chenge the time filter window
h
Oh, that's pretty old, it didn't have that feature. How did you apply lineage ? Is it possible that lineage got overwritten by another ingestion job ?
n
Ah yes I have pipelines running everyday But I need to keep the lineage and pipelines running
@hundreds-photographer-13496 Is there some way to keep lineage and pipelines running?
h
to avoid the lineage getting overwritten by another, you can write a custom transformer that adds lineage as part of ingestion process. Another option is to implement patch lineage mcps in both - lineage source and ingestion source to avoid overwrites.
n
@hundreds-photographer-13496 which is the easier option here that requires minimum efforts?
@hundreds-photographer-13496 @dazzling-judge-80093 It’s been 3 days, I am still stuck in this issue
a
Hey @numerous-account-62719 both the options require some development effort. I am guessing that first one would be easiest for you to implement as it requires development on transformer side alone. However, if your are using UI based ingestion, adding a custom transformer will not work out of the box I would suggest you take a deeper look at the options and figure out the best choice based on your setup. Have you already confirmed that the issue is with lineage is getting overwritten ? I am not clear 1. How did you apply custom lineage? - https://datahubspace.slack.com/archives/C04DK4ADVL3/p1680182523997519?thread_ts=1680160930.626679&cid=C04DK4ADVL3 2. which ingestion source are you using (that emits lineage) that interferes/overwrites with your custom lineage
n
Hi @hundreds-photographer-13496 I have applied the custom lineage using the dataset-to-dataset lineage script So I have multiple data sources 1. Hive 2. Postgres 3. Kafka 4. Mongodb These sources have lineage between their datasets. All the lineage is getting reset after running ingestion jobs Based on this can you suggest which option should I go with??
a
Hey @numerous-account-62719 AFAIK, these sources do not emit upstream lineage but I may be missing something. Can you help me with an example custom edge that's going missing ? When ingestion job runs, does a different upstream edge get added overwriting the older(custom lineage script) edge ?
n
@hundreds-photographer-13496 So I have a lineage from hive to postgres having airflow as orchestration tool Now when I run postgres and hive ingestion jobs the lineage tab is disabled in the dataset I don’t think any upstream pr downstream is added, the lineage itself gets disabled
a
So I have a lineage from hive to postgres having airflow as orchestration tool
could you share the link to docs pointing to this example ? Without ingestion job, does above lineage edge show up just fine ? Please share screenshots for one of the concerned downstream entities so that we understand this better. 1. Screenshot of lineage viz or tab before the ingestion job (that shows custom lineage edges, you've added) 2. Screenshot lineage viz or tab after the ingestion job (that shows disabled lineage) Also, I would highly encourage upgrading datahub to latest versions in any case, given that your DataHub version is quite old.
n
@hundreds-photographer-13496 I have observed one thing I have deployed datahub on openshift cluster, The gms and acryl pod was restarted on 30th March and I lost the Lineage when I checked it on 31st March. Can you tell me where the lineage is stored so that we can assign the PVC to it??
a
by default, lineage is typically stored on Mysql (graph index is elastic search or neo4j).
n
so if I attach the PVC to prerequisites MySQL so the issue will be resolved right?
a
I am not much familiar with Kubernetes/Helm and PVC. If you believe that the entire mysql data was lost during restart, you can check about this issue in #all-things-deployment channel. One quick way you can check if mysql data before 30th was lost is to verify the
select min(createdon) from metadata_aspect_v2
table in mysql.
n
@hundreds-photographer-13496 I just checked this I have date of 16th Feb present there So I guess this pod recreation is not the issue
n
Hi @hundreds-photographer-13496 You can see there are 2 screenshots, one is of friday where you can see the lineage and one is of today where the lineage is disabled
a
Okay, so looks like the lineage was via a DataJob. Are you able to find this dataJob/task in airflow entities ? The name is empty so not able to mention it. How did you introduce this task ? Will you be able to re-ingest it ? IF you know urn of this task - you can check in RDS the
dataJobInputOutput
aspect associated with it, to confirm if still has this lineage set up correctly..
n
@hundreds-photographer-13496 Yes I can see the data task, the name is empty
a
How are these tasks ingested ? how do urns look like ?
n
@hundreds-photographer-13496 This is how it looks like urnlidataJob:(urnlidataFlow:(airflow,inv-stg-node_edge-pg_hive,prod)
@hundreds-photographer-13496?? Any solution?
a
So if you go to Lineage tab here, does it show upstream and downstream correctly as
dim_site_master
and
bbip_route_details
?
n
@hundreds-photographer-13496 No, Actually the thing is that some lineage flows are as it is and some are getting reset You can refer the below screenshot dim_site_master had 2 flows, one from bbip_route_details and another from bbip_route_details_alarms. The second flow is as it is but the first one is disabled
a
This appears to be issue with airflow task related setup - where you are emitting lineage for
dim_site_master
and
bbip_route_details
. I would request you to figure out that particular task urn and monitor versions of
dataJobInputOutput
aspect for that task in RDS (select * from metadata_aspect_v2 where urn=task_urn) to understand what's causing the reset. This appears to be particular to how you have set up lineage and not an issue from DataHub side. https://datahubproject.io/docs/debugging/#how-can-i-check-if-data-has-been-loaded-into-mysql-properly Once RDS data looks correct, i.e. the entry corresponding to version=0 has the expected lineage (inputs and outputs). If the UI continues to not show lineage, a simple restore indices should work - https://datahubproject.io/docs/how/restore-indices.
n
@hundreds-photographer-13496 How will I get that particular task urn if the lineage is reset and the task is not there in airflow tasks?
a
Please re-apply lineage in that case. It appears that there is nothing we can do about the earlier missing lineage. Do note down the concerned entity urns, just in case the problem re-appears, so that you would know where to start debugging.
n
Hi @hundreds-photographer-13496 I have observed one thing in the code of data lineage. I will send you 2 codes, one is the old code and one is the new. In both codes, I have removed input_data_jobs as there are jobs for input In the old one I had kept same name for flow_id and job_id. Is this the reason why the lineage got reset? I am attaching both codes for your reference
Copy code
from typing import List

import datahub.emitter.mce_builder as builder
from datahub.emitter.mcp import MetadataChangeProposalWrapper
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.com.linkedin.pegasus2avro.datajob import DataJobInputOutputClass
from datahub.metadata.schema_classes import ChangeTypeClass

# Construct the DataJobInputOutput aspect.
input_datasets: List[str] = [
    builder.make_dataset_urn(platform="postgres", name="inventory_data.public.alarms")
 
    
    
    

]

output_datasets: List[str] = [
    builder.make_dataset_urn(
        platform="postgres", name="heatmap.public.bbip_route_details_alarms")
]

datajob_input_output = DataJobInputOutputClass(
    inputDatasets=input_datasets,
    outputDatasets=output_datasets
)


airflow_dag = "inv_rsnac_uc10"
orches = "airflow"


datajob_input_output_mcp = MetadataChangeProposalWrapper(
    entityType="dataJob",
    changeType=ChangeTypeClass.UPSERT,
    entityUrn=builder.make_data_job_urn(
        orchestrator=orches, flow_id=airflow_dag, job_id=airflow_dag
    ),
    aspectName="dataJobInputOutput",
    aspect=datajob_input_output,
)

# Create an emitter to the GMS REST API.
emitter = DatahubRestEmitter("****")

# Emit metadata!
emitter.emit_mcp(datajob_input_output_mcp)
Copy code
from typing import List

import datahub.emitter.mce_builder as builder
from datahub.emitter.mcp import MetadataChangeProposalWrapper
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.com.linkedin.pegasus2avro.datajob import DataJobInputOutputClass

# Construct the DataJobInputOutput aspect.
input_datasets: List[str] = [
    builder.make_dataset_urn(platform="postgres", name="inventory_data.public.alarms", env="PROD")
]

output_datasets: List[str] = [
    builder.make_dataset_urn(
        platform="postgres", name="inventory_data.public.corellated_alarms_updated", env="PROD"
    )
]

# input_data_jobs: List[str] = [
#     builder.make_data_job_urn(
#         orchestrator="airflow", flow_id="flow0_Alarm-data-pipeline-UC-1-2-3", job_id="job0_Alarm-data-pipeline-UC-1-2-3", cluster="PROD"
#     )
# ]

datajob_input_output = DataJobInputOutputClass(
    inputDatasets=input_datasets,
    outputDatasets=output_datasets,
    # inputDatajobs=input_data_jobs,
)

# Construct a MetadataChangeProposalWrapper object.
# NOTE: This will overwrite all of the existing lineage information associated with this job.
datajob_input_output_mcp = MetadataChangeProposalWrapper(
    entityUrn=builder.make_data_job_urn(
        orchestrator="airflow", flow_id="flow1_Alarm-data-pipeline-UC-1-2-3", job_id="job1_Alarm-data-pipeline-UC-1-2-3", cluster="PROD"
    ),
    aspect=datajob_input_output,
)

# Create an emitter to the GMS REST API.
emitter = DatahubRestEmitter("****")

# Emit metadata!
emitter.emit_mcp(datajob_input_output_mcp)
The first code is old one and second is new one
@hundreds-photographer-13496 Let me know if you get a chance to go through the code
h
I don't believe I understand what you are asking. But for lineage to persist, you'd need t use different job_id for different pairs for input/ouput datasets.