Hi, guys, I am new to DataHub. I could be able to ...
# ingestionn
narrow-kitchen-1309
08/05/2021, 5:03 PMHi, guys, I am new to DataHub. I could be able to ingest my own datasets, but I would like to add customized Business Glossary for my schema. I kept get error paramater’s snapshot is required? Please let me know how to ingest business glossary and add to metadata assets.
g
green-football-43791
08/05/2021, 5:07 PMHey Vicky! Welcome to Datahub slack 🙂
green-football-43791
08/05/2021, 5:07 PMFor ingesting custom MCE's, I recommend writing them to a json file
green-football-43791
08/05/2021, 5:08 PMand then ingesting them via the CLI
green-football-43791
08/05/2021, 5:08 PMhere's an example recipe:
green-football-43791
08/05/2021, 5:08 PMgreen-football-43791
08/05/2021, 5:08 PMthat uses this json file:
green-football-43791
08/05/2021, 5:08 PMgreen-football-43791
08/05/2021, 5:08 PMto ingest it, you want to run
datahub ingest -c /path/to/your/recipe/file.ymln
narrow-kitchen-1309
08/05/2021, 7:03 PM@green-football-43791 Thanks for quick response. I did try ingest CLI as well. My json file could successful ingested to datahub. But I couldn’t be able to create correct GlossaryTerms value. Could you give me a json example which include glossaryTerms? Unfortunately I didn’t find any sample yet.
narrow-kitchen-1309
08/05/2021, 7:19 PMOnly I could find is this: https://datahubproject.io/docs/gms/#get-a-glossaryterm. I don’t know what json format I need use for glossaryTerms.
g
green-football-43791
08/06/2021, 5:38 PMHey @narrow-kitchen-1309! Yes I can share some with you.
green-football-43791
08/06/2021, 5:38 PM Copy code
{
"auditHeader": null,
"proposedSnapshot": {
"com.linkedin.pegasus2avro.metadata.snapshot.GlossaryTermSnapshot": {
"urn": "urn:li:glossaryTerm:CustomerAccount",
"aspects": [{
"com.linkedin.pegasus2avro.glossary.GlossaryTermInfo": {
"definition": "account that represents an identified, named collection of balances and cumulative totals used to summarize customer transaction-related activity over a designated period of time",
"parentNode": "urn:li:glossaryNode:ClientsAndAccounts",
"sourceRef": "FIBO",
"termSource": "EXTERNAL",
"sourceUrl": "<https://spec.edmcouncil.org/fibo/ontology/FBC/ProductsAndServices/ClientsAndAccounts/CustomerAccount>",
"customProperties": {
"FQDN": "CustomerAccount"
}
}
},
{
"com.linkedin.pegasus2avro.common.Ownership": {
"owners": [{
"owner": "urn:li:corpuser:jdoe",
"type": "DATAOWNER"
}],
"lastModified": {
"time": 1581407189000,
"actor": "urn:li:corpuser:jdoe"
}
}
}
]
}
},
"proposedDelta": null
},
{
"auditHeader": null,
"proposedSnapshot": {
"com.linkedin.pegasus2avro.metadata.snapshot.GlossaryTermSnapshot": {
"urn": "urn:li:glossaryTerm:SavingAccount",
"aspects": [{
"com.linkedin.pegasus2avro.glossary.GlossaryTermInfo": {
"definition": "a product provided to consumers and businesses by a bank or similar depository institution such as a checking account, savings account, certificate of deposit, debit or pre-paid card, or credit card",
"parentNode": "urn:li:glossaryNode:ClientsAndAccounts",
"sourceRef": "FIBO",
"termSource": "EXTERNAL",
"sourceUrl": "<https://spec.edmcouncil.org/fibo/ontology/FBC/FunctionalEntities/FinancialServicesEntities/BankingProduct>",
"customProperties": {
"FQDN": "SavingAccount"
}
}
},
{
"com.linkedin.pegasus2avro.common.Ownership": {
"owners": [{
"owner": "urn:li:corpuser:jdoe",
"type": "DATAOWNER"
}],
"lastModified": {
"time": 1581407189000,
"actor": "urn:li:corpuser:jdoe"
}
}
}
]
}
},
"proposedDelta": null
},
{
"auditHeader": null,
"proposedSnapshot": {
"com.linkedin.pegasus2avro.metadata.snapshot.GlossaryTermSnapshot": {
"urn": "urn:li:glossaryTerm:AccountBalance",
"aspects": [{
"com.linkedin.pegasus2avro.glossary.GlossaryTermInfo": {
"definition": "amount of money available or owed",
"parentNode": "urn:li:glossaryNode:ClientsAndAccounts",
"sourceRef": "FIBO",
"termSource": "EXTERNAL",
"sourceUrl": "<https://spec.edmcouncil.org/fibo/ontology/FBC/ProductsAndServices/ClientsAndAccounts/Balance>",
"customProperties": {
"FQDN": "AccountBalance"
}
}
},
{
"com.linkedin.pegasus2avro.common.Ownership": {
"owners": [{
"owner": "urn:li:corpuser:jdoe",
"type": "DATAOWNER"
}],
"lastModified": {
"time": 1581407189000,
"actor": "urn:li:corpuser:jdoe"
}
}
}
]
}
},
"proposedDelta": null
},
{
"auditHeader": null,
"proposedSnapshot": {
"com.linkedin.pegasus2avro.metadata.snapshot.GlossaryNodeSnapshot": {
"urn": "urn:li:glossaryNode:ClientsAndAccounts",
"aspects": [{
"com.linkedin.pegasus2avro.glossary.GlossaryNodeInfo": {
"definition": "Provides basic concepts such as account, account holder, account provider, relationship manager that are commonly used by financial services providers to describe customers and to determine counterparty identities"
}
},
{
"com.linkedin.pegasus2avro.common.Ownership": {
"owners": [{
"owner": "urn:li:corpuser:jdoe",
"type": "DATAOWNER"
}],
"lastModified": {
"time": 1581407189000,
"actor": "urn:li:corpuser:jdoe"
}
}
}
]
}
},
"proposedDelta": null
}green-football-43791
08/06/2021, 5:38 PMhere are some sample glossary term snapshots
green-football-43791
08/06/2021, 5:39 PMand here is a snapshot of a dataset with glossary terms attached...
green-football-43791
08/06/2021, 5:39 PM Copy code
{
"auditHeader": null,
"proposedSnapshot": {
"com.linkedin.pegasus2avro.metadata.snapshot.DatasetSnapshot": {
"urn": "urn:li:dataset:(urn:li:dataPlatform:kafka,SampleGlossaryTermDataset,PROD)",
"aspects": [
{
"com.linkedin.pegasus2avro.dataset.DatasetProperties": {
"customProperties": {
"Schema": "saving_account.proto:0.0.1",
"Information Classification": "Confidential"
},
"externalUrl": null,
"description": {
"string": "Dataset To demonstrate Glossary Term"
},
"uri": null,
"tags": []
}
},
{
"com.linkedin.pegasus2avro.common.Ownership": {
"owners": [
{
"owner": "urn:li:corpuser:datahub",
"type": "DATAOWNER",
"source": null
},
{
"owner": "urn:li:corpuser:jdoe",
"type": "PRODUCER",
"source": null
}
],
"lastModified": {
"time": 1626098142773,
"actor": "urn:li:corpuser:jdoe",
"impersonator": null
}
}
},
{
"com.linkedin.pegasus2avro.schema.SchemaMetadata": {
"schemaName": "saving_account.proto",
"platform": "urn:li:dataPlatform:kafka",
"version": 0,
"created": {
"time": 1626098142773,
"actor": "urn:li:corpuser:jdoe",
"impersonator": null
},
"lastModified": {
"time": 1626098142773,
"actor": "urn:li:corpuser:jdoe",
"impersonator": null
},
"deleted": null,
"dataset": null,
"cluster": null,
"hash": "",
"platformSchema": {
"com.linkedin.pegasus2avro.schema.KafkaSchema": {
"documentSchema": "syntax = \"proto3\";\n\npackage datahub.glossaryTerm;\nimport \"datahub.glossaryTerm.account_balance.proto\";\n\n// Defines Saving Account details for customer\nmessage SavingAccount {\n option (term_source) = FIBO;\n option (term_source_ref) = \"<https://spec.edmcouncil.org/fibo/ontology/FBC/ProductsAndServices/ClientsAndAccounts/CustomerAccount>\";\n\n option (msg_term_link) = \"CustomerAccount\";\n\n // Balance of Saving Account\n AccountBalance balance = 1;\n\n // Account identifier of Saving account\n int64 accountIdentifier = 2;\n}"
}
},
"fields": [
{
"fieldPath": "balance",
"jsonPath": null,
"nullable": true,
"description": {
"string": "Balance of Saving Account\r"
},
"type": {
"type": {
"com.linkedin.pegasus2avro.schema.RecordType": {}
}
},
"nativeDataType": "int64",
"recursive": false,
"globalTags": null,
"glossaryTerms": {
"terms": [
{
"urn": "urn:li:glossaryTerm:AccountBalance"
}
],
"auditStamp": {
"time": 1581407189000,
"actor": "urn:li:corpuser:jdoe",
"impersonator": null
}
}
},
{
"fieldPath": "accountIdentifier",
"jsonPath": null,
"nullable": true,
"description": {
"string": "Account identifier of Saving account\r"
},
"type": {
"type": {
"com.linkedin.pegasus2avro.schema.NumberType": {}
}
},
"nativeDataType": "string",
"recursive": false,
"globalTags": null,
"glossaryTerms": null
}
],
"primaryKeys": null,
"foreignKeysSpecs": null
}
},
{
"com.linkedin.pegasus2avro.common.GlossaryTerms": {
"terms": [
{
"urn": "urn:li:glossaryTerm:SavingAccount"
},
{
"urn": "urn:li:glossaryTerm:CustomerAccount"
}
],
"auditStamp": {
"time": 1626098142774,
"actor": "",
"impersonator": null
}
}
}
]
}
},
"proposedDelta": null
}n
narrow-kitchen-1309
08/06/2021, 5:39 PM@green-football-43791 Awesome! This is exactly what I need! Thank you very much.. let me give it try. 😁
g
green-football-43791
08/06/2021, 5:40 PMOk- let me know if you have trouble!
n
narrow-kitchen-1309
08/11/2021, 7:02 PM@green-football-43791 Thanks for your help regarding GlossaryTerms. It works for me. But I may need your help with another entities. I try to add Field Level Lineage, but It’s not working. Could you provide sample json regarding field level lineage?
Copy code
{
"com.linkedin.dataset.FineGrainUpstreamLineage": {
"upstreams": [
{
"auditStamp": {
"time": 0,
"actor": "urn:li:corpuser:user"
},
"dataset": "urn:li:dataset:(urn:li:dataPlatform:postgres,myfirst_db.public.patient_data,PROD",
"fields": [
{
"sourceField": "pn",
"targetField": "ln"
},
{
"sourceField": "pa",
"targetField": "la"
}
]
},
{
"auditStamp": {
"time": 0,
"actor": "urn:li:corpuser:user"
},
"dataset": "urn:li:dataset:(urn:li:dataPlatform:postgres,myfirst_db.public.client_data,PROD",
"fields": [
{
"sourceField": "ln",
"targetField": "pn"
},
{
"sourceField": "la",
"targetField": "pa"
}
]
}
]
}
},g
green-football-43791
08/11/2021, 7:03 PMHey Vicky! Fine-grained, column-level lineage is still a WIP, so I would not expect those MCEs to work as expected
n
narrow-kitchen-1309
08/11/2021, 7:19 PM@green-football-43791 oh.. Thanks for quick response. So We have to wait for this RFC then. Do you have estimate when this could be ready for use? https://datahubproject.io/docs/rfc/active/1841-lineage/field_level_lineage
m
mammoth-bear-12532
08/11/2021, 11:25 PM
@narrow-kitchen-1309: we expect that this will be ready to use by end of September
mammoth-bear-12532
08/11/2021, 11:26 PM
it is slotted for this quarter on our roadmap (https://datahubproject.io/docs/roadmap/)
n
narrow-kitchen-1309
08/12/2021, 12:14 PMThanks @mammoth-bear-12532 for the information. This is really helpful!
m
mammoth-bear-12532
08/26/2021, 6:59 PM
@narrow-kitchen-1309: can you explain what you mean by adding custom business glossary for schema?
mammoth-bear-12532
08/26/2021, 6:59 PM
are you thinking how to attach it to each field in the schema?
n
narrow-kitchen-1309
08/26/2021, 7:06 PMHi @mammoth-bear-12532 I think feature of column level/field level lineage would help with our issues. Assume I could have one dataset ingested from DB, column name as “patient_name”, but I have client imported dataset maybe from file, column name as “client_name”, we would like to know the connection between patient_name and client_name. Is this make sense to you?
m
mammoth-bear-12532
08/26/2021, 7:15 PM
Hi @narrow-kitchen-1309: when you started this thread, the question was:
Copy code
Hi, guys, I am new to DataHub. I could be able to ingest my own datasets, but I would like to add customized Business Glossary for my schema. I kept get error paramater's snapshot is required? Please let me know how to ingest business glossary and add to metadata assets.mammoth-bear-12532
08/26/2021, 7:15 PM
is this related to column-level lineage?
n
narrow-kitchen-1309
08/26/2021, 7:19 PMYes, it maybe a little bit confusion. Based on roadmap, it seems field lineage won’t be ready soon. that’s the reason we consider to use business glossary term as work round option, represent fields relationship between datasets for now. But I have problem with JSON data format of Business Glossary term during data ingestion, so I raise that question.
narrow-kitchen-1309
08/26/2021, 7:22 PMWe are still in POC stage, evaluate and explore datahub, and let me know if you have any suggestions. I would appreciate it!
3 Views