Hello, everyone, I used JSON file to import metada...
# ingestionl
lemon-zoo-63387
07/22/2022, 9:43 AMHello, everyone, I used JSON file to import metadata. The first time I succeeded, and then I deleted it. When I executed it again, the client display was succeed, but there was no data in the UI page. Just modify the source. Why is that?
python3 -m datahub delete --env QA --entity_type dataset --platform hivec
careful-pilot-86309
07/22/2022, 9:49 AMYour json had the env as PROD vs your delete command has env as QA. Was env also modified when you changed the source?
l
lemon-zoo-63387
07/22/2022, 9:50 AMMy screenshot is demo data on GIT. I just mark source and schema,
c
careful-pilot-86309
07/22/2022, 9:54 AMCan you share your recipe?
l
lemon-zoo-63387
07/22/2022, 9:55 AM Copy code
[
{
"auditHeader": null,
"proposedSnapshot": {
"com.linkedin.pegasus2avro.metadata.snapshot.DatasetSnapshot": {
"urn": "urn:li:dataset:(urn:li:dataPlatform:hive,schema-data-nan,QA)",
"aspects": [
{
"com.linkedin.pegasus2avro.schema.SchemaMetadata": {
"schemaName": "schema-data-nan",
"platform": "urn:li:dataPlatform:hive",
"version": 0,
"created": {
"time": 1621882982738,
"actor": "urn:li:corpuser:etl",
"impersonator": null
},
"lastModified": {
"time": 1621882982738,
"actor": "urn:li:corpuser:etl",
"impersonator": null
},
"deleted": null,
"dataset": null,
"cluster": null,
"hash": "",
"platformSchema": {
"com.linkedin.pegasus2avro.schema.MySqlDDL": {
"tableSchema": ""
}
},
"fields": [
{
"fieldPath": "county_id",
"jsonPath": null,
"nullable": true,
"description": null,
"type": {
"type": {
"com.linkedin.pegasus2avro.schema.NumberType": {}
}
},
"nativeDataType": "Integer()",
"recursive": false,
"globalTags": null,
"glossaryTerms": null
},
{
"fieldPath": "county_name",
"jsonPath": null,
"nullable": true,
"description": null,
"type": {
"type": {
"com.linkedin.pegasus2avro.schema.StringType": {}
}
},
"nativeDataType": "String()",
"recursive": false,
"globalTags": null,
"glossaryTerms": null
},
{
"fieldPath": "county_address",
"jsonPath": null,
"nullable": true,
"description": null,
"type": {
"type": {
"com.linkedin.pegasus2avro.schema.StringType": {}
}
},
"nativeDataType": "String()",
"recursive": false,
"globalTags": null,
"glossaryTerms": null
},
{
"fieldPath": "num_infection_isolation_rooms",
"jsonPath": null,
"nullable": true,
"description": null,
"type": {
"type": {
"com.linkedin.pegasus2avro.schema.NumberType": {}
}
},
"nativeDataType": "Integer()",
"recursive": false,
"globalTags": null,
"glossaryTerms": null
}
],
"primaryKeys": null,
"foreignKeysSpecs": null
}
}
]
}
},
"proposedDelta": null
},
{
"auditHeader": null,
"proposedSnapshot": {
"com.linkedin.pegasus2avro.metadata.snapshot.DatasetSnapshot": {
"urn": "urn:li:dataset:(urn:li:dataPlatform:hive,schema-data-test,QA)",
"aspects": [
{
"com.linkedin.pegasus2avro.schema.SchemaMetadata": {
"schemaName": "schema-data-test",
"platform": "urn:li:dataPlatform:hive",
"version": 0,
"created": {
"time": 1621882983026,
"actor": "urn:li:corpuser:etl",
"impersonator": null
},
"lastModified": {
"time": 1621882983026,
"actor": "urn:li:corpuser:etl",
"impersonator": null
},
"deleted": null,
"dataset": null,
"cluster": null,
"hash": "",
"platformSchema": {
"com.linkedin.pegasus2avro.schema.MySqlDDL": {
"tableSchema": ""
}
},
"fields": [
{
"fieldPath": "county_type",
"jsonPath": null,
"nullable": true,
"description": null,
"type": {
"type": {
"com.linkedin.pegasus2avro.schema.StringType": {}
}
},
"nativeDataType": "String()",
"recursive": false,
"globalTags": null,
"glossaryTerms": null
},
{
"fieldPath": "county_name",
"jsonPath": null,
"nullable": true,
"description": null,
"type": {
"type": {
"com.linkedin.pegasus2avro.schema.StringType": {}
}
},
"nativeDataType": "String()",
"recursive": false,
"globalTags": null,
"glossaryTerms": null
},
{
"fieldPath": "state_name",
"jsonPath": null,
"nullable": true,
"description": null,
"type": {
"type": {
"com.linkedin.pegasus2avro.schema.StringType": {}
}
},
"nativeDataType": "String()",
"recursive": false,
"globalTags": null,
"glossaryTerms": null
},
{
"fieldPath": "personnel_code",
"jsonPath": null,
"nullable": true,
"description": null,
"type": {
"type": {
"com.linkedin.pegasus2avro.schema.NumberType": {}
}
},
"nativeDataType": "Integer()",
"recursive": false,
"globalTags": null,
"glossaryTerms": null
},
{
"fieldPath": "total_personnel_count",
"jsonPath": null,
"nullable": true,
"description": null,
"type": {
"type": {
"com.linkedin.pegasus2avro.schema.NumberType": {}
}
},
"nativeDataType": "Integer()",
"recursive": false,
"globalTags": null,
"glossaryTerms": null
}
],
"primaryKeys": null,
"foreignKeysSpecs": null
}
}
]
}
},
"proposedDelta": null
}
]lemon-zoo-63387
07/22/2022, 9:56 AMc
careful-pilot-86309
07/22/2022, 11:59 AMI just tried your code with recipe and json file. Everything works well for me. I can see them on ui as well as on cli.
I have understood correctly, you are not able to view them on UI right?
l
lemon-zoo-63387
07/24/2022, 3:24 AMYes, execute recipe.yaml after the delete command , the client display finished succeddfully, but the UI no data
lemon-zoo-63387
07/24/2022, 3:38 AMUI no hive，It's so strange
c
careful-pilot-86309
07/25/2022, 4:42 AM@lemon-zoo-63387 Can you try to read entities by urn using cli? If you get thing here, it will confirm that ingestion is successful and something on ui side is broken. may be elastic search or something. Check if all containers are healthy.
careful-pilot-86309
07/25/2022, 4:42 AMCommand to read entities:
datahub get --urn <urn>b
better-orange-49102
07/25/2022, 7:34 AMstatus set as removed=True, maybe?
c
careful-pilot-86309
07/25/2022, 7:42 AMYou can use --soft, in cli.
l
lemon-zoo-63387
07/25/2022, 9:39 AMCan you give me more details about the command? I'm a rookie😁😁
b
better-orange-49102
07/25/2022, 9:41 AMdatahub get --urn="urn:li:dataset:(urn:li:dataPlatform:hive,schema-data-test,QA)"c
careful-pilot-86309
07/25/2022, 9:42 AMIt should be:
datahub get --urn "urn:li:dataset:(urn:li:dataPlatform:hive,schema-data-test,QA)"l
lemon-zoo-63387
07/25/2022, 10:57 AMI execute it again, status
lemon-zoo-63387
07/25/2022, 10:58 AM Copy code
"status":{
"removed": true
}b
better-orange-49102
07/25/2022, 11:02 AMyou probably should do a hard delete instead of updating the status
Copy code
datahub delete --env QA --entity_type dataset --platform hive --hardl
lemon-zoo-63387
07/25/2022, 11:13 AMb
better-orange-49102
07/25/2022, 11:16 AMoh,that doesnt work. then the more tedious way is to append this to each workunit in the json file:
Copy code
{
"com.linkedin.pegasus2avro.common.Status": {
"removed": false
}
},c
careful-pilot-86309
07/25/2022, 11:18 AM@lemon-zoo-63387 Please correct me, but you are not trying to delete the dataset. You are trying to igest it.
l
lemon-zoo-63387
07/25/2022, 11:24 AMyes,If it is not deleted, the test is successful, and when I change hive to MySQL, Postgres.... All successfully ingested