https://datahubproject.io logo
Join Slack
Powered by
Hi I am trying to delete some glossaryTerm so I ra...
# ingestion
m
Hi I am trying to delete some glossaryTerm so I ran
Copy code
curl "<http://localhost:8080/entities?action=delete>" -X POST --data '{"urn": "urn:li:glossaryTerm:AccountBalance"}'
It deleted the data in mysql but not in UI. I realized it may still exist in elasticsearch. Then I deleted all data in elastic search
Copy code
curl -s -X DELETE <https://vpc-schema-registry-XXXXXX.us-east-1.es.amazonaws.com/*>
Then restart the server. Looks like the server is rebuilding the index and backfill data. But the speed is very slow. Is there a quick way to rebuild the elastic search? Or What is the right way to delete an entity in both mysql and elastic search? Thanks!
Hi @mammoth-bear-12532 Can I schedule a 1:1 with someone today? We are close to the demo but seems we lost all data in the UI. Thanks!
m
Hi @mysterious-lamp-91034 sure! Let me find someone. Meanwhile have you tried the “restore indices” process?
m
Thanks @mammoth-bear-12532 I haven't. Do you have wiki?
m
Yes 👍
cc @incalculable-ocean-74010 do you have bandwidth to assist?
i
I have 15m now if that is acceptable @mysterious-lamp-91034
m
Sure. Thanks!
@incalculable-ocean-74010 Now I know it is 8 cores 16GB EBS only AWS c5.2xlarge
Do you think I should upgrade it?
i
Do you know how much of that GMS is using?
m
No. Let me check with our SRE and get back to you. Thanks!
👍 1
Hi @incalculable-ocean-74010 You said we may disable the cache, that will help show data in mysql in the UI?
i
Disabling the cache will make UI consistent with MySQL when MCP processor has finished processing the message
m
Great. Do you know how to disable cache?
i
Set 
CACHE_TTL_SECONDS
environment variable in GMS to 
1
thank you 1
m
I set CACHE_TTL_SECONDS to 1 in gms environment restart the server ingest an dataset, but I don't think it is browseable or searchable BTW, our SRE said there is no limiting that container use the resource in that host. I have upgraded the host to 36cores 72GB EBS c5.9xlarge and restart the server. And looks like the index-rebuilding does not continue.
@incalculable-ocean-74010 I can see the data if I go to the dataset detail page, but they are not browserable or searchable.
NVM, it is searchable now. Thanks
@incalculable-ocean-74010 Some datasets appear, some do not. 😥
i
Could it be that those datasets were deleted and simply appeared because the search index was not updated? If they are missing can you run ingestion for them?
m
I ran ingestion, still not appear
Copy code
20:41:38.685 [pool-10-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /entities?action=ingest - ingest - 200 - 38ms
20:41:38.690 [qtp544724190-38] INFO  c.l.m.r.entity.AspectResource:125 - INGEST PROPOSAL proposal: {aspectName=subTypes, systemMetadata={lastObserved=1657312897838, runId=pinterest-hive-2022_07_08-20_41_34}, entityUrn=urn:li:dataset:(urn:li:dataPlatform:pinterest-hive,data.users_d,PROD), entityType=dataset, aspect={contentType=application/json, value=ByteString(length=24,bytes=7b227479...65225d7d)}, changeType=UPSERT}
20:41:38.750 [pool-10-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 60ms
20:41:38.755 [qtp544724190-49] INFO  c.l.m.r.entity.AspectResource:125 - INGEST PROPOSAL proposal: {aspectName=container, systemMetadata={lastObserved=1657312897908, runId=pinterest-hive-2022_07_08-20_41_34}, entityUrn=urn:li:dataset:(urn:li:dataPlatform:pinterest-hive,data.core_daily_sessionized_feedview_action_stats,PROD), entityType=dataset, aspect={contentType=application/json, value=ByteString(length=66,bytes=7b22636f...6137227d)}, changeType=UPSERT}
20:41:38.814 [pool-10-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 59ms
20:41:38.822 [qtp544724190-38] INFO  c.l.metadata.entity.EntityService:742 - INGEST urn urn:li:dataset:(urn:li:dataPlatform:pinterest-hive,data.core_daily_sessionized_feedview_action_stats,PROD) with system metadata {lastObserved=1657312897908, runId=pinterest-hive-2022_07_08-20_41_34}
20:41:38.859 [pool-10-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /entities?action=ingest - ingest - 200 - 38ms
20:41:38.863 [qtp544724190-49] INFO  c.l.m.r.entity.AspectResource:125 - INGEST PROPOSAL proposal: {aspectName=subTypes, systemMetadata={lastObserved=1657312897909, runId=pinterest-hive-2022_07_08-20_41_34}, entityUrn=urn:li:dataset:(urn:li:dataPlatform:pinterest-hive,data.core_daily_sessionized_feedview_action_stats,PROD), entityType=dataset, aspect={contentType=application/json, value=ByteString(length=24,bytes=7b227479...65225d7d)}, changeType=UPSERT}
20:41:38.922 [pool-10-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 59ms
This is the server log after I run ingestion. So I am sure it is ingested
To be specific, I ingest one table, here is the log
Copy code
20:45:09.372 [qtp544724190-59] INFO  c.l.metadata.entity.EntityService:742 - INGEST urn urn:li:dataset:(urn:li:dataPlatform:pinterest-hive,bi.core_daily_sessions_stats,PROD) with system metadata {lastObserved=1657313109111, runId=pinterest-hive-2022_07_08-20_45_06}
20:45:09.411 [pool-10-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /entities?action=ingest - ingest - 200 - 39ms
20:45:09.414 [qtp544724190-31] INFO  c.l.m.r.entity.AspectResource:125 - INGEST PROPOSAL proposal: {aspectName=subTypes, systemMetadata={lastObserved=1657313109112, runId=pinterest-hive-2022_07_08-20_45_06}, entityUrn=urn:li:dataset:(urn:li:dataPlatform:pinterest-hive,bi.core_daily_sessions_stats,PROD), entityType=dataset, aspect={contentType=application/json, value=ByteString(length=24,bytes=7b227479...65225d7d)}, changeType=UPSERT}
20:45:09.477 [pool-10-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 63ms
But
double checked the server log, it only ingest the proposal with aspectName=
subType or container
, Curious why it does not ingest schemaMetadata? Is it because it found no change then skip the ingestion?
If my assumption is correct, is there a way to force ingesting the MCP?
i
Unfortunately I do not know. @dazzling-judge-80093 can you help here?
thank you 1
In the meantime @mysterious-lamp-91034 can you try to run “datahub get - -urn <entity name>” ?
In the cli, it will output what was ingested. My thought is that not enough information was ingested to be rendered, if any
m
Sure. I guess I should go to the server to run that command. I only have binary file in the server, so I guess I need to clone the codebase to server.
@incalculable-ocean-74010 I think I have to install lots of environment in server. Can I visit the dataset detail page instead https://playbook-test.pinadmin.com/dataset/urn:li:dataset:(urn:li:dataPlatform:pinterest-h[…]daily_sessions_stats,PROD)/Schema?is_lineage_mode=false It give me 200, The content is same as mysql db
i
As long as you can ping the gms instance from your machine you should be able to download the cli in your local laptop and run the cli there directly
m
Pinterest require me to add a special header to call gms. I don't see custom header option in .datahubenv I will add them manually.
It works
The response is long
I folded them
i
Ok, so aspects were ingested for the entity you dont see in the ui
Is “browserPath” among them?
m
Yes. The first one
If I want to go to the init state, should I delete all rows in metadata_aspect_v2 and metadata_index?
m
@mysterious-lamp-91034 did restore indices not work for you
m
I pruned all database. and manually added all data. Luckly the data is not too much.
It is fine now.
I will never delete the elasticsearch data any more 🥲
m
ok great to hear... would be great to understand which pieces didn't work perfectly the first time, so we can improve the restore scripts.
thank you 1
4 Views
Open in Slack
PreviousNext
Powered by