Hello, team.
After some ingestion/deleting procedu...
# troubleshootc
colossal-easter-99672
04/13/2022, 10:06 AMHello, team.
After some ingestion/deleting procedures i can't find dbt datasets via search and browse paths. Only by direct url with urn.
For other platforms all works fine.
Elastic re-index dont help.
What should i can try to fix it?
colossal-easter-99672
04/13/2022, 10:06 AMFor re-index using this
Copy code
kubectl create job --from=cronjob/datahub-datahub-restore-indices-job-template datahub-restore-indices-adhocb
better-orange-49102
04/13/2022, 10:17 AMmost probably the deleted entities have the removed:true Status aspect. (Not sure which version of datahub you have, the latest version have a banner on the profile to indicate that the dataset is soft deleted)
better-orange-49102
04/13/2022, 10:20 AMdoes reingesting help to reactivate them?
c
colossal-easter-99672
04/13/2022, 10:28 AM Copy code
does reingesting help to reactivate them?b
better-orange-49102
04/13/2022, 10:34 AMis there alot of datasets affected? wondering if you should try the rollback or ingest a status aspect. not sure why only dbt is affected
c
colossal-easter-99672
04/13/2022, 12:02 PMYes, a lot. I did rollback and now i can see some datasets (not all). And in list-runs there are some runs that i cant delete
No entities found. Payload used: {blabla}colossal-easter-99672
04/13/2022, 12:09 PMHard delete by urn and reingest fix my problem )
Copy code
datahub delete --urn "blabla" --hard Copy code
datahub delete --platform dbt --hardb
better-orange-49102
04/13/2022, 12:30 PMDoesn't seem possible. I would either ingest status aspect to make them not soft deleted, or go into MySQL to delete the rows and reindex ES
thank you 1
c
colossal-easter-99672
04/13/2022, 12:39 PMgo into MySQL to delete the rows and reindex ESi did it and it didn't help
colossal-easter-99672
04/13/2022, 12:39 PMingest status aspectDo you have any example?
b
better-orange-49102
04/13/2022, 1:08 PMhttps://datahubproject.io/docs/metadata-service/ under section soft deleting an entity
better-orange-49102
04/13/2022, 1:09 PMI'm surprised deleting in mysql didn't work though
better-orange-49102
04/13/2022, 1:10 PMAre you deleting them so that they can be subsequently re ingested?
c
colossal-easter-99672
04/15/2022, 7:19 AMI think reindex don't work.
Because now i have some datasets visible in UI which don't have any info in mysql. After reindex same result.
b
better-orange-49102
04/15/2022, 7:32 AMcould you share more on where those "doesnt exist in mysql" datasets are displayed? in the Datasets view page? inside a dataset lineage tab?
c
colossal-easter-99672
04/15/2022, 7:51 AMsome screenshots
b
better-orange-49102
04/15/2022, 8:05 AMoooh, interesting. No idea how this could have happened. ^ @big-carpet-38439
c
colossal-easter-99672
04/15/2022, 8:22 AMhow i do reindex:
Copy code
C:\Users\ntsyplakov>kubectl --namespace datahub create job --from=cronjob/datahub-datahub-restore-indices-job-template datahub-restore-indices-adhoc
job.batch/datahub-restore-indices-adhoc created
C:\Users\ntsyplakov>kubectl --namespace datahub get pod
NAME READY STATUS RESTARTS AGE
...
datahub-restore-indices-adhoc-jd4wr 0/1 Completed 0 28mcolossal-easter-99672
04/15/2022, 8:23 AMlogs
Copy code
Starting upgrade with id RestoreIndices...
Cleanup has not been requested.
Skipping Step 1/3: ClearSearchServiceStep...
Cleanup has not been requested.
Skipping Step 2/3: ClearGraphServiceStep...
Executing Step 3/3: SendMAEStep...
...
Successfully sent MAEs for 16797 rows
Completed Step 3/3: SendMAEStep successfully.
Success! Completed upgrade with id RestoreIndices successfully.
Upgrade RestoreIndices completed with result SUCCEEDED. Exiting...b
better-orange-49102
04/15/2022, 8:35 AMthat should have been the correct way to reindex
thank you 1
2 Views