crooked-rose-22807
08/19/2022, 12:10 PMsoft
delete. Since, Datahub still does not support stateful ingestion for Looker and LookML. I’m thinking of a way to have its own “stateful” ingestion by using soft delete and then ingest back the metadata. However, I noticed that after soft
delete, not all Looker components are showing as expected. For eg: before soft
delete, there will be 6.6k items and only 500++ showed up.
Is something wrong with the elasticsearch
indexes that causing this? Could this also be another bug?? Regardless, what is the best way to manage Looker metadata ingestion?mammoth-bear-12532
mammoth-bear-12532
crooked-rose-22807
08/24/2022, 12:54 PMdatahub delete --entity_type dashboard --platform looker --force && datahub delete --entity_type chart --platform looker --force && datahub delete --platform looker --force
crooked-rose-22807
08/24/2022, 12:56 PMcrooked-rose-22807
08/24/2022, 1:12 PMmammoth-bear-12532
miniature-policeman-55414
08/25/2022, 3:59 AMmammoth-bear-12532
crooked-rose-22807
08/26/2022, 7:27 AMadamant-van-21355
08/31/2022, 2:43 PMjava.lang.RuntimeException: Unknown aspect inputFields for entity dashboard
or java.lang.RuntimeException: Unknown aspect inputFields for entity chart
which did not seem to be the case before. Any clues about these warnings and the impact of those?
Thanksadamant-van-21355
08/31/2022, 2:43 PM...
...
{'warning': 'Unable to emit metadata to DataHub GMS',
'info': {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException',
'stackTrace': 'com.linkedin.restli.server.RestLiServiceException [HTTP Status:500]: java.lang.RuntimeException: Unknown '
'aspect inputFields for entity chart\n'
'\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:42)\n'
'\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:50)',
'message': 'java.lang.RuntimeException: Unknown aspect inputFields for entity chart',
'status': '500',
'id': 'urn:li:chart:(looker,dashboard_elements.5d477df629d88a5e1bd3e94f4c11db00)'}},
{'warning': 'Unable to emit metadata to DataHub GMS',
'info': {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException',
'stackTrace': 'com.linkedin.restli.server.RestLiServiceException [HTTP Status:500]: java.lang.RuntimeException: Unknown '
'aspect inputFields for entity dashboard\n'
'\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:42)\n'
'\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:50)',
'message': 'java.lang.RuntimeException: Unknown aspect inputFields for entity dashboard',
'status': '500',
'id': 'urn:li:dashboard:(looker,dashboards.finance::seu_executive_gemstone)'}}],
'failures': [],
'start_time': '2022-08-31 16:11:53.025862',
'current_time': '2022-08-31 16:35:37.729545',
'total_duration_in_seconds': '1424.7',
'gms_version': 'v0.8.43',
'pending_requests': '0'}
Pipeline finished with 8487 warnings ; produced 14109 events
adamant-van-21355
08/31/2022, 3:52 PMadamant-van-21355
09/01/2022, 4:31 PMmammoth-bear-12532
inputFields
is expected as the server (I'm assuming you are running 0.8.43) doesn't have that model. This warning should go away if you upgrade to 0.8.44