Hi folks, I recently ingest the metadata using con...
# troubleshoot
f
Hi folks, I recently ingest the metadata using connector. After creating ingestion source, there is nothing showed up even the successful notification appeared. I have run
docker system prune
and restart Elasticsearch and gms container to clean up. docker system df:
Copy code
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          25        8         16.19GB   7.369GB (45%)
Containers      8         8         929.7MB   0B (0%)
Local Volumes   147       9         26.47GB   24.84GB (93%)
Build Cache     25        0         0B        0B
The logs from Elasticsearch:
Copy code
{"type": "server", "timestamp": "2023-04-03T04:25:30,850Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "flood stage disk watermark [95%] exceeded on [CA9sNvGOSr2sNBFRMst0JQ][elasticsearch][/usr/share/elasticsearch/data/nodes/0] free: 9.3gb[1.9%], all indices on this node will be marked read-only", "cluster.uuid": "M3xdtmw8TFCGL_RqIP650Q", "node.id": "CA9sNvGOSr2sNBFRMst0JQ"  }
{"type": "server", "timestamp": "2023-04-03T04:26:00,855Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "flood stage disk watermark [95%] exceeded on [CA9sNvGOSr2sNBFRMst0JQ][elasticsearch][/usr/share/elasticsearch/data/nodes/0] free: 9.3gb[1.9%], all indices on this node will be marked read-only", "cluster.uuid": "M3xdtmw8TFCGL_RqIP650Q", "node.id": "CA9sNvGOSr2sNBFRMst0JQ"  }
{"type": "server", "timestamp": "2023-04-03T04:26:30,857Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "flood stage disk watermark [95%] exceeded on [CA9sNvGOSr2sNBFRMst0JQ][elasticsearch][/usr/share/elasticsearch/data/nodes/0] free: 9.3gb[1.9%], all indices on this node will be marked read-only", "cluster.uuid": "M3xdtmw8TFCGL_RqIP650Q", "node.id": "CA9sNvGOSr2sNBFRMst0JQ"  }
Logs from gms:
Copy code
04:26:25.397 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.h.i.IngestionSchedulerHook:56 - Received UPSERT to Ingestion Source. Rescheduling the source (if applicable). urn: urn:li:dataHubIngestionSource:3b33a8fc-b106-460f-b90b-3ca816c77910, key: {value=ByteString(length=45,bytes=7b226964...3130227d), contentType=application/json}.
04:26:25.398 [ThreadPoolTaskExecutor-1] INFO  c.d.m.ingestion.IngestionScheduler:105 - Unscheduling ingestion source with urn urn:li:dataHubIngestionSource:3b33a8fc-b106-460f-b90b-3ca816c77910
04:26:25.399 [ThreadPoolTaskExecutor-1] INFO  c.d.m.ingestion.IngestionScheduler:138 - Scheduling next execution of Ingestion Source with urn urn:li:dataHubIngestionSource:3b33a8fc-b106-460f-b90b-3ca816c77910. Schedule: 0 1 * * *
04:26:25.400 [ThreadPoolTaskExecutor-1] INFO  c.d.m.ingestion.IngestionScheduler:167 - Scheduled next execution of Ingestion Source with urn urn:li:dataHubIngestionSource:3b33a8fc-b106-460f-b90b-3ca816c77910 in 48814600ms.
04:26:27.448 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.t.DataHubUsageEventTransformer:74 - Invalid event type: CreateIngestionSourceEvent
04:26:27.449 [ThreadPoolTaskExecutor-1] WARN  c.l.m.k.DataHubUsageEventsProcessor:56 - Failed to apply usage events transform to record: {"type":"CreateIngestionSourceEvent","sourceType":"bigquery","interval":"0 1 * * *","actorUrn":"urn:li:corpuser:datahub","timestamp":1680495987418,"date":"Mon Apr 03 2023 11:26:27 GMT+0700 (Indochina Time)","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36","browserId":"c7e9ab30-25a7-4874-a4e6-f2c694501ccc"}
04:26:27.474 [qtp522764626-22] INFO  c.l.m.r.entity.AspectResource:143 - INGEST PROPOSAL proposal: {aspectName=dataHubExecutionRequestResult, entityKeyAspect={contentType=application/json, value=ByteString(length=46,bytes=7b226964...3463227d)}, entityType=dataHubExecutionRequest, aspect={contentType=application/json, value=ByteString(length=51,bytes=7b227374...3437307d)}, changeType=UPSERT}
04:26:27.485 [pool-12-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 11ms
04:26:27.495 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.t.DataHubUsageEventTransformer:74 - Invalid event type: ExecuteIngestionSourceEvent
04:26:27.495 [ThreadPoolTaskExecutor-1] WARN  c.l.m.k.DataHubUsageEventsProcessor:56 - Failed to apply usage events transform to record: {"type":"ExecuteIngestionSourceEvent","actorUrn":"urn:li:corpuser:datahub","timestamp":1680495987487,"date":"Mon Apr 03 2023 11:26:27 GMT+0700 (Indochina Time)","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36","browserId":"c7e9ab30-25a7-4874-a4e6-f2c694501ccc"}
04:26:28.968 [I/O dispatcher 1] ERROR c.l.m.s.e.update.BulkListener:25 - Failed to feed bulk request. Number of events: 4 Took time ms: -1 Message: failure in bulk execution:
[0]: index [datahubingestionsourceindex_v2], type [_doc], id [urn%3Ali%3AdataHubIngestionSource%3A3b33a8fc-b106-460f-b90b-3ca816c77910], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [datahubingestionsourceindex_v2] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]
[1]: index [system_metadata_service_v1], type [_doc], id [D9J9LxuD6yBN4lJYY0FaMg==], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [system_metadata_service_v1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]
[2]: index [datahubingestionsourceindex_v2], type [_doc], id [urn%3Ali%3AdataHubIngestionSource%3A3b33a8fc-b106-460f-b90b-3ca816c77910], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [datahubingestionsourceindex_v2] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]
[3]: index [system_metadata_service_v1], type [_doc], id [F7B9+ecZWomYYe5LYipdcA==], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [system_metadata_service_v1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]
04:26:29.528 [qtp522764626-17] INFO  c.l.m.r.entity.AspectResource:143 - INGEST PROPOSAL proposal: {aspectName=dataHubExecutionRequestResult, entityKeyAspect={contentType=application/json, value=ByteString(length=46,bytes=7b226964...3463227d)}, entityType=dataHubExecutionRequest, aspect={contentType=application/json, value=ByteString(length=389,bytes=7b227374...3437307d)}, changeType=UPSERT}
04:26:29.555 [pool-12-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 27ms
04:26:29.749 [qtp522764626-17] INFO  c.l.m.r.entity.AspectResource:93 - GET ASPECT urn: urn:li:telemetry:clientId aspect: telemetryClientId version: 0
04:26:29.752 [pool-12-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - GET /aspects/urn%3Ali%3Atelemetry%3AclientId?aspect=telemetryClientId&version=0 - get - 200 - 3ms
04:26:31.566 [qtp522764626-23] INFO  c.l.m.r.entity.AspectResource:143 - INGEST PROPOSAL proposal: {aspectName=dataHubExecutionRequestResult, entityKeyAspect={contentType=application/json, value=ByteString(length=46,bytes=7b226964...3463227d)}, entityType=dataHubExecutionRequest, aspect={contentType=application/json, value=ByteString(length=654,bytes=7b227374...3437307d)}, changeType=UPSERT}
04:26:31.986 [I/O dispatcher 1] ERROR c.l.m.s.e.update.BulkListener:25 - Failed to feed bulk request. Number of events: 8 Took time ms: -1 Message: failure in bulk execution:
[0]: index [datahubexecutionrequestindex_v2], type [_doc], id [urn%3Ali%3AdataHubExecutionRequest%3A67e66178-9cf9-40ac-8496-ac26761ba24c], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [datahubexecutionrequestindex_v2] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]
[1]: index [datahubexecutionrequestindex_v2], type [_doc], id [urn%3Ali%3AdataHubExecutionRequest%3A67e66178-9cf9-40ac-8496-ac26761ba24c], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [datahubexecutionrequestindex_v2] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]
[2]: index [graph_service_v1], type [_doc], id [t/DPfLYDIXzDBvDNKYzsBA==], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [graph_service_v1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]
[3]: index [system_metadata_service_v1], type [_doc], id [mK6kjzTpk8iRRsx5Eg7tJA==], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [system_metadata_service_v1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]
[4]: index [system_metadata_service_v1], type [_doc], id [yss7L2uhB/qEk71o8Vkc8w==], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [system_metadata_service_v1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]
[5]: index [datahubexecutionrequestindex_v2], type [_doc], id [urn%3Ali%3AdataHubExecutionRequest%3A67e66178-9cf9-40ac-8496-ac26761ba24c], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [datahubexecutionrequestindex_v2] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]
[6]: index [datahubexecutionrequestindex_v2], type [_doc], id [urn%3Ali%3AdataHubExecutionRequest%3A67e66178-9cf9-40ac-8496-ac26761ba24c], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [datahubexecutionrequestindex_v2] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]
[7]: index [system_metadata_service_v1], type [_doc], id [QVHUM/x8ssrKDaAcXWOgng==], message [ElasticsearchException[Elasticsearch exception [type=cluster_block_exception, reason=index [system_metadata_service_v1] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]
04:26:31.987 [pool-12-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 421ms
I run datahub locally. Is it because of resource issue? how can I check and solve it? Thanks in advance!
✅ 1
f
It seems your ES is unhealthy. Take a look at your Elasticsearch resource.
Copy code
message": "flood stage disk watermark [95%] exceeded on [CA9sNvGOSr2sNBFRMst0JQ][elasticsearch][/usr/share/elasticsearch/data/nodes/0] free: 9.3gb[1.9%], all indices on this node will be marked read-only
f
Yes, I just checked again. It's because the disk is almost full 😀 I did a clean-up again and it worked. Thank you for your support!