Hello, I had faced some strange issue when ingesting data where delta-lake is a source with tables located in the s3. In my case the pipeline logs after ingestion show that the number of ingested rows in not 0, although nothing in the UI is visible due to the error:

[6]: index [containerindex_v2], type [_doc], id [urn%3Ali%3Acontainer%3A25826ac8da9e99f119c577ef4ccc2194], message [[containerindex_v2/0uhknRQkSmOD3pb23xdi0g][[containerindex_v2][0]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][urn%3Ali%3Acontainer%3A25826ac8da9e99f119c577ef4ccc2194]: document missing]]]

A little bit later this error in logs pops up, but I'm not sure if it's connected to the elasticsearch issue:

10:56:59.617 [pool-13-thread-1] ERROR c.l.g.factory.telemetry.DailyReport:110 - Error reporting telemetry:
java.net.SocketTimeoutException: connect timed out
        at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
        at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
        at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.base/java.net.Socket.connect(Socket.java:609)
        at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305)
        at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:507)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:602)
        at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
        at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:373)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:207)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:193)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1367)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1342)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246)
        at com.mixpanel.mixpanelapi.MixpanelAPI.sendData(MixpanelAPI.java:134)
        at com.mixpanel.mixpanelapi.MixpanelAPI.sendMessages(MixpanelAPI.java:172)
        at com.mixpanel.mixpanelapi.MixpanelAPI.deliver(MixpanelAPI.java:103)
        at com.mixpanel.mixpanelapi.MixpanelAPI.deliver(MixpanelAPI.java:83)
        at com.mixpanel.mixpanelapi.MixpanelAPI.sendMessage(MixpanelAPI.java:71)
        at com.linkedin.gms.factory.telemetry.DailyReport.ping(DailyReport.java:108)
        at com.linkedin.gms.factory.telemetry.DailyReport.dailyReport(DailyReport.java:97)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

Logs from the ingestion:

'warnings': {},
 'failures': {},
 'filtered': [],
 'start_time': '2022-11-22 10:31:13.916942 (1.15 seconds ago).',
 'running_time': '1.15 seconds'}
Sink (datahub-rest) report:
{'total_records_written': '32',
 'records_written_per_second': '17',
 'warnings': [],
 'failures': [],
 'start_time': '2022-11-22 10:31:13.248079 (1.82 seconds ago).',
 'current_time': '2022-11-22 10:31:15.069306 (now).',
 'total_duration_in_seconds': '1.82',
 'gms_version': 'v0.9.2',
 'pending_requests': '0'}

 Pipeline finished successfully; produced 32 events in 1.15 seconds.

So far as I understood the pipeline has finished, but there is some weird error within the elasticsearch itself, can you guys help resolve the issue? docker prune did not help to resolve the problem as was suggested somewhere else. Thanks in advance!

hey Pawel! yeah I'm not sure if those error logs are related but rather an issue with telemetry, but shouldn't affect your ability to ingest and see data. as far as the elasticsearch exception goes, I'm not positive what could have caused it. But based on looking at some other threads and doing a search on some ES forums, it appears that ES is querying for data that doesn't exist yet. This could be because you're ingesting partial data where it expects some data to exist already (like the case in this thread) or maybe because there could be an issue with your kafka instance (like the case in this thread). Let us know if this sounds like it might be helpful or if you continue to have issues!

Hi @bulky-soccer-26729 I'm not sure why kafka would be related here, my source is a delta-lake and as for the thread with missing partial data, I'm not sure what does it mean to "Ingest the definitions" do you know what that could mean?

hmm well kafka could potentially be a problem since that's how datahub passes information between systems, so after ingesting your delta-lake data, we pass those metadata change events to elasticsearch so that we can index the data properly (not that you're ingesting external kafka data itself). and I think what they mean about ingesting definitions is that is it possible you're only ingesting part of the whole picture of your data? like some data that you ingest is expecting there to be container data present that isn't? i'm not sure how likely this situation is

@bulky-soccer-26729 Actually I tried to do fresh docker-compose rerun of datahub and guess what - it helped, no longer this issue was visible -> restart helped. It looks like guys from "IT - crowd" were right after all, turning off and on again does help... who would have thought. Anyways, thanks for helping!

Haha amazing! The old turn it off and on strikes again