Hi All - We are running v10.4 and v10.5 in our different environments. We have recently seen that in the Analytics screen, the KPI metrics are no longer showing up. For e.g the weekly active users. Is there some config which controls this behaviour or do these KPIs show up automatically ? If the latter, how can we troubleshoot whats wrong ? @bulky-soccer-26729 @big-carpet-38439 @gray-shoe-75895

Hi - i'm curious, the KPI does not show up after searching/browsing through entities? @bulky-soccer-26729 could you confirm if this is an expected behavior?

hey Saral! so did this start happening after an upgrade? and you used to be able to see KPI metrics but you no longer can? also - by KPI metrics do you mean the metrics you screenshotted there or the usage stats like monthly active users etc?

I mean the monthly,weekly users. yes it seems after 10.4 this problem started. I noticed the getAnalyticsCharts query is returning empty. Is it possible the datahub_usage index is corrupted or broken ?

yeah definitely. looking into this now!

yes we are...its just the top row weekly/monthly user counts and dataset counts etc. that have dissappeared

okay gotcha. do you see charts down below for DataHub Usage Analytics like my screenshot?

understood. SOrry for not showing the entire screen. here you go -

hmm okay gotcha. and one more thing if you could - do you mind opening up your browser's network tab and check out the graphql calls we make to get data for this page (you can see them when you first navigate to this Analytics page) that are called

getAnalyticsCharts

and

getHighlights

? I'm curious if there's any errors or anything that we're not surfacing in the UI. same thing with in your GMS logs

image.png,image.png,image.png

okay nice thanks for that

2023-09-19 14:22:26,989 [Thread-11656] WARN  org.elasticsearch.client.RestClient - request [POST <http://elasticsearch-master-hl:9200/datasetindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true>] returned 1 warnings: [299 Elasticsearch-8.9.1-a813d015ef1826148d9d389bd1c0d781c6e349f0 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]

2023-09-19 14:22:26,991 [Thread-11653] WARN  org.elasticsearch.client.RestClient - request [POST <http://elasticsearch-master-hl:9200/chartindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true>] returned 1 warnings: [299 Elasticsearch-8.9.1-a813d015ef1826148d9d389bd1c0d781c6e349f0 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]

2023-09-19 14:22:27,002 [Thread-11662] WARN  org.elasticsearch.client.RestClient - request [POST <http://elasticsearch-master-hl:9200/datasetindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true>] returned 1 warnings: [299 Elasticsearch-8.9.1-a813d015ef1826148d9d389bd1c0d781c6e349f0 "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]

2023-09-19 14:22:27,131 [I/O dispatcher 1] ERROR c.l.m.s.e.update.BulkListener - Error feeding bulk request. No retries left. Request: Failed to perform bulk request: index [datahub_usage_event], optype: [CREATE], type [_doc], id [PageViewEvent_urn%3Ali%3Acorpuser%3Agky249_1695133346825_38667]

java.io.IOException: Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=<http://elasticsearch-master-hl:9200>, response=HTTP/1.1 200 OK}

at org.elasticsearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:1783)

at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:636)

at org.elasticsearch.client.RestClient$1.completed(RestClient.java:376)

at org.elasticsearch.client.RestClient$1.completed(RestClient.java:370)

at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)

at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)

at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)

at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)

at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)

at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)

at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)

at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)

at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)

at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)

at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)

at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)

at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)

at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)

at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: java.lang.NullPointerException: null

at java.base/java.util.Objects.requireNonNull(Objects.java:221)

at org.elasticsearch.action.DocWriteResponse.<init>(DocWriteResponse.java:127)

at org.elasticsearch.action.index.IndexResponse.<init>(IndexResponse.java:54)

at org.elasticsearch.action.index.IndexResponse.<init>(IndexResponse.java:39)

at org.elasticsearch.action.index.IndexResponse$Builder.build(IndexResponse.java:107)

at org.elasticsearch.action.index.IndexResponse$Builder.build(IndexResponse.java:104)

at org.elasticsearch.action.bulk.BulkItemResponse.fromXContent(BulkItemResponse.java:159)

at org.elasticsearch.action.bulk.BulkResponse.fromXContent(BulkResponse.java:188)

at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911)

at org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAsyncAndParseEntity$10(RestHighLevelClient.java:1699)

at org.elasticsearch.client.RestHighLevelClient$1.onSuccess(RestHighLevelClient.java:1781)

... 18 common frames omitted

Would you be able to query ElasticSearch directly to see if the data is still present? I remember there was a bug with the usage indices getting truncated a few versions back. If that's the case, you may want to restore them from snapshots.