Hi! I’m running DataHub on K8S. Can someone help me with error situation where I get 500 from the backend and nothing gets displayed on frontend when I try to access a dataset. Following is the error message in DataHub GMS logs.

org.elasticsearch.client.ResponseException: method [POST], host [<https://opensearch-service.com:11862>], URI [/dataset_datasetusagestatisticsaspect_v1/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 500 Internal Server Error]
{"error":{"root_cause":[{"type":"exception","reason":"java.util.concurrent.ExecutionException: java.lang.IllegalStateException: unexpected docvalues type NONE for field 'topSqlQueries' (expected one of [SORTED, SORTED_SET]). Re-index with correct docvalues type."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"dataset_datasetusagestatisticsaspect_v1","node":"lSc8kpchRyGhN_lPDqNyvQ","reason":{"type":"exception","reason":"java.util.concurrent.ExecutionException: java.lang.IllegalStateException: unexpected docvalues type NONE for field 'topSqlQueries' (expected one of [SORTED, SORTED_SET]). Re-index with correct docvalues type.","caused_by":{"type":"execution_exception","reason":"java.lang.IllegalStateException: unexpected docvalues type NONE for field 'topSqlQueries' (expected one of [SORTED, SORTED_SET]). Re-index with correct docvalues type.","caused_by":{"type":"illegal_state_exception","reason":"unexpected docvalues type NONE for field 'topSqlQueries' (expected one of [SORTED, SORTED_SET]). Re-index with correct docvalues type."}}}}]},"status":500}

Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ button ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? ✅ button Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!

Thats strange - it says there's something wrong with topSqlQueries. Did you ingest usage stats from a data source? how was the recipe looked like and was it successful?

I have only Kafka and BigQuery datasets in my DataHub both ingested from recipes using source: rest I assume usage stats are generated and ingested by GMS backend server based on usage from the front end. I looked at the index in error message, it has the field ’topSqlQueries’. Can’t see anything on how docvalues can be changed on this individual field as it complains default NONE is the problem Btw, I’m using Aiven managed OpenSearch as index service

Okay, this seems to be an issue on our side - we'll get back to you! @gentle-hamburger-31302

Facing the same problem, has this issue been resolved?

how it's going ? the same problem happen in v0.10.4

@big-carpet-38439 / @bulky-soccer-26729 might help you

Seeing this today too out of nowhere. Effecting every asset