https://datahubproject.io logo
Join Slack
Powered by
Hey all, we’re currently running into some issues ...
# troubleshoot
e
Hey all, we’re currently running into some issues with our analytics tab. We’re running the graph implementation using AWS elastic search. Here’s the error message we’ve been getting:
Copy code
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "datahub_usage_event",
        "node": "3d1IH_U4T1OXbZJkCbNWtw",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
        }
      }
    ],
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory.",
      "caused_by": {
        "type": "illegal_argument_exception",
        "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
      }
    }
  },
  "status": 400
}
l
@early-lamp-41924 ^
e
Hi. did the elasticsearch setup job run? Could you check its logs?
Make sure you set USE_AWS_ELASTICSEARCH to true when setting up elasticsearch setup job as shown here! https://datahubproject.io/docs/deploy/aws#elasticsearch-service
👍 1
e
Alright let me rerun and see what happens
e
ok so
Need to follow these steps since the index was already created
you can delete index by running a curl like below
Copy code
curl --location --request DELETE 'https://<<elasticsearch-host-url>>/datahub_usage_event'
e
nice. Yeah we’re restarting gms now
we’re still getting the error, but we can at least see the indices in elasticsearch now
also, is there a way to reindex all created users? We might have deleted some things in our elastic search
do we have to use a specific engine type? We’re using opensearch 1.1
e
This should work!
e
@early-lamp-41924 thanks for that! We managed to restore our indicies. However, we’re still running into the same issue when we go to the analytics tab:
Copy code
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=illegal_argument_exception, reason=Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory.]
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
	at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437)
	... 28 common frames omitted
this is after we’ve recreated our index policy and all our indexes
e
so did you make sure you deleted the index while gms is down?
if there is an index named datahub_usage_event
instead of datahub_usage_event_00000 (or something like that)
that means gms dynamically created the index before your index template was there
need to fix this behavior but for now, this is the case 😞
so make sure gms is down
delete the index
run setup job
then bring gms back up
e
yep that was done
I’ll do it again to show
Here’s my process: 1. Stop the gms job on nomad 2. Delete the policy in opensearch (does this need to be an elastic search version or is opensearch ok?) 3. rerun the elastic search setup job 4. restart gms 5. observe error
here’s the created logs
here’s the env block for the setup job
here’s our GMS block env vars:
e
you see on line 82 datahub_usage_event_index_template exists
this means that the index was there before
you need to delete the datahub_usage_event index not the policy
👍 1
e
k let me give it a try
ok so I deleted the index and tried it again and got the same message
let me try deleting the index_template
ran this 
curl -XDELETE <https://vpc-datahub-xpzxeg6kfb7lm3iwl7wylmmi5y.us-east-1.es.amazonaws.com/_template/datahub_usage_event_index_template>
which deleted the template
deleted the index and the policy
reran the setup job and gms and the screen works now!
thanks @early-lamp-41924!
e
awesome!!!!
sorry this was a gruesome process
we will be making this better
e
it’s cool. The opensearch UI doesn’t have a place to see “templates”
and if you want we can try and contribute our nomad files or something similar back to the repo
e
That sounds great! Or at least a docs page detailing the bug and the resolution steps would be amazing!!!
l
Gah i just ran into this after rebuilding my environment. Good thing i saw this!
2 Views
Open in Slack
PreviousNext
Powered by