Hi all, I am using Datahub with helm deployment an...
# troubleshoot
b
Hi all, I am using Datahub with helm deployment and was moving it to use our own elastic instance (ver 7.17.3). though it’s working I’m getting some errors (inside the thread) also I can’t see any index template was created and my datahub usage event index name is
datahub_datahub_usage_event
(I set datahub prefix for all indices) could anyone assist please?
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: 1️⃣ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? 2️⃣ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues?
b
Error:
Copy code
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=illegal_argument_exception, reason=Fielddata is disabled on text fields by default. Set fielddata=true on [browserId] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.]
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
	at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437)
	at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:603)
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:179)
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=illegal_argument_exception, reason=Fielddata is disabled on text fields by default. Set fielddata=true on [browserId] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.]
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
	at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437)
f
Did you run
restore-indices-job
after moving to your own elastic-search instance? https://datahubproject.io/docs/how/restore-indices/#kubernetes
b
not at first
nope
I did run it now, but I think I should take more steps before
the problem still persist. i tried it from scratch on anoter ES instance and it worked but UI is empty and Im getting this error
Copy code
FORBIDDEN/8/index write
f
I don’t know how did you deploy your own ES instance. As this error, it seems like the user or role that you are using to write to ES index does not have permissions to do so. Would you mind checking this information?
b
I will check. From what I already know, we need to create each index in a ticket, stating the retention and possible size. But there are many indices created (20+) And im not sure what policy to ask for each one. Thats why im asking about the template
I’m not sure what’s the requirements for each index and possible sizes Is there any defaults here? I see indices was ceeated automatically by datahub but probably thats why I don’t have permissions. Maybe the template is covering all and need to give him permission, but when trying to fetch the template i see it’s empty. Can you give any direction here please?
f
Here are a few things you can try to resolve this issue (kudos to ChatGPT 🤖): 1. Check the Elasticsearch security settings. Make sure that the user or role that you are using to write to the index has the necessary permissions. You can check the security settings by running the following command:
Copy code
curl -u {{username}}:{{password}} -XGET 'http://{{elasticsearch_host}}:{{elasticsearch_port}}/_security/_authenticate?pretty'
Replace
{{username}}
and
{{password}}
with the credentials for the user or role that you are using to write to the index, and replace
{{elasticsearch_host}}
and
{{elasticsearch_port}}
with the hostname and port of your Elasticsearch instance. This command will return information about the security settings for your Elasticsearch instance, including the roles and permissions for each user. 2. Check the index settings. Make sure that the index that you are trying to write to exists and that it has the correct settings. You can check the index settings by running the following command:
Copy code
curl -u {{username}}:{{password}} -XGET 'http://{{elasticsearch_host}}:{{elasticsearch_port}}/{{index_name}}/_settings?pretty'
Replace
{{username}}
and
{{password}}
with the credentials for the user or role that you are using to write to the index, replace
{{elasticsearch_host}}
and
{{elasticsearch_port}}
with the hostname and port of your Elasticsearch instance, and replace
{{index_name}}
with the name of the index that you are trying to write to. This command will return information about the settings for the specified index, including the permissions and access control settings. 3. Check the Elasticsearch logs. The Elasticsearch logs may contain more information about the error, including the specific reason why the write operation was forbidden. Check the logs for any error messages or warnings that may be related to the issue.
b
thanks! I meant how do I know which indices to add and what is the recommended retention? how can I get the list besides fetching the current ones created by datahub
I didn’t find it in the docs
a
is there a reason you’re unable to read them from DH?
b
I am unable to write, this is internal issue with our ES, and I need to ask for permissions but I wonder how can I map all the relevant indices of datahub?
I see there 52 indices that were created using the kibana console
GET _cat/indices/*datahub*?v&s=index
are they all being used (read/write)?
a
That seems like a lot, how much data are you ingesting ?
b
I can see all the indices in the doc you sent there and with some others. here is the list
Copy code
green open datahub_mlmodeldeploymentindex_v2                                ZBicOmpwQnWTqJmhbRSvpg 1 1   0 0    452b    226b
green open datahub_datahubretentionindex_v2                                 zu-I-Z0pTKuIRiiIu01vAg 1 1   0 0    452b    226b
green open datahub_dataprocessinstanceindex_v2                              ew81a-BmQ7uQAQ_lql9k6w 1 1   0 0    452b    226b
green open datahub_corpuserindex_v2                                         2KbK-dfnTTmqPN4c_RXtaA 1 1   0 0    452b    226b
green open datahub_system_metadata_service_v1                               OIQCoYBdQ1azfIsDPzAPCg 1 1   0 0    452b    226b
green open datahub_corpgroupindex_v2                                        K8nuGQmLTrmD0gjT0Z99UA 1 1   0 0    452b    226b
green open datahub_datajob_datahubingestionrunsummaryaspect_v1              55bUuVCVSpanaEVtjtZURA 1 1   0 0    452b    226b
green open datahub_mlfeatureindex_v2                                        ikl8qx-tTsagXy90ylbFBw 1 1   0 0    452b    226b
green open datahub_datajob_datahubingestioncheckpointaspect_v1              m93R_VzaQPW87If98gfGeA 1 1   0 0    452b    226b
green open datahub_datahubingestionsourceindex_v2                           ML94hTvLSeKajsSakhdxzg 1 1   0 0    452b    226b
green open datahub_datajobindex_v2                                          D8IeeCasQLeBKUtMVUwUEA 1 1   0 0    452b    226b
green open datahub_dataplatforminstanceindex_v2                             Rn4e_1UwRuqoQV70pgkMbQ 1 1   0 0    452b    226b
green open datahub_tagindex_v2                                              sXftiFrXT_idwlnyGSuxKw 1 1   0 0    452b    226b
green open datahub_dataplatformindex_v2                                     Lzfcq-JRSAGNFmBskyBJ7Q 1 1   0 0    452b    226b
green open .ds-datahub_datahub_usage_event-2023.04.18-000003                40d0cua6QnioGVkig16qVw 1 1 186 0 472.1kb 240.6kb
green open datahub_domainindex_v2                                           XqyoBsQUS3S4hgGfEglVJg 1 1   0 0    452b    226b
green open datahub_dataset_datasetusagestatisticsaspect_v1                  VjizoVvnR9egjOFX_N6J-Q 1 1   0 0    452b    226b
green open datahub_postindex_v2                                             MR3BqAWuTriFPLawaq0aNg 1 1   0 0    452b    226b
green open datahub_dataprocessindex_v2                                      Li_p4AN-SUOHO4tQSF3XRw 1 1   0 0    452b    226b
green open datahub_dataprocessinstance_dataprocessinstanceruneventaspect_v1 AgJskrK8QeO9jIEncdf_7g 1 1   0 0    452b    226b
green open datahub_dashboardindex_v2                                        Xg-Rbx6lRpCdX9EM7YjEPQ 1 1   0 0    452b    226b
green open datahub_datahubaccesstokenindex_v2                               lYtUc14DTh6lHiR_RbUXcQ 1 1   0 0    452b    226b
green open datahub_schemafieldindex_v2                                      SWU3ZUHHQneWikEDqk8o-w 1 1   0 0    452b    226b
green open datahub_datahubviewindex_v2                                      tyFN8MjfSqCxXxnUQgc4XA 1 1   0 0    452b    226b
green open datahub_chartindex_v2                                            eHxiVMfySTSB8mwPKoYHiA 1 1   0 0    452b    226b
green open datahub_dataflowindex_v2                                         WgTmB_qoSMyPMFyTvaJBgg 1 1   0 0    452b    226b
green open datahub_globalsettingsindex_v2                                   kYVjDsm8QDSWG_lGLWfriw 1 1   0 0    452b    226b
green open datahub_assertion_assertionruneventaspect_v1                     YyMTQ1McRxG9nbikMjkSiQ 1 1   0 0    452b    226b
green open datahub_mlfeaturetableindex_v2                                   R3mLhdLKT8GqUYhlL9C1JQ 1 1   0 0    452b    226b
green open datahub_glossarytermindex_v2                                     3cekXFUeS0e2afbeITk2Uw 1 1   0 0    452b    226b
green open datahub_dataset_datasetprofileaspect_v1                          49QqAMnAQtiLEKbLjoKmeA 1 1   0 0    452b    226b
green open datahub_mlmodelindex_v2                                          wu2AFq1cSbu6rKvdaNZInw 1 1   0 0    452b    226b
green open datahub_mlmodelgroupindex_v2                                     L-02sdv9RqOIVcK7yzFD9A 1 1   0 0    452b    226b
green open datahub_datahubroleindex_v2                                      7UEqDEULQUWHVSpoQ-5JxA 1 1   0 0    452b    226b
green open datahub_datahubstepstateindex_v2                                 juvEpiXETIS-vO981rFXPQ 1 1   0 0    452b    226b
green open datahub_dataset_operationaspect_v1                               SXuHVb3qSGuQX9eCqBaYxg 1 1   0 0    452b    226b
green open datahub_datasetindex_v2                                          5Pm0nv1OSPWR5BzR_ey8GA 1 1   0 0    452b    226b
green open datahub_datahubexecutionrequestindex_v2                          8J5qC3uLQBOIEco6wh3Ozg 1 1   0 0    452b    226b
green open datahub_notebookindex_v2                                         eZmBK8zuS2m1GKULGSR3KA 1 1   0 0    452b    226b
green open datahub_containerindex_v2                                        I1rOK8JOSvK4HOXond_v4g 1 1   0 0    452b    226b
green open datahub_datahubpolicyindex_v2                                    oQAOuamiRh6PDEWaWSU69Q 1 1   0 0    452b    226b
green open datahub_dashboard_dashboardusagestatisticsaspect_v1              rG_jeFxxSw686wLpxLb5HA 1 1   0 0    452b    226b
green open datahub_telemetryindex_v2                                        _ObwBSe4SficdUPXAq6ZXA 1 1   0 0    452b    226b
green open datahub_mlprimarykeyindex_v2                                     ZJRtR9vrSkeoWpXb8906Sw 1 1   0 0    452b    226b
green open datahub_datahubsecretindex_v2                                    ZzC_2VVLTQmS4Rm-nXVRbw 1 1   0 0    452b    226b
green open datahub_glossarynodeindex_v2                                     spTsecNqRzyj2MqnhxuUXw 1 1   0 0    452b    226b
green open datahub_datahubupgradeindex_v2                                   uzTIjMaeTCSXTZUCB7iyhw 1 1   0 0    452b    226b
green open datahub_invitetokenindex_v2                                      k8St4GhUStKz2QiCsOz7YQ 1 1   0 0    452b    226b
green open datahub_chart_chartusagestatisticsaspect_v1                      7ZWeH-MMRKKE9CBJio0BIg 1 1   0 0    452b    226b
green open .ds-datahub_datahub_usage_event-2023.04.04-000001                y37A_vXOTOmpk59mR-1CMQ 1 1  11 0  87.7kb  33.2kb
green open datahub_testindex_v2                                             xK8rpFtTQRCUFF_kAT0aaA 1 1   0 0    452b    226b
green open datahub_assertionindex_v2                                        NAvuLtdXTcyCGL_IczdvtQ 1 1   0 0    452b    226b
green open datahub_graph_service_v1                                         L2w3bxGpT0yHaV2jTOIxuA 1 1   0 0    452b    226b
green open .ds-datahub_datahub_usage_event-2023.04.11-000002                _abEzxhFRqiEb0JGAgyajw 1 1  14 0 131.8kb  96.8kb
isn’t there a closed list of indices names so I can create them beforehand to fit our policy? why is there so many indices? what is the retention recomended for each?
a
I’m unsure here- is this due to resource limitations on your ES? I’ll see what our platform team says
b
Yes, we have a procedure for creating indices, where we must write the retention. If I just let datahub create the indices i have no write permissions to them. So im asking is there any retention for the indexes?
o
Retention for DataHub indices should be configured to permanent (as is the default with ElasticSearch). If you set retention you will have to restore data or it will not longer be searchable. Supported size is unbounded, it will depend on your data size. Can vary from a few hundred MB -> dozens of GB. We don't have insight into what data you plan on onboarding so can't really give a recommendation here. The indices created are dynamically generated by the model, if you add entities to the model in a fork, then you would be adding new indices. Since we provide an extensible model, we can't technically give an exhaustive list of all the indices that will be generated unless you are running with the default code, in which case the above list is reflective of the current list of indices. 52 seems right to me, we create an index for every main entity along with indices for any time series aspects on each entity. We have 41 supported entities: https://github.com/datahub-project/datahub/blob/master/metadata-models/src/main/resources/entity-registry.yml and several of them have multiple timeseries aspects associated.
b
Thank you for the detailed answer! so in that case it might be better for us to use the dedicated elastic in the prerequisites 🤔 so if indices size can get to dozens of GB, we should set the persistence in k8s accordingly right?
o
Yep, for larger scale deployments the default configuration for the prerequisite container is probably not sufficient. That being said, dozens of GB would be for very large companies onboarding all of their data. It's much more likely to fall into the 500 MB -> 1.5 GB range for smaller or medium sized companies, with that mostly being the size for the dataset index with most of the other indices being smaller. Generally we see the graph, dataset and timeseries aspect indices being the disproportionately large indices.
There are definite drawbacks to using the prerequisite container over a managed service for the backing system as well since you are taking the managed portion into your own hands, but that may be your best option if your infrastructure team is inflexible.
b
Thanks 🙏🏼