rapid-sundown-8805
10/04/2021, 9:07 AMsome-cricket-23089
10/04/2021, 11:15 PMred-pizza-28006
10/05/2021, 12:32 PMhandsome-football-66174
10/05/2021, 5:42 PMkind-dawn-17532
10/05/2021, 7:00 PMhandsome-football-66174
10/08/2021, 1:52 PMSuppressed: org.elasticsearch.client.ResponseException: method [POST], host [<hostname>:443], URI [/demo_datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request]
Copy code{ "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory." } ], "type": "search_phase_execution_exception", "reason": "all shards failed", "phase": "query", "grouped": true, "failed_shards": [ { "shard": 0, "index": "demo_datahub_usage_event", "node": "abc", "reason": { "type": "illegal_argument_exception", "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory." } } ], "caused_by": { "type": "illegal_argument_exception", "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory.", "caused_by": { "type": "illegal_argument_exception", "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory." } } }, "status": 400 }
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
... 21 common frames omitted
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=illegal_argument_exception, reason=Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory.]
at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496)
at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407)
at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437)
at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:603)
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:179)
... 24 common frames omitted
handsome-football-66174
10/11/2021, 3:51 PMvictorious-ambulance-32469
10/11/2021, 5:20 PMfresh-fish-73471
10/11/2021, 9:09 PMquaint-lighter-4354
10/13/2021, 11:58 AM2021/10/13 11:17:40 Waiting for: https://<my-es-url>:443
2021/10/13 11:17:45 Received 200 from https://<my-es-url>:443
creating datahub_usage_event_policy
{
"policy": {
"policy_id": "datahub_usage_event_policy",
"description": "Datahub Usage Event Policy",
"default_state": "Rollover",
"schema_version": 1,
"states": [
{
"name": "Rollover",
"actions": [
{
"rollover": {
"min_index_age": "1d"
}
}
],
"transitions": [
{
"state_name": "ReadOnly",
"conditions": {
"min_index_age": "7d"
}
}
]
},
{
"name": "ReadOnly",
"actions": [
{
"read_only": {}
}
],
"transitions": [
{
"state_name": "Delete",
"conditions": {
"min_index_age": "60d"
}
}
]
},
{
"name": "Delete",
"actions": [
{
"delete": {}
}
],
"transitions": []
}
],
"ism_template": {
"index_patterns": [
"datahub_usage_event-*"
],
"priority": 100
}
}
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 1061 0 0 100 1061 0 5055 --:--:-- --:--:-- --:--:-- 5052
100 1807 100 746 100 1061 842 1198 --:--:-- --:--:-- --:--:-- 2041
}{"_id":"datahub_usage_event_policy","_version":1,"_primary_term":1,"_seq_no":0,"policy":{"policy":{"policy_id":"datahub_usage_event_policy","description":"Datahub Usage Event Policy","last_updated_time":1634123866020,"schema_version":1,"error_notification":null,"default_state":"Rollover","states":[{"name":"Rollover","actions":[{"rollover":{"min_index_age":"1d"}}],"transitions":[{"state_name":"ReadOnly","conditions":{"min_index_age":"7d"}}]},{"name":"ReadOnly","actions":[{"read_only":{}}],"transitions":[{"state_name":"Delete","conditions":{"min_index_age":"60d"}}]},{"name":"Delete","actions":[{"delete":{}}],"transitions":[]}],"ism_template":[{"index_patterns":["datahub_usage_event-*"],"priority":100,"last_updated_time":1634123866020}]}}}
creating datahub_usagAe_event_index_template
{
"index_patterns": ["datahub_usage_event-*"],
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"type": {
"type": "keyword"
},
"timestamp": {
"type": "date"
},
"userAgent": {
"type": "keyword"
},
"browserId": {
"type": "keyword"
}
}
},
"settings": {
"index.opendistro.index_state_management.rollover_alias": "datahub_usage_event"
}
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 467 100 21 100 446 180 3824 --:--:-- --:--:-- --:--:-- 4025
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 59 0 0 100 59 0 287 --:--:-- --:--:-- --:--:-- 287
100 144 100 85 100 59 283 196 --:--:-- --:--:-- --:--:-- 480
2021/10/13 11:17:47 Command finished successfully.
}{"acknowledged":true}{"acknowledged":true,"shards_acknowledged":true,"index":"datahub_usage_event-000001"}
So, it seems good. Then, when I am trying to access datahub, I am getting the following error on GMS:
11:47:09.155 [qtp544724190-13] ERROR c.l.metadata.dao.search.ESSearchDAO - Search query failed:Elasticsearch exception [type=index_not_found_exception, reason=no such index [corpuserinfodocument]]
11:47:09.155 [qtp544724190-15] ERROR c.l.metadata.dao.search.ESSearchDAO - Search query failed:Elasticsearch exception [type=index_not_found_exception, reason=no such index [dataflowdocument]]
11:47:09.155 [qtp544724190-11] ERROR c.l.metadata.dao.search.ESSearchDAO - Search query failed:Elasticsearch exception [type=index_not_found_exception, reason=no such index [dashboarddocument]]
11:47:09.129 [qtp544724190-11] ERROR c.l.metadata.dao.search.ESSearchDAO - Search query failed:Elasticsearch exception [type=index_not_found_exception, reason=no such index [datajobdocument]]
11:47:09.129 [qtp544724190-12] ERROR c.l.metadata.dao.search.ESSearchDAO - Search query failed:Elasticsearch exception [type=index_not_found_exception, reason=no such index [chartdocument]]
11:47:09.129 [qtp544724190-9] ERROR c.l.metadata.dao.search.ESSearchDAO - Search query failed:Elasticsearch exception [type=index_not_found_exception, reason=no such index [datasetdocument]]
Does anyone know why is this happening? I am running the elasticsearch job, but then it looks like there are indexes still missing.. I am using the AWS opensearch and I am passing these env variables into the elasticsearch init job: DATAHUB_ANALYTICS_ENABLED: true and USE_AWS_ELASTICSEARCH: truemillions-elephant-81123
10/14/2021, 1:00 AMquaint-lighter-4354
10/14/2021, 2:46 PMCaused by:
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'entityClientFactory': Unsatisfied dependency expressed through field 'gmsPort'; nested exception is org.springframework.beans.TypeMismatchException: Failed to convert value of type 'java.lang.String' to required type 'int'; nested exception is java.lang.NumberFormatException: For input string: "<tcp://172.20.248.129:8080>"
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:643)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:116)
Does anyone know where it's coming from, as I am kind of stuck?
In my values.yaml(under global)
datahub:
gms:
port: "8080"
mae_consumer:
port: "9091"
appVersion: "1.0"
Thanks,
Yiannihandsome-football-66174
10/14/2021, 7:03 PMfierce-action-87313
10/15/2021, 7:17 AMbetter-orange-49102
10/19/2021, 11:45 PMagreeable-hamburger-38305
10/22/2021, 7:40 PMhelm install
and ran helm uninstall
this morning, but the pods are still running now (2 hours later). Anyone know what might be the reason? helm ls -a
shows nothinghandsome-football-66174
10/26/2021, 8:09 PMcalm-morning-92759
10/28/2021, 3:28 PMAUTH_OIDC_ENABLED=true
AUTH_OIDC_CLIENT_ID=your-client-id
AUTH_OIDC_CLIENT_SECRET=your-client-secret
AUTH_OIDC_DISCOVERY_URI=<https://accounts.google.com/.well-known/openid-configuration>
AUTH_OIDC_BASE_URL=your-datahub-url
AUTH_OIDC_SCOPE="openid profile email"
AUTH_OIDC_USER_NAME_CLAIM=email
AUTH_OIDC_USER_NAME_CLAIM_REGEX=([^@]+)
Anyway to change the deployment config manually seems not the right way. I am wondering what could be the best way to combine custom config with helm charts.
It seems that it is not possible to add the values in the values.yaml file.
Hope, that someone can share his experience...salmon-cricket-21860
10/29/2021, 8:22 AM- name: AUTH_OIDC_USER_NAME_CLAIM
value: "email"
- name: AUTH_OIDC_USER_NAME_CLAIM_REGEX
value: "([^@]+)"
- name: AUTH_OIDC_JIT_PROVISIONING_ENABLED
value: "false"
- name: AUTH_OIDC_PRE_PROVISIONING_REQUIRED
value: "true"
But failed to restrict non-existing users to login.
1. Removed urn rows from RDB (DELETE FROM metadata_aspect_v2 WHERE urn = 'urn:li:corpuser:test-user'
)
2. Deployed datahub with those env variables
3. Tried to login with <mailto:test-use@my.company.com|test-use@my.company.com>
Still possible to login, but extraction of group and other information didn't work. (included screenshot too)
Am I missing something? I want to disable login for non-registered users. test-user
in this case. (I am using google oauth. FYI)high-hospital-85984
10/29/2021, 4:26 PMpsql -U $POSTGRES_USERNAME -h $POSTGRES_HOST -p $POSTGRES_PORT -tc "SELECT 1 FROM pg_database WHERE datname = '${DATAHUB_DB_NAME}'"
where POSTGRES_USERNAME=datahub
and DATAHUB_DB_NAME=<something else>
I get an error psql: error: FATAL: database "datahub" does not exist
. Looks like psql tries to attach to a database named after the user, and this fails when we want to create a db with a non-standard name.
Or am I missing something?some-cricket-23089
11/01/2021, 10:34 AMbulky-intern-2942
11/01/2021, 3:16 PMred-pizza-28006
11/01/2021, 3:25 PMhandsome-football-66174
11/01/2021, 8:51 PMvictorious-dream-46349
11/02/2021, 7:03 AMchilly-analyst-561
11/02/2021, 6:14 PMagreeable-hamburger-38305
11/02/2021, 9:34 PMagreeable-hamburger-38305
11/02/2021, 9:36 PM0.2.1
is tied to Datahub version 0.8.16
. A little confused about the relationship between the helm chart version and Datahub versioncurved-jordan-15657
11/04/2021, 12:01 AMred-pizza-28006
11/04/2021, 6:17 PM