https://datahubproject.io logo
Join Slack
Powered by
Hello, Can someone help me? I'm getting an error o...
# getting-started
g
Hello, Can someone help me? I'm getting an error on GMS server. Error details on thread.
Copy code
14:57:15.460 [gmsEbeanServiceConfig.heartBeat] ERROR i.e.datasource.pool.PooledConnection:311 - Error when fully closing connection [name[gmsEbeanServiceConfig1] slot[44] startTime[1655218506367] busySeconds[129] stackTrace[] stmt[select urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ?]]
java.sql.SQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
        at com.mysql.cj.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:1848)
        at com.mysql.cj.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:1705)
        at com.mysql.cj.jdbc.ConnectionImpl.close(ConnectionImpl.java:721)
        at io.ebean.datasource.pool.PooledConnection.closeConnectionFully(PooledConnection.java:308)
        at io.ebean.datasource.pool.FreeConnectionBuffer.trim(FreeConnectionBuffer.java:91)
        at io.ebean.datasource.pool.PooledConnectionQueue.trimInactiveConnections(PooledConnectionQueue.java:442)
        at io.ebean.datasource.pool.PooledConnectionQueue.trim(PooledConnectionQueue.java:422)
        at io.ebean.datasource.pool.ConnectionPool.trimIdleConnections(ConnectionPool.java:441)
        at io.ebean.datasource.pool.ConnectionPool.checkDataSource(ConnectionPool.java:459)
        at io.ebean.datasource.pool.ConnectionPool.access$000(ConnectionPool.java:43)
        at io.ebean.datasource.pool.ConnectionPool$HeartBeatRunnable.run(ConnectionPool.java:260)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
This could be problems with MySQL authentication?
b
it definitely could be. do you get this error when spinning up datahub or when do you get it? and have you always seen this error?
g
I used DataHub in a VM with a docker-compose file in the repository, and I've got no problem. But now I'm trying to execute it in a GKE. I will check the secrets. Maybe this is the problem.
b
gotcha, sounds good! come back to this thread and let us know if that fixed your issue or not
g
I found another possible problem:
Copy code
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event]]
@bulky-soccer-26729 do you have any idea what could be? The elasticsearch setup job executed perfectly
Elasticsearch setup job logs:
b
okay gotcha, so yeah the setup looks good. Do you see any error logs in the running elasticsearch pod? also did updating your secrets to ensure you're authenticated to mysql not pan out?
g
Some errors on the Elasticsearch logs:
Copy code
{"type": "deprecation.elasticsearch", "timestamp": "2022-06-14T14:16:17,626Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "datahub-dev-elasticsearch", "node.name": "datahub-dev-elasticsearch-master-0", "message": "[<http://node.ml|node.ml>] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "<http://node.ml|node.ml>", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-06-14T14:16:17,758Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "datahub-dev-elasticsearch", "node.name": "datahub-dev-elasticsearch-master-0", "message": "[node.data] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.data", "category": "settings" }
{"type": "server", "timestamp": "2022-06-14T14:16:17,845Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "datahub-dev-elasticsearch", "node.name": "datahub-dev-elasticsearch-master-0", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sdb)]], net usable_space [29.3gb], net total_space [29.3gb], types [ext4]" }
{"type": "server", "timestamp": "2022-06-14T14:16:17,846Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "datahub-dev-elasticsearch", "node.name": "datahub-dev-elasticsearch-master-0", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-06-14T14:16:17,955Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "datahub-dev-elasticsearch", "node.name": "datahub-dev-elasticsearch-master-0", "message": "[node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.master", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-06-14T14:16:17,959Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "datahub-dev-elasticsearch", "node.name": "datahub-dev-elasticsearch-master-0", "message": "[node.remote_cluster_client] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.remote_cluster_client", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-06-14T14:16:17,961Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "datahub-dev-elasticsearch", "node.name": "datahub-dev-elasticsearch-master-0", "message": "[node.ingest] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.ingest", "category": "settings" }
...
And
Copy code
{
  "type": "server",
  "timestamp": "2022-06-14T14:16:53,625Z",
  "level": "ERROR",
  "component": "o.e.i.g.DatabaseNodeService",
  "cluster.name": "datahub-dev-elasticsearch",
  "node.name": "datahub-dev-elasticsearch-master-0",
  "message": "failed to download database [GeoLite2-Country.mmdb]",
  "stacktrace": [
    "org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];",
    "at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:927) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:761) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:397) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:485) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:283) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:99) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:145) ~[?:?]",
    "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:57) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.client.OriginSettingClient.doExecute(OriginSettingClient.java:43) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:542) ~[elasticsearch-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.ingest.geoip.DatabaseNodeService.lambda$retrieveDatabase$11(DatabaseNodeService.java:367) [ingest-geoip-7.16.2.jar:7.16.2]",
    "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.2.jar:7.16.2]",
    "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
    "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
    "at java.lang.Thread.run(Thread.java:833) [?:?]"
  ]
}
And
Copy code
{
  "type": "deprecation.elasticsearch",
  "timestamp": "2022-06-14T14:18:19,549Z",
  "level": "CRITICAL",
  "component": "o.e.d.a.s.IndicesOptions",
  "cluster.name": "datahub-dev-elasticsearch",
  "node.name": "datahub-dev-elasticsearch-master-0",
  "message": "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices.",
  "key": "ignore_throttled_param",
  "category": "api",
  "cluster.uuid": "NO_NYrgnRhmsL8i9Db43ww",
  "node.id": "upK5gtjSSnS1SKy7PnDUZw"
}
And after updating the secrets, I still have the following error:
Copy code
15:35:09.206 [Thread-68] WARN  org.elasticsearch.client.RestClient:65 - request [POST <http://datahub-dev-elasticsearch-master:9200/glossarynodeindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true>] returned 2 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See <https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html> to enable security."],[299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
15:35:09.769 [gmsEbeanServiceConfig.heartBeat] ERROR i.e.datasource.pool.PooledConnection:311 - Error when fully closing connection [name[gmsEbeanServiceConfig4] slot[49] startTime[1655220776937] busySeconds[132] stackTrace[] stmt[select urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ?]]
java.sql.SQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
        at com.mysql.cj.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:1848)
        at com.mysql.cj.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:1705)
        at com.mysql.cj.jdbc.ConnectionImpl.close(ConnectionImpl.java:721)
        at io.ebean.datasource.pool.PooledConnection.closeConnectionFully(PooledConnection.java:308)
        at io.ebean.datasource.pool.FreeConnectionBuffer.trim(FreeConnectionBuffer.java:91)
        at io.ebean.datasource.pool.PooledConnectionQueue.trimInactiveConnections(PooledConnectionQueue.java:442)
        at io.ebean.datasource.pool.PooledConnectionQueue.trim(PooledConnectionQueue.java:422)
        at io.ebean.datasource.pool.ConnectionPool.trimIdleConnections(ConnectionPool.java:441)
        at io.ebean.datasource.pool.ConnectionPool.checkDataSource(ConnectionPool.java:459)
        at io.ebean.datasource.pool.ConnectionPool.access$000(ConnectionPool.java:43)
        at io.ebean.datasource.pool.ConnectionPool$HeartBeatRunnable.run(ConnectionPool.java:260)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
@big-carpet-38439 Can you help here?
Another error:
Copy code
18:04:12.440 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:93 - Skipping ingestion of editable policy with urn urn:li:dataHubPolicy:7
18:04:12.456 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:93 - Skipping ingestion of editable policy with urn urn:li:dataHubPolicy:view-entity-page-all
18:04:12.463 [main] INFO  c.l.m.boot.steps.IngestPoliciesStep:93 - Skipping ingestion of editable policy with urn urn:li:dataHubPolicy:view-dataset-sensitive
18:04:12.541 [pool-13-thread-1] ERROR c.l.d.g.a.service.AnalyticsService:264 - Search query failed: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event]]
18:04:12.545 [pool-13-thread-1] ERROR o.s.s.s.TaskUtils$LoggingErrorHandler:95 - Unexpected error occurred in scheduled task
java.lang.RuntimeException: Search query failed:
        at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:265)
        at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.getHighlights(AnalyticsService.java:236)
        at com.linkedin.gms.factory.telemetry.DailyReport.dailyReport(DailyReport.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event]]
        at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
        at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892)
        at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626)
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
        at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1069)
        at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:260)
        ... 15 common frames omitted
        Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [<http://datahub-dev-elasticsearch-master:9200>], URI [/datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found]
Warnings: [Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See <https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html> to enable security., [ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices.]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [datahub_usage_event]","resource.type":"index_or_alias","resource.id":"datahub_usage_event","index_uuid":"_na_","index":"datahub_usage_event"}],"type":"index_not_found_exception","reason":"no such index [datahub_usage_event]","resource.type":"index_or_alias","resource.id":"datahub_usage_event","index_uuid":"_na_","index":"datahub_usage_event"},"status":404}
                at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
                at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
                at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
                at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
                ... 19 common frames omitted
Im using:
Copy code
dependencies:
  # This chart deploys a
  - name: elasticsearch
    version: 7.16.2
    repository: <https://helm.elastic.co>
    condition: elasticsearch.enabled
With this config:
Copy code
elasticsearch:
  enabled: true
  replicas: 2
  minimumMasterNodes: 1
  clusterName: "datahub-dev-elasticsearch"
  resources:
    requests:
      cpu: "500m"
      memory: "1024M"
b
Hey Patrick, sorry for some delays here. Just to try and narrow this down a bit, can you tell me the main issue that you're seeing and what the consequences of it are? and does this happen when spinning up GMS or is it on ingestion time or something?
g
When I log onto the platform using the Google OIDC format my user is created correctly but it is not apparent when I search the user list. Also, when I create a group or domain, it is not returned when I run the search.
Creating a glossary
But:
When i search for the GMS logs:
Copy code
18:12:29.878 [gmsEbeanServiceConfig.heartBeat] ERROR i.e.datasource.pool.PooledConnection:311 - Error when fully closing connection [name[gmsEbeanServiceConfig3] slot[44] startTime[1655230217251] busySeconds[132] stackTrace[] stmt[select urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ?]]
java.sql.SQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
        at com.mysql.cj.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:1848)
        at com.mysql.cj.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:1705)
        at com.mysql.cj.jdbc.ConnectionImpl.close(ConnectionImpl.java:721)
        at io.ebean.datasource.pool.PooledConnection.closeConnectionFully(PooledConnection.java:308)
        at io.ebean.datasource.pool.FreeConnectionBuffer.trim(FreeConnectionBuffer.java:91)
        at io.ebean.datasource.pool.PooledConnectionQueue.trimInactiveConnections(PooledConnectionQueue.java:442)
        at io.ebean.datasource.pool.PooledConnectionQueue.trim(PooledConnectionQueue.java:422)
        at io.ebean.datasource.pool.ConnectionPool.trimIdleConnections(ConnectionPool.java:441)
        at io.ebean.datasource.pool.ConnectionPool.checkDataSource(ConnectionPool.java:459)
        at io.ebean.datasource.pool.ConnectionPool.access$000(ConnectionPool.java:43)
        at io.ebean.datasource.pool.ConnectionPool$HeartBeatRunnable.run(ConnectionPool.java:260)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
Services are running:
I ran the restore indices job:
Copy code
Reading rows 106000 through 107000 from the aspects table.
Failed to find aspect with name clientId associated with entity named telemetry
Successfully sent MAEs for 106537 rows
Number of MAEs sent 106537 does not equal the number of input rows 106538...
Failed Step 3/3: SendMAEStep. Failed after 0 retries.
Exiting upgrade RestoreIndices with failure.
Upgrade RestoreIndices completed with result FAILED. Exiting...
2022-06-14 18:23:20.692  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutdown requested
2022-06-14 18:23:20.692  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutting down
2022-06-14 18:23:20.709  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:23:20.715  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:23:20.716  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:23:20.781  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:23:20.782  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:23:20.782  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:23:20.784  INFO 1 --- [ Event Loop-3-1] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:23:20.789  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutdown requested
2022-06-14 18:23:20.789  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutting down
2022-06-14 18:23:20.789  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:23:20.789  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:23:20.789  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:23:20.790  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:23:20.790  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:23:20.790  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:23:20.801  INFO 1 --- [ Event Loop-3-1] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:23:20.802  INFO 1 --- [ Event Loop-3-2] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:23:20.804  INFO 1 --- [ Event Loop-3-2] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:23:21.070  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutdown requested
2022-06-14 18:23:21.071  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutting down
2022-06-14 18:23:21.072  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:23:21.072  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:23:21.072  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:23:21.074  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:23:21.074  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:23:21.074  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:23:21.075  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutdown requested
2022-06-14 18:23:21.075  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutting down
2022-06-14 18:23:21.075  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:23:21.075  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:23:21.075  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:23:21.075  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:23:21.075  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:23:21.075  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:23:21.077  INFO 1 --- [ionShutdownHook] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2022-06-14 18:23:21.091  INFO 1 --- [ Event Loop-1-1] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:23:21.092  INFO 1 --- [ Event Loop-1-1] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:23:21.092  INFO 1 --- [ Event Loop-1-2] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:23:21.093  INFO 1 --- [ Event Loop-1-2] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
But:
b
okay yeah so it looks like that job failed. I'm seeing someone else with the same issue here: https://datahubspace.slack.com/archives/C029A3M079U/p1654701868283659?thread_ts=1653676101.255129&amp;cid=C029A3M079U
it sounds like they pointed the upgrade job to 
head
and that solved their issue
g
I just ran with tag 'head' doing a 'helm upgrade' but I still have the same problem
No metadata...
b
so weird.. and did you see the same error about your restore indices job failing?
g
Now the job is finished with a success status
Copy code
Reading rows 106000 through 107000 from the aspects table.
Failed to find aspect with name clientId associated with entity named telemetry
Successfully sent MAEs for 106537 rows
Failed to send MAEs for 1 rows...
Completed Step 3/3: SendMAEStep successfully.
Success! Completed upgrade with id RestoreIndices successfully.
Upgrade RestoreIndices completed with result SUCCEEDED. Exiting...
2022-06-14 18:39:04.815  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutdown requested
2022-06-14 18:39:04.816  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutting down
2022-06-14 18:39:04.822  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:39:04.825  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:39:04.825  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:39:04.837  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:39:04.837  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:39:04.837  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:39:04.838  INFO 1 --- [ Event Loop-3-1] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:39:04.839  INFO 1 --- [ Event Loop-3-2] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:39:04.840  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutdown requested
2022-06-14 18:39:04.840  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutting down
2022-06-14 18:39:04.840  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:39:04.840  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:39:04.840  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:39:04.840  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:39:04.841  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:39:04.841  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:39:04.841  INFO 1 --- [ Event Loop-3-1] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:39:04.842  INFO 1 --- [ Event Loop-3-2] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:39:05.067  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutdown requested
2022-06-14 18:39:05.067  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutting down
2022-06-14 18:39:05.068  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:39:05.068  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:39:05.068  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:39:05.069  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:39:05.069  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:39:05.070  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:39:05.070  INFO 1 --- [ Event Loop-1-1] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:39:05.071  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutdown requested
2022-06-14 18:39:05.071  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutting down
2022-06-14 18:39:05.071  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:39:05.071  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:39:05.071  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:39:05.071  INFO 1 --- [ Event Loop-1-2] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:39:05.072  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-06-14 18:39:05.072  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown
2022-06-14 18:39:05.072  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shut down, closing all channels
2022-06-14 18:39:05.072  INFO 1 --- [ Event Loop-1-1] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:39:05.075  INFO 1 --- [ Event Loop-1-2] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutdown complete
2022-06-14 18:39:05.075  INFO 1 --- [ionShutdownHook] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
b
and you're still not seeing metadata even after a success?
g
But on GMS have a lot of this error:
Copy code
18:44:02.393 [gmsEbeanServiceConfig.heartBeat] ERROR i.e.datasource.pool.PooledConnection:311 - Error when fully closing connection [name[gmsEbeanServiceConfig3] slot[44] startTime[1655232111583] busySeconds[130] stackTrace[] stmt[select urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ? UNION ALL SELECT urn, aspect, version, metadata, createdOn, createdBy, createdFor FROM metadata_aspect_v2 WHERE urn = ? AND aspect = ? AND version = ?]]
java.sql.SQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown.
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
        at com.mysql.cj.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:1848)
        at com.mysql.cj.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:1705)
        at com.mysql.cj.jdbc.ConnectionImpl.close(ConnectionImpl.java:721)
        at io.ebean.datasource.pool.PooledConnection.closeConnectionFully(PooledConnection.java:308)
        at io.ebean.datasource.pool.FreeConnectionBuffer.trim(FreeConnectionBuffer.java:91)
        at io.ebean.datasource.pool.PooledConnectionQueue.trimInactiveConnections(PooledConnectionQueue.java:442)
        at io.ebean.datasource.pool.PooledConnectionQueue.trim(PooledConnectionQueue.java:422)
        at io.ebean.datasource.pool.ConnectionPool.trimIdleConnections(ConnectionPool.java:441)
        at io.ebean.datasource.pool.ConnectionPool.checkDataSource(ConnectionPool.java:459)
        at io.ebean.datasource.pool.ConnectionPool.access$000(ConnectionPool.java:43)
        at io.ebean.datasource.pool.ConnectionPool$HeartBeatRunnable.run(ConnectionPool.java:260)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
GMS and restore index job uses the same secret to access the MySQL database
b
gotcha so yeah it looks like we're having issues access mysql here. if they use the same secrets we can assume that's working.
hmmm
what's your mysql version?
b
Are you using Postgres or MySQL?
mysql i suppose
g
Yes. MySQL
I'm having this problem withing the analytics page:
Copy code
8:47:58.694 [Thread-33] ERROR c.datahub.graphql.GraphQLController:93 - Errors while executing graphQL query: "query getAnalyticsCharts {\n  getAnalyticsCharts {\n    groupId\n    title\n    charts {\n      ...analyticsChart\n      __typename\n    }\n    __typename\n  }\n}\n\nfragment analyticsChart on AnalyticsChart {\n  ... on TimeSeriesChart {\n    title\n    lines {\n      name\n      data {\n        x\n        y\n        __typename\n      }\n      __typename\n    }\n    dateRange {\n      start\n      end\n      __typename\n    }\n    interval\n    __typename\n  }\n  ... on BarChart {\n    title\n    bars {\n      name\n      segments {\n        label\n        value\n        __typename\n      }\n      __typename\n    }\n    __typename\n  }\n  ... on TableChart {\n    title\n    columns\n    rows {\n      values\n      cells {\n        value\n        linkParams {\n          searchParams {\n            types\n            query\n            filters {\n              field\n              value\n              __typename\n            }\n            __typename\n          }\n          entityProfileParams {\n            urn\n            type\n            __typename\n          }\n          __typename\n        }\n        __typename\n      }\n      __typename\n    }\n    __typename\n  }\n  __typename\n}\n", result: {errors=[{message=An unknown error occurred., locations=[{line=2, column=3}], path=[getAnalyticsCharts], extensions={code=500, type=SERVER_ERROR, classification=DataFetchingException}}], data=null}, errors: [DataHubGraphQLError{path=[getAnalyticsCharts], code=SERVER_ERROR, locations=[SourceLocation{line=2, column=3}]}]
18:47:58.728 [Thread-34] WARN  org.elasticsearch.client.RestClient:65 - request [POST <http://datahub-dev-elasticsearch-master:9200/datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true>] returned 2 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See <https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html> to enable security."],[299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
18:47:58.729 [Thread-34] ERROR c.l.d.g.a.service.AnalyticsService:264 - Search query failed: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event]]
18:47:58.730 [Thread-34] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.lang.RuntimeException: Search query failed:
        at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:265)
        at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.getHighlights(AnalyticsService.java:236)
        at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.getHighlights(GetHighlightsResolver.java:50)
        at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.get(GetHighlightsResolver.java:29)
        at com.linkedin.datahub.graphql.analytics.resolver.GetHighlightsResolver.get(GetHighlightsResolver.java:22)
        at graphql.execution.ExecutionStrategy.fetchField(ExecutionStrategy.java:270)
        at graphql.execution.ExecutionStrategy.resolveFieldWithInfo(ExecutionStrategy.java:203)
        at graphql.execution.AsyncExecutionStrategy.execute(AsyncExecutionStrategy.java:60)
        at graphql.execution.Execution.executeOperation(Execution.java:165)
        at graphql.execution.Execution.execute(Execution.java:104)
        at graphql.GraphQL.execute(GraphQL.java:557)
        at graphql.GraphQL.parseValidateAndExecute(GraphQL.java:482)
        at graphql.GraphQL.executeAsync(GraphQL.java:446)
        at graphql.GraphQL.execute(GraphQL.java:377)
        at com.linkedin.datahub.graphql.GraphQLEngine.execute(GraphQLEngine.java:88)
        at com.datahub.graphql.GraphQLController.lambda$postGraphQL$0(GraphQLController.java:89)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahub_usage_event]]
        at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
        at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892)
        at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626)
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
        at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1069)
        at com.linkedin.datahub.graphql.analytics.service.AnalyticsService.executeAndExtract(AnalyticsService.java:260)
        ... 17 common frames omitted
        Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [<http://datahub-dev-elasticsearch-master:9200>], URI [/datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found]
Warnings: [Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See <https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html> to enable security., [ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices.]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [datahub_usage_event]","resource.type":"index_or_alias","resource.id":"datahub_usage_event","index_uuid":"_na_","index":"datahub_usage_event"}],"type":"index_not_found_exception","reason":"no such index [datahub_usage_event]","resource.type":"index_or_alias","resource.id":"datahub_usage_event","index_uuid":"_na_","index":"datahub_usage_event"},"status":404}
                at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
                at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
                at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
                at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
                ... 21 common frames omitted
18:47:58.731 [Thread-34] ERROR c.datahub.graphql.GraphQLController:93 - Errors while executing graphQL query: "query getHighlights {\n  getHighlights {\n    value\n    title\n    body\n    __typename\n  }\n}\n", result: {errors=[{message=An unknown error occurred., locations=[{line=2, column=3}], path=[getHighlights], extensions={code=500, type=SERVER_ERROR, classification=DataFetchingException}}], data=null}, errors: [DataHubGraphQLError{path=[getHighlights], code=SERVER_ERROR, locations=[SourceLocation{line=2, column=3}]}]
Very strange as everything ran perfectly in docker-compose
b
i've actually seen this a couple times: datahub_usage_event not setup. Do you mind sending the logs from the successful elasticsearch-setup job? It should be creating that index for us cc @early-lamp-41924
g
Here:
Copy code
pbraz@hunb228:~/Workspace/HURB/Projects/git/dsc-datahub$ kubectl logs -n dev datahub-dev-elasticsearch-setup-job-q8t4j
2022/06/14 18:33:31 Waiting for: <http://datahub-dev-elasticsearch-master:9200>
2022/06/14 18:33:32 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:33:34 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:33:36 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:33:38 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:33:40 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:33:42 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:33:44 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:33:46 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:33:48 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:33:50 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:33:52 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:33:54 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:33:56 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:33:58 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:00 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:02 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:04 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:06 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:08 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:10 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:12 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:15 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:17 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:19 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:21 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:23 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:25 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:27 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:29 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:31 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:33 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:35 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:37 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:39 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:41 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:43 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:45 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:47 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:49 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:51 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:53 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:56 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:34:58 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:35:00 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:35:02 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:35:04 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:35:06 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:35:08 Problem with request: Get <http://datahub-dev-elasticsearch-master:9200>: dial tcp 10.53.0.110:9200: connect: connection refused. Sleeping 1s
2022/06/14 18:35:09 Received 200 from <http://datahub-dev-elasticsearch-master:9200>
2022/06/14 18:35:09 Command finished successfully.
The elasticsearch version is: 7.16.2
b
Interesting it seems to suggest that the index was indeed created... so very strange the error is appearing... still looking. On the MySQL side im super perplexed. If you just go try to create a group or domain you said that its just not showing anything at all?
g
Yes. I receive a message with successful creation but when I refresh the page, nothing appears.
b
Okay got it - do you have access to your SQL?
g
I don't know if it could be a problem, but I'm still using the VM with the docker-compose, and nothing I'm creating in the DataHub inside GKE appears on the VM version. OBS: They are using the same connection.
Using the same secret to access with dbeaver: Everything is ok but the metadata_index table is empty. It's normal?
b
Its not normal
Its failing to write to the db
My guess is your connection to the DB is bad
g
Let me see the mysql setup job and the user permissions
mysql setup job:
Copy code
-- create datahub database
CREATE DATABASE IF NOT EXISTS datahub CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
USE datahub;

-- create metadata aspect table
create table if not exists metadata_aspect_v2 (
  urn                           varchar(500) not null,
  aspect                        varchar(200) not null,
  version                       bigint(20) not null,
  metadata                      longtext not null,
  systemmetadata                longtext,
  createdon                     datetime(6) not null,
  createdby                     varchar(255) not null,
  createdfor                    varchar(255),
  constraint pk_metadata_aspect_v2 primary key (urn,aspect,version)
);

-- create default records for datahub user if not exists
CREATE TABLE temp_metadata_aspect_v2 LIKE metadata_aspect_v2;
INSERT INTO temp_metadata_aspect_v2 (urn, aspect, version, metadata, createdon, createdby) VALUES(
  'urn:li:corpuser:datahub',
  'corpUserInfo',
  0,
  '{"displayName":"Data Hub","active":true,"fullName":"Data Hub","email":"<mailto:datahub@linkedin.com|datahub@linkedin.com>"}',
  now(),
  'urn:li:corpuser:__datahub_system'
), (
  'urn:li:corpuser:datahub',
  'corpUserEditableInfo',
  0,
  '{"skills":[],"teams":[],"pictureLink":"<https://raw.githubusercontent.com/datahub-project/datahub/master/datahub-web-react/src/images/default_avatar.png>"}',
  now(),
  'urn:li:corpuser:__datahub_system'
);
-- only add default records if metadata_aspect is empty
INSERT INTO metadata_aspect_v2
SELECT * FROM temp_metadata_aspect_v2
WHERE NOT EXISTS (SELECT * from metadata_aspect_v2);
DROP TABLE temp_metadata_aspect_v2;

-- create metadata index table
CREATE TABLE IF NOT EXISTS metadata_index (
 `id` BIGINT NOT NULL AUTO_INCREMENT,
 `urn` VARCHAR(200) NOT NULL,
 `aspect` VARCHAR(150) NOT NULL,
 `path` VARCHAR(150) NOT NULL,
 `longVal` BIGINT,
 `stringVal` VARCHAR(200),
 `doubleVal` DOUBLE,
 CONSTRAINT id_pk PRIMARY KEY (id),
 INDEX longIndex (`urn`,`aspect`,`path`,`longVal`),
 INDEX stringIndex (`urn`,`aspect`,`path`,`stringVal`),
 INDEX doubleIndex (`urn`,`aspect`,`path`,`doubleVal`)
);
2022/06/14 18:33:24 Command finished successfully.
I'm using the 'datahub' user. Running this query:? SHOW GRANTS FOR 'datahub'; Returns:
Copy code
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, PROCESS, REFERENCES, INDEX, ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER, CREATE TABLESPACE ON *.* TO 'datahub'@'%' WITH GRANT OPTION
b
I think it might be network connectivity
Like the pods in GKS unable to talk to docker compose mysql
g
Hmmm, maybe. But this is a Cloud SQL instance on Google.
Both the DataHub in docker-compose and in GKE is acessing this managed instance.
I'll see if something is preventing GMS from accessing Cloud SQL. Maybe that's the problem.
But now I have a question. How the job to restore indices managed to read the MAE?
b
Yeah you’re right about that
But also
How did it find anything?
Didn’t you say there were 0 rows?
g
The job was able to perfectly read the entire aspect table. Here the logs:
Copy code
2022-06-14 18:37:22.053  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 5.5.1-ccs
2022-06-14 18:37:22.053  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: cb1873c1fdf5f5f9
2022-06-14 18:37:22.053  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1655231842048
2022-06-14 18:37:22.132  INFO 1 --- [           main] c.l.m.m.registry.ConfigEntityRegistry    : Loading bare config entity registry file at /datahub/datahub-gms/resources/entity-registry.yml
2022-06-14 18:37:22.675  INFO 1 --- [ad | producer-1] org.apache.kafka.clients.Metadata        : [Producer clientId=producer-1] Cluster ID: WV8aBxieT-STpq4kqkPebQ
2022-06-14 18:37:23.508  INFO 1 --- [           main] org.reflections.Reflections              : Reflections took 1292 ms to scan 45 urls, producing 817 keys and 7665 values 
2022-06-14 18:37:23.873  INFO 1 --- [           main] org.reflections.Reflections              : Reflections took 10 ms to scan 6 urls, producing 23 keys and 52 values 
2022-06-14 18:37:25.851  WARN 1 --- [           main] c.l.m.m.r.PluginEntityRegistryLoader     : /etc/datahub/plugins/models directory does not exist or is not a directory. Plugin scanning will be disabled.
2022-06-14 18:37:25.984  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : dataHubPolicyKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.985  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : dataHubPolicyInfo schema is compatible with previous schema due to 
2022-06-14 18:37:25.985  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : corpGroupInfo schema is compatible with previous schema due to 
2022-06-14 18:37:25.986  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : corpGroupKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.986  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : globalTags schema is compatible with previous schema due to 
2022-06-14 18:37:25.986  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : status schema is compatible with previous schema due to 
2022-06-14 18:37:25.986  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : mlFeatureTableKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.986  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : mlModelGroupKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.987  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : dataJobKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.987  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : ownership schema is compatible with previous schema due to 
2022-06-14 18:37:25.987  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : tagProperties schema is compatible with previous schema due to 
2022-06-14 18:37:25.987  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : tagKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.988  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : ownership schema is compatible with previous schema due to 
2022-06-14 18:37:25.988  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : glossaryTermInfo schema is compatible with previous schema due to 
2022-06-14 18:37:25.988  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : glossaryTermKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.988  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : mlPrimaryKeyKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.989  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : dashboardKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.989  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : dataPlatformKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.989  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : dataPlatformInfo schema is compatible with previous schema due to 
2022-06-14 18:37:25.989  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : dataHubRetentionConfig schema is compatible with previous schema due to 
2022-06-14 18:37:25.989  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : dataHubRetentionKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.989  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : corpUserKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.990  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : corpUserEditableInfo schema is compatible with previous schema due to 
2022-06-14 18:37:25.990  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : corpUserInfo schema is compatible with previous schema due to 
2022-06-14 18:37:25.990  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : corpUserStatus schema is compatible with previous schema due to 
2022-06-14 18:37:25.991  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : globalTags schema is compatible with previous schema due to 
2022-06-14 18:37:25.991  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : groupMembership schema is compatible with previous schema due to 
2022-06-14 18:37:25.991  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : status schema is compatible with previous schema due to 
2022-06-14 18:37:25.992  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : mlModelKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.992  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : dataFlowKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.992  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : glossaryNodeInfo schema is compatible with previous schema due to 
2022-06-14 18:37:25.992  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : glossaryNodeKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.993  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : ownership schema is compatible with previous schema due to 
2022-06-14 18:37:25.993  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : status schema is compatible with previous schema due to 
2022-06-14 18:37:25.993  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : mlFeatureKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.993  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : datasetKey schema is compatible with previous schema due to 
2022-06-14 18:37:25.993  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : viewProperties schema is compatible with previous schema due to 
2022-06-14 18:37:25.995  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : schemaMetadata schema is compatible with previous schema due to 
2022-06-14 18:37:25.995  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : status schema is compatible with previous schema due to 
2022-06-14 18:37:25.995  INFO 1 --- [           main] c.l.m.m.registry.MergedEntityRegistry    : chartKey schema is compatible with previous schema due to 
2022-06-14 18:37:26.221  WARN 1 --- [           main] c.l.r.t.http.client.HttpClientFactory    : No scheduled executor is provided to HttpClientFactory, using it's own scheduled executor.
2022-06-14 18:37:26.222  WARN 1 --- [           main] c.l.r.t.http.client.HttpClientFactory    : No callback executor is provided to HttpClientFactory, using it's own call back executor.
2022-06-14 18:37:26.222  WARN 1 --- [           main] c.l.r.t.http.client.HttpClientFactory    : No Compression executor is provided to HttpClientFactory, using it's own compression executor.
2022-06-14 18:37:26.232  INFO 1 --- [           main] c.l.r.t.http.client.HttpClientFactory    : The service 'null' has been assigned to the ChannelPoolManager with key 'noSpecifiedNamePrefix 1138266797 ', http.protocolVersion=HTTP_1_1, usePipelineV2=false, requestTimeout=10000ms, streamingTimeout=-1ms
2022-06-14 18:37:26.824  INFO 1 --- [           main] Driver                                   : Direct driver instance 179379830 created for server address datahub-dev-neo4j-community:7687
2022-06-14 18:37:28.219  WARN 1 --- [           main] c.l.r.t.http.client.HttpClientFactory    : No scheduled executor is provided to HttpClientFactory, using it's own scheduled executor.
2022-06-14 18:37:28.220  WARN 1 --- [           main] c.l.r.t.http.client.HttpClientFactory    : No callback executor is provided to HttpClientFactory, using it's own call back executor.
2022-06-14 18:37:28.220  WARN 1 --- [           main] c.l.r.t.http.client.HttpClientFactory    : No Compression executor is provided to HttpClientFactory, using it's own compression executor.
2022-06-14 18:37:28.220  INFO 1 --- [           main] c.l.r.t.http.client.HttpClientFactory    : The service 'null' has been assigned to the ChannelPoolManager with key 'noSpecifiedNamePrefix 1138266797 ', http.protocolVersion=HTTP_1_1, usePipelineV2=false, requestTimeout=10000ms, streamingTimeout=-1ms
2022-06-14 18:37:30.221  INFO 1 --- [           main] c.l.g.f.k.s.AwsGlueSchemaRegistryFactory : Creating AWS Glue registry
2022-06-14 18:37:30.243  INFO 1 --- [           main] c.l.g.f.k.SimpleKafkaConsumerFactory     : Simple KafkaListenerContainerFactory built successfully
2022-06-14 18:37:30.266  INFO 1 --- [           main] c.l.g.f.kafka.KafkaEventConsumerFactory  : Event-based KafkaListenerContainerFactory built successfully
2022-06-14 18:37:31.580  INFO 1 --- [           main] c.l.d.upgrade.UpgradeCliApplication      : Started UpgradeCliApplication in 15.042 seconds (JVM running for 16.044)
Starting upgrade with id RestoreIndices...
Cleanup has not been requested.
Skipping Step 1/3: ClearSearchServiceStep...
Cleanup has not been requested.
Skipping Step 2/3: ClearGraphServiceStep...
Executing Step 3/3: SendMAEStep...
Sending MAE from local DB...
Found 106538 latest aspects in aspects table
Reading rows 0 through 1000 from the aspects table.
@big-carpet-38439 Do you need more informations like the schema registry config creation?
Important: MAE and MCE consumers are disabled. Have any relation with this problem?
@big-carpet-38439 I created an ingress for the GMS server with the same host as the frontend but with a path /gms. I'm goting the error in the image below when I search for https://host/gms Do you think this could be the problem?
@big-carpet-38439 and @bulky-soccer-26729 After doing a lot of research and testing, I found out where the problem was, but I don't know exactly what solved it. But apparently, both the GMS and the MAE and MCE consumers then lack some necessary environment variables like 'DATAHUB_USAGE_EVENT_NAME' and 'KAFKA_TOPIC_NAME'. After correctly initializing the variables in the containers, everything worked correctly again. Except for an error I'm having in the MAE consumer and in the GMS. The GMS is getting an error executing the query to render the graphs on the analysis screen, and MAE cant build what is supposed to be the GMS host (I think). GMS logs example:
Copy code
23:50:46.687 [Thread-548] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.util.concurrent.CompletionException: java.lang.StackOverflowError
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.StackOverflowError: null
        at java.util.HashMap.hash(HashMap.java:339)
        at java.util.HashMap.computeIfAbsent(HashMap.java:1099)
        at io.ebeaninternal.api.BindParams.getParam(BindParams.java:174)
        at io.ebeaninternal.api.BindParams.getParameter(BindParams.java:248)
...
23:50:46.698 [Thread-511] ERROR c.datahub.graphql.GraphQLController:93 - Errors while executing graphQL query: "query getSearchResultsForMultiple($input: SearchAcrossEntitiesInput!) {\n  searchAcrossEntities(input: $input) {\n    ...searchResults\n    __typename\n  }\n}\n\nfragment searchResults on SearchResults {\n  start\n  count\n  total\n  searchResults {\n    entity {\n      ...searchResultFields\n      __typename\n    }\n    matchedFields {\n      name\n      value\n      __typename\n    }\n    insights {\n      text\n      icon\n      __typename\n    }\n    __typename\n  }\n  facets {\n    ...facetFields\n    __typename\n  }\n  __typename\n}\n\nfragment searchResultFields on Entity {\n  urn\n  type\n  ... on Dataset {\n    name\n    origin\n    uri\n    platform {\n      ...platformFields\n      __typename\n    }\n    dataPlatformInstance {\n      ...dataPlatformInstanceFi
And the Mae consumer error:
Copy code
2022/06/15 00:01:17 Waiting for: http://:
2022/06/15 00:01:17 Waiting for: <tcp://datahub-dev-kafka:9092>
2022/06/15 00:01:17 Waiting for: <http://datahub-dev-neo4j-community:7474>
2022/06/15 00:01:17 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/06/15 00:01:17 Connected to <tcp://datahub-dev-kafka:9092>
2022/06/15 00:01:17 Received 200 from <http://datahub-dev-neo4j-community:7474>
2022/06/15 00:01:18 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/06/15 00:01:19 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/06/15 00:01:20 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/06/15 00:01:21 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/06/15 00:01:22 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/06/15 00:01:23 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
b
I see - did you customize the Kafka topic names?
g
Not. But with this test, I declared the same value for all services that need this parameter to maintain consistency.
b
You should not need to declare any variables like this…. The defaults should all be consistent…
Cc @early-lamp-41924 to check me on this
g
What I did: I checked every .env file within this directory: https://github.com/datahub-project/datahub/tree/master/docker I found that some environtment variables weren't declared on the deployment.
e
Hmn. Yeah, there should be defaults set for all topic names, so you shouldn’t have to explicitly set them
g
@early-lamp-41924 @big-carpet-38439 What is the purpose of the environment variable 'PE_CONSUMER_ENABLED' ?
I'm working on a DataHub chart refactoring that is more compliant with other projects of my platform team and I'm setting the default values for the environment variables at deployment.yaml file. Would be interesting for me to know the meaning of each variable.
@big-carpet-38439 My 'metadata_index' table is still empty but now I can see and create entities on the UI. I don't think this is normal. What do you think?
b
@orange-night-91387 This is why we need a detailed doc of all environment vars^^^! Patrick, metadata_aspect_v2 should be populated, metadata_index is legacy table - it's okay if empty. PE_CONSUMER_ENABLED dictates whether the Platform Event consumer is enabled. Currently, there is no logic that is critical to operation in that Kafka consumer so its okay if its disabled
thank you 1
nod 1
o
Yep! We'll have that one soon 🙂
12 Views
Open in Slack
PreviousNext
Powered by