https://datahubproject.io logo
Join Slack
Powered by
Hi Team we are upgrading Datahub using helm from 0...
# all-things-deployment
s
Hi Team we are upgrading Datahub using helm from 0.8.45 to 0.10.4, we updated helm-chart to 0.2.181..but we are getting below error for mae-consumer and mce-consumer. Could anyone please help us? MCE error org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'restliServletRegistration' defined in class path resource [com/linkedin/metadata/restli/RestliServletConfig.class]: Unsatisfied dependency expressed through method 'restliServletRegistration' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'restliHandlerServlet': Unsatisfied dependency expressed through field '_r2Servlet'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'rapServlet' defined in class path resource [com/linkedin/restli/server/RAPServletFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.r2.transport.http.server.RAPServlet]: Factory method 'rapServlet' threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'authorizerChainFactory': Unsatisfied dependency expressed through field 'dataHubAuthorizer'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dataHubAuthorizerFactory': Unsatisfied dependency expressed through field 'entityClient'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'javaEntityClientFactory': Unsatisfied dependency expressed through field '_entityService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dataHubKafkaEventProducerFactory': Unsatisfied dependency expressed through field 'kafkaProducer'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dataHubKafkaProducerFactory': Unsatisfied dependency expressed through field '_schemaRegistryConfig'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'awsGlueSchemaRegistryFactory': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'kafka.schemaRegistry.awsGlue.region' in value "${kafka.schemaRegistry.awsGlue.region}" MAE error: Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled. 2023-08-22 111532,623 [main] ERROR o.s.boot.SpringApplication - Application run failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dataHubKafkaEventProducerFactory': Unsatisfied dependency expressed through field 'kafkaProducer'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dataHubKafkaProducerFactory': Unsatisfied dependency expressed through field '_schemaRegistryConfig'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'awsGlueSchemaRegistryFactory': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'kafka.schemaRegistry.awsGlue.region' in value "${kafka.schemaRegistry.awsGlue.region}" Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dataHubKafkaProducerFactory': Unsatisfied dependency expressed through field '_schemaRegistryConfig'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'awsGlueSchemaRegistryFactory': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'kafka.schemaRegistry.awsGlue.region' in value "${kafka.schemaRegistry.awsGlue.region}" Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'awsGlueSchemaRegistryFactory': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'kafka.schemaRegistry.awsGlue.region' in value "${kafka.schemaRegistry.awsGlue.region}"
o
AWS Glue support got broken in v0.10.3 from the Internal Schema Registry changes, there was a recent bug fix for it that will be in the next release
s
Thanks Ryan it worked, Now I am facing issue with gms pod below are the errors INFO c.l.g.f.k.s.DUHESchemaRegistryFactory:29 - DataHub System Update Registry 2023-08-24 120841,767 [main] INFO c.l.g.f.k.KafkaEventConsumerFactory:116 - Event-based DUHE KafkaListenerContainerFactory built successfully. Consumer concurrency = 1 2023-08-24 120844,003 [main] ERROR i.c.k.s.client.rest.RestService:267 - Failed to send HTTP request to endpoint: http://schema-registry:port/subjects/MetadataChangeLog_Versioned_v1-value/versions java.net.UnknownHostException: schema-registry 2023-08-24 120913,589 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-24 120914,595 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry: Name does not resolve 2023-08-24 120914,595 [pool-13-thread-1] ERROR c.l.m.boot.OnBootApplicationListener:73 - Failed to bootstrap DataHub, OpenAPI servlet was not ready after 30 seconds 2023-08-24 120914.601INFOoejs.AbstractConnectorJettyShutdownThread Stopped ServerConnector@4387b79e{HTTP/1.1, (http/1.1)}{0.0.0.0:8080} 2023-08-24 120914.601INFOoejs.sessionJettyShutdownThread node0 Stopped scavenging 2023-08-24 120914.606INFOoejshC.ROOTJettyShutdownThread Destroying Spring FrameworkServlet 'schemaRegistryServlet' 2023-08-24 120914.613INFOoejshC.ROOTJettyShutdownThread Destroying Spring FrameworkServlet 'openapiServlet' 2023-08-24 120914.619INFOoejshC.ROOTJettyShutdownThread Destroying Spring FrameworkServlet 'healthServlet' 2023-08-24 120914.621INFOoejshC.ROOTJettyShutdownThread Destroying Spring FrameworkServlet 'authServlet' 2023-08-24 120914.621INFOoejshC.ROOTJettyShutdownThread Destroying Spring FrameworkServlet 'apiServlet' 2023-08-24 120914.626INFOoejshC.ROOTJettyShutdownThread Closing Spring root WebApplicationContext 2023-08-24 120914.979INFOoejsh.ContextHandlerJettyShutdownThread Stopped o.e.j.w.WebAppContext@6eda5c9{Open source GMS,/,null,STOPPED}{file:///datahub/datahub-gms/bin/war.war} 2023/08/24 120915 Command exited with error: exit status 1 Could anyone please suggest?
o
Yeah this is the second part of the bug in that thread: https://github.com/datahub-project/datahub/pull/8693 I have a fix slated for the next release
Since glue doesn't give you a schema registry url you can put in the configuration, that check fails. The way to work around it would be to temporarily deploy the prerequisite Confluent Schema registry or there may be another url you could pass in there that will return one of the accepted status codes
s
Thanks again Ryan,OnBootApplicationListener.java schema registry issues is resolved Now getting issues with Elasticsearch index 023-08-24 165746.370INFOoejshC.ROOTmain Initializing Spring DispatcherServlet 'authServlet' 2023-08-24 165746,370 [main] INFO o.s.web.servlet.DispatcherServlet:525 - Initializing Servlet 'authServlet' 2023-08-24 165746,429 [pool-16-thread-1] ERROR c.l.g.factory.telemetry.DailyReport:107 - Unable to send telemetry metrics, MixPanel API not initialized 2023-08-24 165752,014 [pool-10-thread-1] ERROR c.l.m.s.e.query.ESSearchDAO:98 - Search query failed org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed] at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [url:443], URI [/datahubpolicyindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request] {"error":{"root_cause":[{"type":"query_shard_exception","reason":"[simple_query_string] analyzer [query_word_delimited] not found","index_uuid":"","index":"datahubpolicyindex_v2"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"datahubpolicyindex_v2","node":"68p-15EqQDKradMUiqt-9g","reason":{"type":"query_shard_exception","reason":"[simple_query_string] analyzer [query_word_delimited] not found","index_uuid":"","index":"datahubpolicyindex_v2"}}]},"status":400} at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632) ... 17 common frames omitted 2023-08-24 165752,016 [pool-10-thread-1] ERROR c.d.authorization.DataHubAuthorizer:230 - Failed to retrieve policy urns! Skipping updating policy cache until next refresh. start: 0, count: 30 com.datahub.util.exception.ESQueryException: Search query failed: at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.executeAndExtract(ESSearchDAO.java:99) at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.search(ESSearchDAO.java:202) at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.search(ElasticSearchService.java:123) at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.search(ElasticSearchService.java:113)
subchart pods running fine except for gms...Please assist me.
o
Is this the full log? It just gets to here and GMS does not start up? Sometimes there are issues with updating the policy cache while GMS starts up so it's unclear if this is blocking. How are you running ElasticSearch? Docker?
s
running using helm chart, we have elasticsearch on AWS
Hi @orange-night-91387 I couldn't share full logs 2023-08-28 172915,879 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172915,879 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172916,880 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172916,881 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172917,638 [pool-10-thread-1] ERROR c.l.m.s.e.query.ESSearchDAO:98 - Search query failed org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahubpolicyindex_v2]] at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911) at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1888) at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1645) at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1602) at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1572) at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1088) at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.executeAndExtract(ESSearchDAO.java:92) at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.search(ESSearchDAO.java:202) at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.search(ElasticSearchService.java:123) at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.search(ElasticSearchService.java:113) at com.linkedin.metadata.client.JavaEntityClient.search(JavaEntityClient.java:304) at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:51) at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:43) at com.datahub.authorization.DataHubAuthorizer$PolicyRefreshRunnable.run(DataHubAuthorizer.java:223) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [https://abc.us-east-1.es.amazonaws.com:443], URI [/datahubpolicyindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found] {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [datahubpolicyindex_v2]","resource.type":"index_or_alias","resource.id":"datahubpolicyindex_v2","index_uuid":"na","index":"datahubpolicyindex_v2"}],"type":"index_not_found_exception","reason":"no such index [datahubpolicyindex_v2]","resource.type":"index_or_alias","resource.id":"datahubpolicyindex_v2","index_uuid":"na","index":"datahubpolicyindex_v2"},"status":404} at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632) ... 17 common frames omitted 2023-08-28 172917,639 [pool-10-thread-1] ERROR c.d.authorization.DataHubAuthorizer:230 - Failed to retrieve policy urns! Skipping updating policy cache until next refresh. start: 0, count: 30 com.datahub.util.exception.ESQueryException: Search query failed: at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.executeAndExtract(ESSearchDAO.java:99) at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.search(ESSearchDAO.java:202) at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.search(ElasticSearchService.java:123) at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.search(ElasticSearchService.java:113) at
com.linkedin.metadata.client.JavaEntityClient.search(JavaEntityClient.java:304) at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:51) at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:43) at com.datahub.authorization.DataHubAuthorizer$PolicyRefreshRunnable.run(DataHubAuthorizer.java:223) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahubpolicyindex_v2]] at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911) at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1888) at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1645) at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1602) at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1572) at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1088) at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.executeAndExtract(ESSearchDAO.java:92) ... 13 common frames omitted Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [https://abc.us-east-1.es.amazonaws.com:443], URI [/datahubpolicyindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found] {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [datahubpolicyindex_v2]","resource.type":"index_or_alias","resource.id":"datahubpolicyindex_v2","index_uuid":"na","index":"datahubpolicyindex_v2"}],"type":"index_not_found_exception","reason":"no such index [datahubpolicyindex_v2]","resource.type":"index_or_alias","resource.id":"datahubpolicyindex_v2","index_uuid":"na","index":"datahubpolicyindex_v2"},"status":404} at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632) ... 17 common frames omitted 2023-08-28 172917,882 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172917,882 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172918,883 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172918,884 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172919,885 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172919,885 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172920,891 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry: Name does not resolve 2023-08-28 172920,891 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172921,892 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172921,892 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172922,893 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172922,893 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172923,894 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172923,894 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172924,895 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172924,896 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172925,896 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172925,897 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172926,898 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172926,898 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172927,899 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172927,899 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172928,900 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172928,901 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172929,902 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172929,902 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172930,908 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry: Name does not resolve 2023-08-28 172930,909 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172931,910 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172931,910 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172932,911 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172932,911 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172933,912 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172933,912 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172934,913 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172934,914 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172935,914 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172935,915 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172936,916 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172936,916 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172937,917 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172937,917 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172938,918 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172938,919 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172939,920 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry 2023-08-28 172939,920 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:60 - Sleeping for 1 second 2023-08-28 172940,927 [pool-13-thread-1] INFO c.l.m.boot.OnBootApplicationListener:68 - Failed to connect to open servlet: schema-registry: Name does not resolve 2023-08-28 172940,928 [pool-13-thread-1] ERROR c.l.m.boot.OnBootApplicationListener:73 - Failed to bootstrap DataHub, OpenAPI servlet was not ready after 30 seconds 2023-08-28 172940.942INFOoejs.AbstractConnectorJettyShutdownThread Stopped ServerConnector@4387b79e{HTTP/1.1, (http/1.1)}{0.0.0.0:8080} 2023-08-28 172940.943INFOoejs.sessionJettyShutdownThread node0 Stopped scavenging 2023-08-28 172940.945INFOoejshC.ROOTJettyShutdownThread Destroying Spring FrameworkServlet 'schemaRegistryServlet' 2023-08-28 172940.948INFOoejshC.ROOTJettyShutdownThread Destroying Spring FrameworkServlet 'openapiServlet' 2023-08-28 172940.956INFOoejshC.ROOTJettyShutdownThread Destroying Spring FrameworkServlet 'healthServlet' 2023-08-28 172940.957INFOoejshC.ROOTJettyShutdownThread Destroying Spring FrameworkServlet 'authServlet' 2023-08-28 172940.958INFOoejshC.ROOTJettyShutdownThread Destroying Spring FrameworkServlet 'apiServlet' 2023-08-28 172940.962INFOoejshC.ROOTJettyShutdownThread Closing Spring root WebApplicationContext 2023-08-28 172941.327INFOoejsh.ContextHandlerJettyShutdownThread Stopped o.e.j.w.WebAppContext@6eda5c9{Open source GMS,/,null,STOPPED}{file:///datahub/datahub-gms/bin/war.war} 2023/08/28 172941 Command exited with error: exit status 1
a
So seems like there are two things going on there: 1. Your datahubpolicy index doesn't exist, which indicates there may have been some issues with the ElasticSearch initialization steps 2. It's still running into the error I mentioned above, you have an invalid schema registry url specified so the OnBootApplicationListener kills GMS after a bit. This will get fixed for Glue based deployments in the next release
s
I have added changes from your PR to my repository in OnBootApplicationListener, but still same error
a
And you're set to run with Glue rather than Internal?
s
Is any other files also required changes? As we have elsaticsearch, kafka, rds on AWS
7 Views
Open in Slack
PreviousNext
Powered by