https://datahubproject.io logo
Join Slack
Powered by
I build up datahub with source code and deployed o...
# ingestion
s
I build up datahub with source code and deployed on kubernetes, but mae-consumer component has below errors, any config i missed? Thanks in advanced.
Copy code
11:32:24.627 [main] WARN  o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'elasticsearchConnector' defined in class path resource [com/linkedin/metadata/kafka/elasticsearch/ElasticsearchConnectorFactory.class]: Unsatisfied dependency expressed through method 'createInstance' parameter 0; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.elasticsearch.client.RestHighLevelClient' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
Copy code
11:32:25.091 [main] DEBUG o.s.b.d.LoggingFailureAnalysisReporter - Application failed to start due to an exception
org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.elasticsearch.client.RestHighLevelClient' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.raiseNoMatchingBeanFound(DefaultListableBeanFactory.java:1695)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1253)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1207)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:885)
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:789)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:539)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:142)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:316)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
	at com.linkedin.metadata.kafka.MaeConsumerApplication.main(MaeConsumerApplication.java:15)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
11:32:25.091 [main] ERROR o.s.b.d.LoggingFailureAnalysisReporter -
o
This error indicates that your ElasticSearch configuration is invalid. There should be some more errors above this stacktrace indicating what exactly went wrong with connecting the the ElasticSearch instance 🙂
b
Hmm we can try to reproduce this - have you tried deploying without the standalone consumer?
s
@orange-night-91387 Sorry i find nothing related, would you please help me? I enable the spring boot debug and here is the output. Thank you.
dev.mae.exclude.log
@big-carpet-38439 I can’t ingest data without mae (or am i wrong in here? 🤔). Also i find mae create index if not exist, but it’s not working so i create indices manually. So wondering can we deploy without standalone consumer (mae)?
b
You can deploy without standalone containers for consumers yes!
Are you deploying via helm charts?
s
Sorry no, i deployed with skaffold + Kustomize (suggested by company IT). I just rewrite them. It works on last Aug, I pull all code to local and build and deploy, but two weeks ago i pull latest source code again and find it’s not working anymore.
You can deploy without standalone containers for consumers yes!
I am interested about it, do you have more information? thank you
b
To start the consumer in GMS, set the following environment variables when launching GMS
Copy code
MAE_CONSUMER_ENABLED=true
MCE_CONSUMER_ENABLED=true
And then just don't start the standalone containers...
I think something must be wrong with the latest code you've pulled and built -- we are stilling trying to repro on our end
Also -- we recently fixed a bug in this area. Do you mind re-pulling the latest code and we can go from here?
s
Will do and thank you very much !
@big-carpet-38439 Thank you so much is works after enable gms ingestion. BTW, the 
MCE_CONSUMER_ENABLED
should be 
MCL_CONSUMER_ENABLED
from source code.😀
b
oh man -- thanks!!
appreciate the clarification
2 Views
Open in Slack
PreviousNext
Powered by