https://datahubproject.io logo
Join Slack
Powered by
HI team, need to help find logs for `mae-consumer-...
# getting-started
a
HI team, need to help find logs for 
mae-consumer-job
. I have run the 
mae-consumer-job
in two approaches 1: 
docker
2: 
java -jar mae-consumer-job.jar
. My goal is to see some logs by log4j either in console, or 
mae-consumer-job.log
file. for example.
Copy code
<https://github.com/linkedin/datahub/blob/master/metadata-jobs/mae-consumer/src/main/java/com/linkedin/metadata/kafka/MetadataAuditEventsProcessor.java#L96>
I changed 
log.debug
to 
<http://log.info|log.info>
but no matter what I do, I won't see any log printed. Did I do something wrong obviously?
I have used 
datahub_to_kafka
to ingest dataset, and validated 
MetadataAuditEvent_v4
topic does have the message ingested. I am also able to search and find ingested message from datahub-ui.
b
Is anything at all printing to the console?
a
Copy code
13:02:10.599 [main] WARN  c.l.r.t.h.client.HttpClientFactory - No scheduled executor is provided to HttpClientFactory, using it's own scheduled executor.
13:02:10.599 [main] WARN  c.l.r.t.h.client.HttpClientFactory - No callback executor is provided to HttpClientFactory, using it's own call back executor.
13:02:10.599 [main] WARN  c.l.r.t.h.client.HttpClientFactory - No Compression executor is provided to HttpClientFactory, using it's own compression executor.
13:02:10.599 [main] INFO  c.l.r.t.h.client.HttpClientFactory - The service 'null' has been assigned to the ChannelPoolManager with key 'noSpecifiedNamePrefix 1138266797 ' 
13:02:12.291 [main] INFO  o.s.s.c.ThreadPoolTaskExecutor - Initializing ExecutorService 'applicationTaskExecutor'
13:02:12.825 [main] INFO  o.s.b.a.e.web.EndpointLinksResolver - Exposing 3 endpoint(s) beneath base path '/actuator'
13:02:12.911 [main] INFO  o.e.j.s.h.ContextHandler.application - Initializing Spring DispatcherServlet 'dispatcherServlet'
13:02:12.911 [main] INFO  o.s.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet'
13:02:12.922 [main] INFO  o.s.web.servlet.DispatcherServlet - Completed initialization in 10 ms
13:02:12.964 [main] INFO  o.e.jetty.server.AbstractConnector - Started ServerConnector@540b0448{HTTP/1.1,[http/1.1]}{0.0.0.0:9091}
13:02:12.967 [main] INFO  o.s.b.w.e.jetty.JettyWebServer - Jetty started on port(s) 9091 (http/1.1) with context path '/'
13:02:12.973 [main] INFO  c.l.m.kafka.MaeConsumerApplication - Started MaeConsumerApplication in 7.708097227 seconds (JVM running for 8.697)
b
Hm. Interesting. We are getting some logs
a
for example? I have tried the docker mae-consumer-job also
b
No i mean you seem to be getting some ^^ are you sure that MAE consumer and MCE consumer are disabled in your GMS pod?
a
I started the other service with quickstart.sh. which is using this
Copy code
`docker-compose-without-neo4j.quickstart.yml
`then I looked at this compose file,
Copy code
- MAE_CONSUMER_ENABLED=true
    - MCE_CONSUMER_ENABLED=true
so I assume the 
mae
and 
mce
enabled. also 
MetadataAuditEvent_v4
message is published by 
gms
. If I saw there is a message in 
MetadataAuditEvent_v4 topic
, it means `mae-consumer`job enabled. right?
b
right so this means that the MAE and MCE consumers will be embedded inside the 
datahub-gms
container
So the logs you're looking for are likely inside 
datahub-gms
instead of the others
a
OK. I can dig a little bit
or what should I do if I want to use 
mae-consumer-job
and 
mce-consumer-job
explicitly?
b
You can set MAE_CONSUMER_ENABLED=false and MCE_CONSUMER_ENABLED=false on the 
datahub-gms
pod
Please let me know if this helps^^
a
actually, it doesn't work when I want to explicitly to use mae, mce. I did this, then run gms docker
Copy code
MAE_CONSUMER_ENABLED=false MCE_CONSUMER_ENABLED=false
then I run 
docker-compose-without-neo4j.quickstart.yml
, also I tried another ways (run mae jar and mce jar ), I didn't see a dataset can be ingested into datahub (I searched, and result won't show up in UI)
b
So i think the issue is that docker-compose-without-neo4j.quickstart.yml doesn't start up the standalone consumers
try docker-compose.consumers-without-neo4j.yml
a
I typed it wrong, I actually started 
docker-compose.consumers-without-neo4j.yml
and I also run the two jobs with IntelliJ
I am able to see some logs now with mae & mce embedded gms
thanks for the help to understand how those new docker-cmpose file works
Open in Slack
PreviousNext
Powered by