I am trying to run in K8s with the latest updates ...
# all-things-deployments
sticky-television-18623
07/02/2021, 6:22 PMI am trying to run in K8s with the latest updates and I am seeing an odd behavior. I did helm uninstall and then helm install and now MCE is trying to process old messages that do not exist anymore. I am confused as to where the MCE is getting these old messages since the topics are empty.
m
mammoth-bear-12532
07/02/2021, 6:36 PM
@early-lamp-41924: thoughts?
e
early-lamp-41924
07/02/2021, 6:38 PMDid you uninstall prerequisites as well? Kafka uses a persistent store, and even if you reinstall it it will pick it up again
s
sticky-television-18623
07/02/2021, 6:40 PMWe are using Kafka outside of K8s. I lowered the message timeout and then increased it after a few minutes.
sticky-television-18623
07/02/2021, 6:41 PMThat should have cleared it. We also have monitoring on the topics and can see that there is no activity.
sticky-television-18623
07/02/2021, 6:51 PMIt looks like the offset was set back to zero but the messages still exist. Let me confirm.
sticky-television-18623
07/02/2021, 7:02 PMThat was it. Sorry for the bother.
m
mammoth-bear-12532
07/02/2021, 7:31 PM
That's great to hear. @sticky-television-18623: any thoughts on what a more improved debugging experience might look like?
s
sticky-television-18623
07/02/2021, 7:39 PMOff the top of mind I would say more detailed logging on exceptions. For example, right now I looking at an MCE error:
Copy code
19:25:30.988 [mce-consumer-job-client-0-C-1] ERROR c.l.m.k.MetadataChangeEventsProcessor - MCE Processor Error
java.lang.ClassCastException: com.linkedin.r2.RemoteInvocationException cannot be cast to com.linkedin.restli.client.RestLiResponseExceptionsticky-television-18623
07/02/2021, 7:41 PMThis does not tell me much about what the cause of the exception might be.
m
mammoth-bear-12532
07/02/2021, 9:32 PM
thanks! we'll take a look here
2 Views