Hii team! While upgarding images to 0.8.43 rediness and liveness probe are not working for mae-consumer and mce-consumer

Hello Vimpy, Could you paste stack trace exceptions in chat? It seems cut off in the screenshot

facing same one

Hello Dmytro how did you deploys DataHub?

official chart 0.2.88 (just released)

And the values.yaml? Could you share yours?

what particular thing you re looking for

Can you please set:

datahub_standalone_consumers_enabled: true

to

false

?

this will disable mce / mae will it work afterwards?

It will not disable MCE/MAE, they will be run within GMS pod rather than as isolated processes.

0.8.41 over 0.2.87 works as a charm with given one set to true

I’m able to reproduce this when consumers run standalone. Let me get back to you on this, in the mean time running the consumers in GMS should work as a workaround

+

Yeah i will also try this once they will get back

Are you running a fork? I don't think we use Solr anywhere

I want to ask one more thing like we are using datahub-rest so will mae and mce consumer job will impact on gms? Bcoz i upgraded all images to 0.8.43 at that time mae and mce consumer has failed

When you say

datahub-rest

do you mean the sink method for ingestion recipes?

Yeah

There should be no noticeable impact

So i can disable mae and mce

I don’t understand… you can disable mae and mce from running standalone, that will make them run in the GMS pod but they will still run

Okay okay

cc @dazzling-judge-80093

@limited-forest-73733 please, can you start a separate thread about this? This way it is hard to track. Please, when you start the new thread, can you let us know which source are you using and what is in your ingestion run summary?

Yeah sure i added separatw thread on ingestion but no one replied

I’m able to reproduce this when consumers run standalone. Let me get back to you on this, in the mean time running the consumers in GMS should work as a workaround

Hey @incalculable-ocean-74010, we encountered the same error when deploying MAE Consumer and MCE Consumer in standalone mode too with latest release 0.8.43. Let us know if you’ve some findings 🙏

afaiks chart:0.2.89 + 0.0.9 has been released - does it contains a fix?

Not for this. Release started prior to knowing this bug. Still working on a fix

Hi there any updates on issue?

Hey if i am upgrading all the images to 0.8.43 then i am not able to run recipe on UI even nothing is showing in ingestion on UI

@incalculable-ocean-74010 is chart v0.2.91 fixing this?

No, this is an application level bug. You can set the mae & mce consumer tags in the charts to v0.8.41 and it should work. The latest version of mae & mce consumers are known to be broken

or keep them within gms container?

standalone : false

?

Hello Dmytro, DataHub v0.8.44 + helm chart v0.2.93 does not fix this. I'm looking into it, if and when a fix is found it will be released as a DataHub release not helm charts. In the meantime the workaround is to not run the consumers in standalone mode.

Have you got a GitHub issue to reference for this please? I believe that we're affected by the same problem since our upgrade to 0.8.43 and it would be useful to be able to follow the progress of a fix. Many thanks.

I have a tentative fix that I’m trying to validate: https://github.com/datahub-project/datahub/pull/5827

Great. We may be able to help you check to see whether or not this fixes the issue. Thanks @incalculable-ocean-74010.

Hey folks, This should be fixed on the latest commit in datahub, use tag 20138a3 for all datahub components and it should work 🙂

Hi @incalculable-ocean-74010 @little-megabyte-1074can I suggest that the statement about the standalone consumers being broken should have been added to the release notes for v0.8.44 as well, and that users should wait for 0.8.45 to deploy them again? else, I would have thought that I could deploy the consumers again for current release. For better clarity to users.

Thanks for the call out, @better-orange-49102! I’ll work with Pedro to get this updated

Hey what's the plan of new release that fix consumer standalone issue

0.8.45 mae cannot be started 401 - unathorized. however

elasticsearch-setup-job

finished successfully are there different auth schemes used? moreover when revert back

datahub_standalone_consumers_enabled: false

- given error not reproduced

base64: unrecognized option: wrap                                                      
BusyBox v1.33.1 () multi-call binary.                                                  
                                                                                       
Usage: base64 [-d] [-w COL] [FILE]                                                     
                                                                                       
Base64 encode or decode FILE to standard output                                        
    -d    Decode data                                                                  
    -w COL    Wrap lines at COL (default 76, 0 disables)                               
2022/09/26 18:17:36 Waiting for: <http://logs-es-http.logging:9200>                      
2022/09/26 18:17:36 Waiting for: <tcp://kafka.cloud.pmcorp.loc:9092>                     
2022/09/26 18:17:36 Waiting for: <http://datahub-prerequisites-neo4j-community:7474>     
2022/09/26 18:17:36 Received 401 from <http://logs-es-http.logging:9200>. Sleeping 1s    
2022/09/26 18:17:36 Received 200 from <http://datahub-prerequisites-neo4j-community:7474>
2022/09/26 18:17:36 Connected to <tcp://kafka.cloud.pmcorp.loc:9092>                     
2022/09/26 18:17:37 Received 401 from <http://logs-es-http.logging:9200>. Sleeping 1s    
2022/09/26 18:17:38 Received 401 from <http://logs-es-http.logging:9200>. Sleeping 1s    
2022/09/26 18:17:39 Received 401 from <http://logs-es-http.logging:9200>. Sleeping 1s    
2022/09/26 18:17:40 Received 401 from <http://logs-es-http.logging:9200>. Sleeping 1s