We were thinking about adding a failed topic for the MAE consumer, but just had some questions before looking into that. Found that the following files for the MCE consumer have failed topics: https://github.com/datahub-project/datahub/blob/master/metadata-jobs/mce-consumer/[…]/com/linkedin/metadata/kafka/MetadataChangeEventsProcessor.java https://github.com/datahub-project/datahub/blob/master/metadata-jobs/mce-consumer/[…]m/linkedin/metadata/kafka/MetadataChangeProposalsProcessor.java However, this file for the MAE consumer doesn't send to a failed topic in the event of an error: https://github.com/datahub-project/datahub/blob/master/metadata-jobs/mae-consumer/[…]ava/com/linkedin/metadata/kafka/MetadataChangeLogProcessor.java Is there a reason that the MetadataChangeLogProcessor doesn't have a failed topic like the MCE consumer or is that something that we can look into?

There's a similar failed topic for MCPs https://datahubproject.io/docs/next/what/mxe/#failed-metadata-change-proposal-fmcp

I think the general idea was that since the MCP was processed and committed to the primary datastore (SQL), that the MCL message cannot be skipped (or shouldn't rather) because at that point there would be an inconsistent state between SQL and Elasticsearch. Reprocessing of failed MCLs would lead to consistency issues with out of order messages.