most-nightfall-36645
03/30/2023, 2:12 PM2023-03-30 13:40:08.175 ERROR 1 --- [ad | producer-1] c.l.m.dao.producer.KafkaHealthChecker : Failed to emit MCL for entity urn:li:dataset:(urn:li:dataPlatform:XXXXXXX,PROD)
org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.
I set the brokers max.message.bytes and replica.fetch.max.bytes to 1GB (much larger than intended final config) and set the emitter jobs SPRING_KAFKA_PRODUCER_PROPERTIES_MAX_REQUEST_SIZE environment variable to 1GB.
However I still can emit data. I checked the size of the meta data and it is around 1.4MB.
Am I missing something?astonishing-answer-96712
04/03/2023, 9:08 PMorange-night-91387
04/03/2023, 9:34 PMmost-nightfall-36645
04/18/2023, 9:03 AMJust to clarify, after updating these properties you still receive the same RecordTooLargeException?Yes we were still seeing the exception My solution was to manually adjust the cluster message sizes and then recreate the topics. Doing so meant that the topics inherited the message size from the broker. Which isnt a great solution.
most-nightfall-36645
04/18/2023, 9:03 AMastonishing-answer-96712
04/18/2023, 4:20 PMbrainy-tent-14503
04/22/2023, 1:35 AMmost-nightfall-36645
04/25/2023, 11:45 AMkafka-topics-worker.sh but it does not surface kafka-topics.sh create --config option needed to adjust topics configs.
This would require a PR.creamy-van-28626
09/15/2023, 6:21 PMhelpful-tent-87247
09/26/2023, 7:28 PM