Hi I am having some trouble emitting metadata to o...
# troubleshoot
m
Hi I am having some trouble emitting metadata to our datahub instance. When I try to restore indices to our opensearch cluster the emitter jobs logs contain the following error:
Copy code
2023-03-30 13:40:08.175 ERROR 1 --- [ad | producer-1] c.l.m.dao.producer.KafkaHealthChecker    : Failed to emit MCL for entity urn:li:dataset:(urn:li:dataPlatform:XXXXXXX,PROD)

org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.
I set the brokers
max.message.bytes
and
replica.fetch.max.bytes
to 1GB (much larger than intended final config) and set the emitter jobs
SPRING_KAFKA_PRODUCER_PROPERTIES_MAX_REQUEST_SIZE
environment variable to 1GB. However I still can emit data. I checked the size of the meta data and it is around 1.4MB. Am I missing something?
a
Hi, what version are you on, this might be a bug on our end- cc @orange-night-91387
o
Just to clarify, after updating these properties you still receive the same RecordTooLargeException? Or what do you see now that you've changed the above settings? These are the settings we have also changed to fix this same issue and should work.
m
Hi sorry for slow response. We are using v0.9.6.1.
Just to clarify, after updating these properties you still receive the same RecordTooLargeException?
Yes we were still seeing the exception My solution was to manually adjust the cluster message sizes and then recreate the topics. Doing so meant that the topics inherited the message size from the broker. Which isnt a great solution.
Is this fixed in a later version?
a
I believe it is, @brainy-tent-14503 might be able to speak to that
b
The helm chart values and environment variables are only controlling the DataHub producers and consumers. The topic’s settings in your kafka cluster for size are not managed by DataHub’s service and are typically only handling the initial creation. This means that often the need to increase the limit happens after the initial topic creation. Open to adding the ability to also set the other topic parameters optionally. Though as I mentioned we are generally hands off once the topic is initially created. For details on how the topic is created located here.
m
Thanks for getting back to me. However
kafka-topics-worker.sh
but it does not surface
kafka-topics.sh create
--config
option needed to adjust topics configs. This would require a PR.
c
I am facing the same issue , I tried to increase the limit to 5 MB IN PRODUCER CONSUMER SERVER AND BROKER PROPERTIES still the same the issue Any suggestions on this ?
plus1 2
h
yep, got the same problem here