Hello Team, I have installed version 0.11.0, 0.12.0, in GKE of GCP with a custom values file, at the moment of activating the restore indices job, it is always generating a kafka conflict with the following error. “apache.kafka.common.errors.RecordTooLargeException” fails to restore the indexes correctly and the application does not load the data. I have used several slack tips but none has worked until adding the kafka environment variables, but it does not work.

By default Kafka has a default limit of 1MB per message in the topic. Increasing the message size limit on the kafka broker may fix the issue maybe this blog can help you resolve the issue https://www.conduktor.io/kafka/how-to-send-large-messages-in-apache-kafka/ if your deployment with Datahub Helm chart you set this param KAFKA_PRODUCER_MAX_REQUEST_SIZE value higher (like 2MB) other configuration you may want to try only in the case above Increasing the message size still not fix the issue

In GMS
    - name: SPRING_KAFKA_PRODUCER_PROPERTIES_MAX_REQUEST_SIZE
      value: "5242880"
    - name: SPRING_KAFKA_CONSUMER_PROPERTIES_MAX_PARTITION_FETCH_BYTES
      value: "5242880"
In Kafka
- name: KAFKA_CFG_MESSAGE_MAX_BYTES
          value: "8242800" -- any value larger than 5242880 should work
- name: KAFKA_CFG_SOCKET_REQUEST_MAX_BYTES
          value: "5242880"

good Day team I was doing the configuration inside the values file that we have for the Datahub installation but the failure persists, the version of Datahub that I am trying to install is version 0.11.0 with the tags of the Datahub Helm chart repo, prerequisites 0.1.4 and values 0.3.6, has anyone else had this happen to them?

How are you applying the configuration? Are you applying it as an extra env on the Restore Indices job template?

to the Job of restoring indexes I don’t make any modification, the configuration I am doing it inside the values and prerequisites file in the key Datahub-GMS and kakfa at the moment of the installation of Datahub inside GKE of GCP, in the version 0.11.0. @orange-night-91387

Modifying it in GMS would not make an impact on the Restore Indices job that is failing in the above screenshots. You will need to apply the configuration to the failing component.

Good Day I was adding the configuration inside the values file that we have inside the restoreIndices, but the failure still persists, this only occurs within the version 0.11.0 onwards, since we have the version 0.10.4 and we are working without any problem. @orange-night-91387 please can you help me, thanks

Did you also change the broker side configuration?: https://www.baeldung.com/java-kafka-send-large-message#kafka-topic-configuration

@orange-night-91387 Is there a way to change it in the Datahub Helm values? please

The Kafka broker is not a part of DataHub helm, typically it would be a managed service or if you’re doing an initial POC then it may be the prerequisites which uses Confluent charts.