Hi all, I'm having an issue running a Flink job on version 1.15.4. When we start this job, it is able to run successfully for a period of time (~17 hours) but after that, checkpoints start failing with the following error:

Failure Message: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: <https://192.168.0.1/api/v1/namespaces/data-acquisition-partner-file-ingestion/configmaps/job-name-d247bf413b1a1539012363d9bad3e2c1-config-map>. Message: ConfigMap "job-name-d247bf413b1a1539012363d9bad3e2c1-config-map" is invalid: []: Too long: must have at most 1048576 bytes. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=[], message=Too long: must have at most 1048576 bytes, reason=FieldValueTooLong, additionalProperties={})], group=null, kind=ConfigMap, name=job-name-d247bf413b1a1539012363d9bad3e2c1-config-map, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=ConfigMap "partner-file-ingestion-d247bf413b1a1539012363d9bad3e2c1-config-map" is invalid: []: Too long: must have at most 1048576 bytes, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

The job checkpoints every minute, and each checkpoint typically takes < 10 seconds. We're not really sure what's going on here. After consulting with our data platform team, they recommended that we reach out to the community for a potential solution! Please let me know if any additional info is needed. Any help is appreciated! Thank you.