Hi Flink experts, I am facing this issue for quite some days as i am not able to submit/run SQL Queries using REST CLIENT OR FLINK SQL CLIENT via SQL gateway. Here's my Setup Cluster Set up via k8s operator to dpeloy Session Cluster, Service gateway and ingress files. I am able to run `` ./flink run exmaples/abc.jar from my local host , Job manager pod, and Service gateway pod`` however when i try to use Rest client/ Flink SQL to submit SQL queries via SQL GATEWAY . It gives following error. Not really sure how to solve this as i have tried to increase my configurations (rest.client.max-content-length, rest.server.max-content-length) as mentioned in the error everywhere but no luck . I would really appreciate if someone can give pointers to debug further as i have run out of idea.

Caused by: org.apache.flink.runtime.rest.util.RestClientException: 413 Request Entity Too Large. Try to raise [rest.client.max-content-length]
	at org.apache.flink.runtime.rest.RestClient$ClientHandler.channelRead0(RestClient.java:615) ~[flink-dist-1.17.0.jar:1.17.0]
	at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) ~[flink-dist-1.17.0.jar:1.17.0]
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[flink-dist-1.17.0.jar:1.17.0]
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[flink-dist-1.17.0.jar:1.17.0]
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[flink-dist-1.17.0.jar:1.17.0]
	at org.apache.flink.shaded.netty4.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[flink-dist-1.17.0.jar:1.17.0]
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[flink-dist-1.17.0.jar:1.17.0]
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[flink-dist-1.17.0.jar:1.17.0]
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[flink-dist-1.17.0.jar:1.17.0]

apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkDeployment
metadata:
  name: session-seven-deployment-only-example
spec:
  image: flink:1.17
  flinkVersion: v1_17
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "10"
    rest.server.max-content-length: "1209715200"
    rest.client.max-content-length: "1209715200"
  serviceAccount: flink
  jobManager:
    resource:
      memory: "2048m"
      cpu: 1
  taskManager:
    resource:
      memory: "2048m"
      cpu: 1
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  annotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: k8-cps-dev
    <http://nginx.ingress.kubernetes.io/client-body-buffer-size|nginx.ingress.kubernetes.io/client-body-buffer-size>: 500M
  name: my-docker-app-ingress-3
spec:
  rules:
  - host: <http://flink-sql-gateway.usb.cloud.bank-dns.com|flink-sql-gateway.usb.cloud.bank-dns.com>
    http:
      paths:
      - backend:
          service:
            name: flink-seven-sql-gateway-svc
            port:
              number: 8086
        path: /
        pathType: Prefix

Gateway

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flink-seven-sql-gateway
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flink-seven-sql-gateway
  template:
    metadata:
      labels:
        app: flink-seven-sql-gateway
    spec:
      containers:
        - name: flink-seven-sql-gateway
          image: flink:1.17.0
          ports:
            - containerPort: 8086
          command: ["/bin/sh", "-c"]
          args:
            - "/opt/flink/bin/sql-gateway.sh start-foreground -Dsql-gateway.endpoint.rest.address=localhost -Dsql-gateway.endpoint.rest.port=8086"
          volumeMounts:
            - name: flink-conf
              mountPath: /opt/flink/conf
      volumes:
        - name: flink-conf
          configMap:
            name: flink-conf

Gateway svc

apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1> apiVersion: v1
kind: Service
metadata:
  name: flink-seven-sql-gateway-svc
spec:
  selector:
    app: flink-seven-sql-gateway
  ports:
    - name: flink-seven-sql-gateway
      port: 8086
      targetPort: 8086
  type: ClusterIP

Ingresses

kind: Ingress
metadata:
  annotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: k8-cps-dev
    <http://nginx.ingress.kubernetes.io/client-body-buffer-size|nginx.ingress.kubernetes.io/client-body-buffer-size>: 500M
  name: ingress-3
spec:
  rules:
  - host: <http://flink-sql-gateway.usb.cloud.bank-dns.com|flink-sql-gateway.usb.cloud.bank-dns.com>
    http:
      paths:
      - backend:
          service:
            name: flink-seven-sql-gateway-svc
            port:
              number: 8086
        path: /
        pathType: Prefix
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  annotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: k8-cps-dev
    <http://nginx.ingress.kubernetes.io/client-body-buffer-size|nginx.ingress.kubernetes.io/client-body-buffer-size>: 500M
  name: ingress-2
spec:
  rules:
  - host: <http://flink-prod-dns.usb.cloud.bank-dns.com|flink-prod-dns.usb.cloud.bank-dns.com>
    http:
      paths:
      - backend:
          service:
            name: session-seven-deployment-only-example-rest
            port:
              number: 8081
        path: /
        pathType: Prefix

Updated flink conf used as configmap

rest.address: <http://flink-prod-dns.usb.cloud.bank-dns.com|flink-prod-dns.usb.cloud.bank-dns.com>
rest.port: 80


jobmanager.bind-host: <http://flink-prod-dns.usb.cloud.bank-dns.com|flink-prod-dns.usb.cloud.bank-dns.com>

rest.server.max-content-length: 1409715200
rest.client.max-content-length: 1409715200