I've got my Flink Cluster (1.20.0) running in Kube...
# troubleshooting
c
I've got my Flink Cluster (1.20.0) running in Kubernetes now. So far it's it's just 1 job manager and 3 task workers. I would like to add SQL Gatweay, but I'm not finding much info on parameters or YAML to use. I could "guess" but need a simple starting point. Does anyone have a lead on info to setup a SQL Gateway on Kubernetes?
y
I'm doing the same
I didn't install the gateway, just used sql-client.
c
Ahh. Thanks!
For anyone out there also trying to do this.. I've made progress. It seems fairly straightforward. There is a shell script in the bin folder to startup the gateway sql-gateway.sh. I'm just adding that as an arg to the pod and it fires up just like the JobManager and TaskManager. There is a required value to set an Address. I am setting that through command args. I've attached my yaml below. The only part I am not having success with is getting the gateway to route to the job manager for execution. I'll keep grinding on it today in hopes of figuring it out. --- apiVersion: apps/v1 kind: Deployment metadata: name: flink-sql-gateway namespace: flink spec: replicas: 1 selector: matchLabels: app: flink-sql-gateway template: metadata: labels: app: flink-sql-gateway spec: containers: - name: sql-gateway image: ghcr.io/cloud-focus-tech/flink-s3:latest securityContext: runAsUser: 1000 runAsGroup: 1000 command: ["/bin/sh", "-c"] args: - "/opt/flink/bin/sql-gateway.sh start-foreground -Dsql-gateway.endpoint.rest.address=flink-jobmanager.flink.svc.cluster.local" ports: - containerPort: 8083 env: - name: FLINK_PROPERTIES value: | jobmanager.rpc.address: flink-jobmanager.flink.svc.cluster.local sql-gateway.endpoint.port: 8083 - name: LOG4J_CONFIGURATION_FILE value: "/opt/flink/conf/log4j2.xml" - name: JOB_MANAGER_RPC_ADDRESS value: "flink-jobmanager" volumeMounts: - name: flink-config-volume mountPath: /opt/flink/conf/config.yaml subPath: config.yaml - name: flink-logs mountPath: /opt/flink/log readOnly: false volumes: - name: flink-config-volume configMap: name: flink-gateway-config - name: flink-logs emptyDir: {} --- apiVersion: v1 kind: Service metadata: name: flink-sql-gateway namespace: flink spec: type: LoadBalancer selector: app: flink-sql-gateway ports: - protocol: TCP port: 8083 targetPort: 8083
Ahh. I see.. posting here to assist others.... this is the error I'm finding in my sql-gateway logs -> 2025-02-23 143349,339 WARN org.apache.flink.client.program.rest.RestClusterClient [] - Attempt to submit job 'SELECT 1' (635f69bcbaa31b6aa8f4e6836c0e16ef) to 'http://localhost:8081' has failed. localhost is wrong..it needs to go to my jobmanager pod and not localhost. The "A-Ha" is the path org.apache.flink.client etc.. This leads me to the code in GIT where I can see what it's looking for. Replace the dots in the error message with slashes and you can find it. https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/program/rest/RestClusterClient.java
y
Thanks for the deployment, I needed that 🙂 I'm wondering how will you productionize the SQL deployments? I used sql-client, which I figured is good enough to deploy SQLs. Have you considered DBT?
c
Hmm.. I'm actually not familiar with DBT other than hearing the name a few times. Can you share more? What does it do? I got hung up this weekend trying to get the SQL Gateway to talk to the Flink cluster. I can run SQL Client right on the Gateway pod and it works, but trying to issue a command from SQL CLient locally to SQL Gateway is a struggle at the moment. It seems like it tries to hit flink cluster on "localhost" (which is the gateway pod) which doesn't have a flink cluster. I was able to get it to attempt to hit the cluster, but it then failed to locate the REST endpoint. Yet the rest endpoint is up and running. So I just need to figure out how the Gateway locates the endpoint.
➕ 1
y
You don't need SQL gateway with SQL client. I will probably bypass it. DBT is the T in the ELT. Basically, helps managing transformation.
I also though if having a k8s job that uses sql-client to load an sql file via a config map
For now, I'm in jar depency hell 🙂
c
Well that sounds unpleasant. I got hung up on jars last week until I watched Rmoffits video on troubleshooting. Yeah I can run SQL Client locally for sure. What I would like to be able to do is deploy a SQL job to job manager from outside the cluster. I'd like to be able to leverage DevOps for SQL Jobs. The Flink repo has an example that seems to be close to doing this.
y
I started watching his video, just didn't finish... I though I knew better 😛
👍 1
All Jars resolved, now, I'm here
java.lang.NullPointerException
I guess more fun to come
🎉 1
just for an fyi, I received much help from @Pedro Mázala The final jar list to get Kafka and Clickhouse connectors going :
Copy code
RUN curl -L -o /opt/flink/lib/flink-connector-kafka-1.20.jar https://dp-assets-public.s3.us-east-1.amazonaws.com/flink-connector-kafka-3.4.0.jar
RUN curl -L -o /opt/flink/lib/kafka-clients-3.4.0.jar https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/3.4.0/kafka-clients-3.4.0.jar
RUN curl -L -o /opt/flink/lib/clickhouse-client-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/clickhouse-client/0.6.4/clickhouse-client-0.6.4.jar
RUN curl -L -o /opt/flink/lib/clickhouse-data-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/clickhouse-data/0.6.4/clickhouse-data-0.6.4.jar
RUN curl -L -o /opt/flink/lib/clickhouse-http-client-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/clickhouse-http-client/0.6.4/clickhouse-http-client-0.6.4.jar
RUN curl -L -o /opt/flink/lib/clickhouse-jdbc-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/clickhouse-jdbc/0.6.4/clickhouse-jdbc-0.6.4.jar
RUN curl -L -o /opt/flink/lib/jdbc-v2-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/jdbc-v2/0.6.4/jdbc-v2-0.6.4.jar
RUN curl -L -o /opt/flink/lib/httpcore5-5.2.5.jar https://repo1.maven.org/maven2/org/apache/httpcomponents/core5/httpcore5/5.2.5/httpcore5-5.2.5.jar
RUN curl -L -o /opt/flink/lib/httpclient5-5.2.3.jar https://repo1.maven.org/maven2/org/apache/httpcomponents/client5/httpclient5/5.2.3/httpclient5-5.2.3.jar
RUN curl -L -o /opt/flink/lib/httpcore5-h2-5.2.5.jar https://repo1.maven.org/maven2/org/apache/httpcomponents/core5/httpcore5-h2/5.2.5/httpcore5-h2-5.2.5.jar
I did have to build and include
flink-connector-clickhouse-1.0.0.jar
from here My null pointer turned out to be a malformed Clickhouse url, the correct one is
jdbc:clickhouse://my-ch-host:8443?ssl=true
I will now work on see how I can deploy this properly in k8s. Was thinking of a dbt or plain k8s script/configmap I will ask the community on how they do that.
🙌 2
c
Yay! Nice work @Pedro Mázala and @Younes Naguib
🙌 1
So I did get the sql-gateway to run on my cluster. It's not exactly how I imagined setting it up though. I just ran the SQL-Gateway Container and the JobManager Containers on the same POD. Not great, but it will work for now. I am unable to find the Argument, Config Value or Environment variable to have the Sql-Gateway speak to the Job Manager API.
y
I did have it work like that too (Well, got in the pod, and start it)... Not ideal at all... I'll take a stab at it tomorrow, cause it annoyed me too
p
So, joining the party. On the productionizing it side of things. I did not have to have a SQL gateway when I did it. Because my queries were static. So I used the TableAPI to execute my queries (DDL and inserts/selects) and have it running forever. I'd only enable the gateway if I had to run manual queries against the cluster.
c
@Pedro Mázala - Yep, solid advice. This is a "dev cluster" so I'm looking for a way to submit a query test and then tear it down. It's all manual at this point.
p
Copy code
public void run() throws IOException {
        Configuration configuration = new Configuration();
        configuration.setString("table.exec.sink.not-null-enforcer", "DROP");

        EnvironmentSettings envSettings =
                EnvironmentSettings.newInstance()
                        .inStreamingMode()
                        .withConfiguration(configuration)
                        .build();

        StreamTableEnvironment tableEnvironment =
                StreamTableEnvironment.create(env, envSettings);

        for (String fileFromPath :
                List.of(
                        "ddl-1.sql",
                        "ddl-2.sql",
                        "stream.sql")) {
            tableEnvironment.executeSql(fileContent.get(QUERIES_PATH + fileFromPath));
        }
    }
This is how I create them
🙌 1
y
At this point, I want to try dbt as we're adopting it. I'll let you know how that goes
a
@Chad Theis +1 on the
WARN  org.apache.flink.client.program.rest.RestClusterClient       [] - Attempt to submit job ''
issue. I am also struggling to get a connection to the Flink cluster from outside the gateway pod. What I would also like to be able to do is deploy a SQL job to job manager from outside the cluster. I couldn't find any configuration to set this communication.
y
Hello there, Got this thing to work in multiple tiers! sql-client->sql-gateway->flink-cluster 1. Install the operator. 2. Create a flink cluster 3. Create the sql gateway, the trick with this one is to use same config map for the config.yaml as flink cluster 4. Run the sql-client as gateway.
Copy code
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: flink-cluster
  namespace: flink
spec:
  flinkConfiguration:
    jobmanager.rpc.address: flink-cluster.flink
    jobmanager.rpc.port: '6123'
    jobmanager.scheduler: adaptive
    taskmanager.memory.managed.fraction: '0.1'
    taskmanager.memory.process.size: 8192m
    taskmanager.numberOfTaskSlots: '8'
  flinkVersion: v1_20
  jobManager:
    replicas: 1
    resource:
      cpu: 1
      memory: 2048m
  podTemplate:
    apiVersion: v1
    kind: Pod
    spec:
      containers:
        - name: flink-main-container
  serviceAccount: flink
  taskManager:
    replicas: 2
    resource:
      cpu: 4
      memory: 8192m
That will generate a nice configmap for you to use for the sql gateway
Copy code
kind: Deployment
apiVersion: apps/v1
metadata:
  name: flink-sql-gateway
  namespace: flink
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flink-sql-gateway
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: flink-sql-gateway
    spec:
      volumes:
        - name: flink-config-volume
          configMap:
            name: flink-config-flink-cluster
            items:
              - key: flink-conf.yaml
                path: config.yaml
            defaultMode: 420
      containers:
        - name: sql-gateway
          command:
            - /docker-entrypoint.sh
          args:
            - bash
            - '-c'
            - >-
              /opt/flink/bin/sql-gateway.sh start-foreground
              -Dsql-gateway.endpoint.rest.address=flink-cluster-api.flink
          ports:
            - containerPort: 8083
              protocol: TCP
          volumeMounts:
            - name: flink-config-volume
              mountPath: /opt/flink/conf
      restartPolicy: Always
      serviceAccount: flink
At this point, you should have a running sql-gateway. Connecting to it using
sql-client.sh gateway --endpoint ....
Creating a service for the gateway would be nice too 🙂 At this point, I just build a deployment that ran sql-client and deployed and monitored my sql script
I don't use a catalog. I monitor consumer lag, as I get data from Kafka, and my sql-client deployment just keep that in check.
a
Thanks @Younes Naguib! To communicate with a remote host, I tried setting the rest.address variable to the remote host name while initialising the gateway and it worked. Below is the command used.
sql-gateway.sh start-foreground -Dsql-gateway.endpoint.rest.address=localhost -Drest.address=flink-host
. FYI:
rest.address
and
rest.port
is the variable which needs to be set for establishing remote communications to the jobmanager.