Chad Theis
02/20/2025, 11:45 PMYounes Naguib
02/21/2025, 7:52 PMYounes Naguib
02/21/2025, 7:52 PMChad Theis
02/21/2025, 10:17 PMChad Theis
02/23/2025, 2:13 PMChad Theis
02/23/2025, 2:40 PMYounes Naguib
02/24/2025, 1:26 PMChad Theis
02/24/2025, 2:03 PMYounes Naguib
02/24/2025, 2:06 PMYounes Naguib
02/24/2025, 2:07 PMYounes Naguib
02/24/2025, 2:07 PMChad Theis
02/24/2025, 2:11 PMYounes Naguib
02/24/2025, 2:14 PMYounes Naguib
02/24/2025, 2:19 PMjava.lang.NullPointerException
I guess more fun to comeYounes Naguib
02/24/2025, 8:00 PMRUN curl -L -o /opt/flink/lib/flink-connector-kafka-1.20.jar https://dp-assets-public.s3.us-east-1.amazonaws.com/flink-connector-kafka-3.4.0.jar
RUN curl -L -o /opt/flink/lib/kafka-clients-3.4.0.jar https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/3.4.0/kafka-clients-3.4.0.jar
RUN curl -L -o /opt/flink/lib/clickhouse-client-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/clickhouse-client/0.6.4/clickhouse-client-0.6.4.jar
RUN curl -L -o /opt/flink/lib/clickhouse-data-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/clickhouse-data/0.6.4/clickhouse-data-0.6.4.jar
RUN curl -L -o /opt/flink/lib/clickhouse-http-client-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/clickhouse-http-client/0.6.4/clickhouse-http-client-0.6.4.jar
RUN curl -L -o /opt/flink/lib/clickhouse-jdbc-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/clickhouse-jdbc/0.6.4/clickhouse-jdbc-0.6.4.jar
RUN curl -L -o /opt/flink/lib/jdbc-v2-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/jdbc-v2/0.6.4/jdbc-v2-0.6.4.jar
RUN curl -L -o /opt/flink/lib/httpcore5-5.2.5.jar https://repo1.maven.org/maven2/org/apache/httpcomponents/core5/httpcore5/5.2.5/httpcore5-5.2.5.jar
RUN curl -L -o /opt/flink/lib/httpclient5-5.2.3.jar https://repo1.maven.org/maven2/org/apache/httpcomponents/client5/httpclient5/5.2.3/httpclient5-5.2.3.jar
RUN curl -L -o /opt/flink/lib/httpcore5-h2-5.2.5.jar https://repo1.maven.org/maven2/org/apache/httpcomponents/core5/httpcore5-h2/5.2.5/httpcore5-h2-5.2.5.jar
I did have to build and include flink-connector-clickhouse-1.0.0.jar from here
My null pointer turned out to be a malformed Clickhouse url, the correct one is jdbc:clickhouse://my-ch-host:8443?ssl=true
I will now work on see how I can deploy this properly in k8s.
Was thinking of a dbt or plain k8s script/configmap
I will ask the community on how they do that.Chad Theis
02/24/2025, 8:02 PMChad Theis
02/24/2025, 8:40 PMYounes Naguib
02/24/2025, 10:05 PMPedro Mázala
02/25/2025, 8:16 AMChad Theis
02/25/2025, 1:24 PMPedro Mázala
02/25/2025, 2:35 PMpublic void run() throws IOException {
Configuration configuration = new Configuration();
configuration.setString("table.exec.sink.not-null-enforcer", "DROP");
EnvironmentSettings envSettings =
EnvironmentSettings.newInstance()
.inStreamingMode()
.withConfiguration(configuration)
.build();
StreamTableEnvironment tableEnvironment =
StreamTableEnvironment.create(env, envSettings);
for (String fileFromPath :
List.of(
"ddl-1.sql",
"ddl-2.sql",
"stream.sql")) {
tableEnvironment.executeSql(fileContent.get(QUERIES_PATH + fileFromPath));
}
}
This is how I create themYounes Naguib
02/25/2025, 4:29 PMAmmu s parvathy
03/04/2025, 4:02 PMWARN org.apache.flink.client.program.rest.RestClusterClient [] - Attempt to submit job '' issue. I am also struggling to get a connection to the Flink cluster from outside the gateway pod. What I would also like to be able to do is deploy a SQL job to job manager from outside the cluster. I couldn't find any configuration to set this communication.Younes Naguib
03/07/2025, 1:33 PMapiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: flink-cluster
namespace: flink
spec:
flinkConfiguration:
jobmanager.rpc.address: flink-cluster.flink
jobmanager.rpc.port: '6123'
jobmanager.scheduler: adaptive
taskmanager.memory.managed.fraction: '0.1'
taskmanager.memory.process.size: 8192m
taskmanager.numberOfTaskSlots: '8'
flinkVersion: v1_20
jobManager:
replicas: 1
resource:
cpu: 1
memory: 2048m
podTemplate:
apiVersion: v1
kind: Pod
spec:
containers:
- name: flink-main-container
serviceAccount: flink
taskManager:
replicas: 2
resource:
cpu: 4
memory: 8192m
That will generate a nice configmap for you to use for the sql gateway
kind: Deployment
apiVersion: apps/v1
metadata:
name: flink-sql-gateway
namespace: flink
spec:
replicas: 1
selector:
matchLabels:
app: flink-sql-gateway
template:
metadata:
creationTimestamp: null
labels:
app: flink-sql-gateway
spec:
volumes:
- name: flink-config-volume
configMap:
name: flink-config-flink-cluster
items:
- key: flink-conf.yaml
path: config.yaml
defaultMode: 420
containers:
- name: sql-gateway
command:
- /docker-entrypoint.sh
args:
- bash
- '-c'
- >-
/opt/flink/bin/sql-gateway.sh start-foreground
-Dsql-gateway.endpoint.rest.address=flink-cluster-api.flink
ports:
- containerPort: 8083
protocol: TCP
volumeMounts:
- name: flink-config-volume
mountPath: /opt/flink/conf
restartPolicy: Always
serviceAccount: flink
At this point, you should have a running sql-gateway. Connecting to it using sql-client.sh gateway --endpoint ....
Creating a service for the gateway would be nice too 🙂
At this point, I just build a deployment that ran sql-client and deployed and monitored my sql scriptYounes Naguib
03/07/2025, 1:35 PMAmmu s parvathy
03/12/2025, 5:08 AMsql-gateway.sh start-foreground -Dsql-gateway.endpoint.rest.address=localhost -Drest.address=flink-host .
FYI: rest.address and rest.port is the variable which needs to be set for establishing remote communications to the jobmanager.