Hi all, we are stuck at a problem in our company -...
# troubleshooting
a
Hi all, we are stuck at a problem in our company - We are using Kafka consumer to read messages and confluent - avro schema registry to deserialize the messages . We need to pass keystore and truststore secret file paths in Flink's Kafka Source table's Connector configuration properties. The property constraint is that- it can only read from local path. How to make these secret files available in local memory of all the nodes? We cannot use HDFS (s3 is the only option where secrets can be stored). Is there any way the secres can be pulled from s3 during runtime and saved in /tmp directory of each node . And this local file path can be set in properties.ssl.keystore.location Anyhelp would be much appreciated. πŸ™
s
a
Hi @Sharath, This SSL keystore configuration is working for apapche kafka with s3import when we save keystore files in s3.
Copy code
"ssl.keystore.location", "${s3import:" + region + ":" + keystoreS3Bucket + "/" + keystoreS3Path + "}");
But confluent schema registry is only accepting local path for keystore/truststore location . Above does not work with confluent cloud schema registry for some reason.
s
Did you try supplying them as PEM certificates ?
a
Pem certificates will also not work. Problem is with Confluent -Avro- it reads secrets only from local file path( Does not read from S3)
j
@Anshuta Awasthi - if ur deploying ur Flink workloads on k8s did u try secrets volume mount into the pod ? You will need additional service to pull the files from s3 and create k8s secrets.
plusone 1