Hi guy, In flink sql, I want to create a table wit...
# troubleshooting
l
Hi guy, In flink sql, I want to create a table with kafka credential:
Copy code
CREATE TABLE src.kafka.kafka_transactions (
  id BIGINT,
  type STRING,
  amount DOUBLE
) WITH (
  'connector' = 'kafka',
  'topic' = 'core-transaction-item',
  'properties.bootstrap.servers' = 'generic.kafka.db-dev:9092',
  'properties.group.id' = 'flink-sql-transactionitem',
  'scan.startup.mode' = 'earliest-offset',
  'format' = 'protobuf',
  'protobuf.message-class-name' = 'TransactionItem',
  'properties.security.protocol' = 'SASL_PLAINTEXT',
  'properties.sasl.mechanism' = 'SCRAM-SHA-512',
  'properties.sasl.jaas.config' = 'org.apache.kafka.common.security.scram.ScramLoginModule required username="${KAFKA_USERNAME}" password="${KAFKA_PASSWORD}";'
);
---> but it's not work. Seem flink sql client can not read directly KAFKA_USERNAME even I passed them when start sql client like this:
Copy code
./bin/sql-client.sh gateway --endpoint 0.0.0.0:8083  -Drest.address=xxxx  -Drest.port=8081 \
    -DKAFKA_USERNAME="$KAFKA_USERNAME" \
    -DKAFKA_PASSWORD="$KAFKA_PASSWORD"
So is there anyway I could pass Kafka credential to flink sql? This syntax run correctly if I use plaintext for KAKFA_USERNAME / Password instead env variable.
t
I don't think we can set environemnt variable in flink directly, although we can pass it in form of following env.java.opts: -DKAFKA_USERNAME=myuser -DKAFKA_PASSWORD=mypass but that too will set java properties not environemnt variable. One of the solution is you can create a sql file and substitute your variables in your file as I have created a template file like the following: SET sql-client.execution.result-mode=TABLEAU; CREATE CATALOG src WITH ('type' = 'generic_in_memory'); USE CATALOG src; CREATE DATABASE kafka; USE src.kafka; CREATE TABLE src.kafka.kafka_transactions ( id STRING, type BIGINT ) WITH ( 'connector' = 'kafka', 'topic' = 'input-topic', 'properties.bootstrap.servers' = '$kafka_bootstrap', 'properties.group.id' = 'flink-sql-transactionitem', 'scan.startup.mode' = 'earliest-offset', 'format' = 'csv' ); select * from src.kafka.kafka_transactions; then used the following commands: 1.export kafka_bootstrap=localhost:9092 2. envsubst < create_table.sql.template > create_table.sql 3. flink-2.0.0/bin/sql-client.sh gateway --endpoint localhost:8085 -Drest.address=localhost -Drest.port=8081 -f create_table.sqlql there are other ways also by storing these variables in var file using it in sql or by substituting through linux.