Hello Team.. Can someone help on this please. We a...
# troubleshootingp
Palani
05/22/2023, 2:31 PMHello Team.. Can someone help on this please. We are using AWS KDA and trying to set few Kafka related properties using properties..* But in KDA log we are seeing the warning as attached image. How to set those kafka related properties?
j
Jeremy Ber
05/22/2023, 2:33 PMis this using legacy KafkaSource or FlinkKafkaConsumer?
p
Palani
05/22/2023, 2:37 PMWe are using Python Table API to read from kafka. If thats what you are asking about.
j
Jeremy Ber
05/22/2023, 2:38 PMCan you show an example of how you are setting the properties?
p
Palani
05/22/2023, 2:42 PM Copy code
CREATE TABLE IF NOT EXISTS TEST_TABLE (
TEST STRING,
user_action_time AS PROCTIME()
) WITH (
'connector' = 'kafka',
'topic' = 'TEST_TOPIC',
'properties.bootstrap.servers' = 'TEST_BOOTSTRAP',
'properties.group.id' = 'GROUP_ID',
'scan.startup.mode' = 'EARLIEST',
'format' = 'raw',
'<http://properties.fetch.max.wait.ms|properties.fetch.max.wait.ms>'='100',
'properties.fetch.min.bytes'='4000',
'properties.auto.offset.reset'='latest',
'properties.enable.auto.commit'='true',
'properties.isolation.level'='read_committed'
)Palani
05/22/2023, 2:44 PMParticularly we are concerned about properties.*
j
Jeremy Ber
05/22/2023, 2:49 PMlooks ok to me, i see its saying
group.idproperties.group.idJeremy Ber
05/22/2023, 2:50 PMJeremy Ber
05/22/2023, 2:51 PMwhen the logs first start up and initialize the kafka consumer you should see it along with the properties that are set
Jeremy Ber
05/22/2023, 2:51 PMwould look something like:
Copy code
2020-06-11 14:22:13,319 INFO org.apache.flink.kafka.shaded.org.apache.kafka.clients.consumer.ConsumerConfig - ConsumerConfig values:
<http://auto.commit.interval.ms|auto.commit.interval.ms> = 5000
auto.offset.reset = latest
bootstrap.servers = [<http://c1-dsj-hadoop144.bj:9092|c1-dsj-hadoop144.bj:9092>, <http://c1-dsj-hadoop145.bj:9092|c1-dsj-hadoop145.bj:9092>, <http://c2-dsj-hadoop184.bj:9092|c2-dsj-hadoop184.bj:9092>, <http://c2-dsj-hadoop185.bj:9092|c2-dsj-hadoop185.bj:9092>]
check.crcs = true
client.dns.lookup = default
client.id =
<http://connections.max.idle.ms|connections.max.idle.ms> = 540000
<http://default.api.timeout.ms|default.api.timeout.ms> = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800p
Palani
05/22/2023, 3:02 PMYes thats correct. In ConsumerConfig log message its showing all the properties and its value we are giving/setting via the table config. after that its also giving the warning as well. For an instance in table properties we are setting properties.fetch.min.bytes = 4000 and its getting reflected as fetch.min.bytes = 4000 in ConsumerConfig log message after that its also throwing error saying fetch.min.bytes provided by unknown. We are thinking its not getting set because there is no change in application behaviour at all.
j
Jeremy Ber
05/22/2023, 4:23 PMI believe these are just warnings as expressed in the pull request and can be ignored if your properties are being honored
p
Palani
05/23/2023, 8:04 AMOkay.. may be we will try to play around. As of now we couldn't find any effect for the properties being set
👍 1
Palani
06/01/2023, 10:40 AMSeems to be yes that error is misleading its taking it is what we think.. there is some difference in application run
j
Jeremy Ber
06/01/2023, 12:36 PMglad to hear that thanks for the update