I m observing a strange behavior using `KafkaSource` vs `Fli Apache Flink #troubleshooting

I'm observing a strange behavior using `KafkaSourc...

Guruguha Marur Sreenivasa

01/20/2023, 4:59 PM

I'm observing a strange behavior using

KafkaSource

FlinkKafkaConsumer

. My streaming application runs just fine using

FlinkKafkaConsumer

but when I use the

KafkaSource

API, it errors out saying, "unable to fetch topic metadata"! Anyone faced this issue before? The configuration to both clients are exactly the same.

Martijn Visser

01/20/2023, 6:04 PM

Which version of Flink are you using?

Guruguha Marur Sreenivasa

01/20/2023, 6:04 PM

1.14.3

Martijn Visser

01/20/2023, 6:06 PM

I can only find Python related errors on that, are you using PyFlink?

Guruguha Marur Sreenivasa

01/20/2023, 6:06 PM

No, its Java

Martijn Visser

01/20/2023, 6:06 PM

Weird

Martijn Visser

01/20/2023, 6:07 PM

The only thing that I can really find on that is that your Kafka service would be running low on memory; there's not an obvious thing what could be the case. We have done a number of improvements in the used Kafka Clients in later versions (1.15 and 1.16). I do think that upgrading would benefit you

Guruguha Marur Sreenivasa

01/20/2023, 6:09 PM

Oh ok. Our Kafka cluster is super healthy - ~30% cluster CPU usage. Let me think on upgrading the clients. Will this upgrade have any impact on the cluster that will be running on

1.14.3

or do we need to upgrade cluster as well

1.15+

Martijn Visser

01/20/2023, 6:12 PM

If you upgrade a job to a later version, you will also need to upgrade the cluster to the same version

Guruguha Marur Sreenivasa

01/20/2023, 6:15 PM

Got it, will check it out. Thanks.

Open in Slack

Previous Next