Hi everyone I ve tried to make my job listen to Kinesis stre Apache Flink #troubleshooting

Hi, everyone, I've tried to make my job listen to ...

Xiaobing Xia

08/07/2023, 1:55 AM

Hi, everyone, I've tried to make my job listen to Kinesis stream, and then writing to s3. I'm using Flink 1.17.1. The code completely works at my local laptop. But when I run the same jar on EMR (Well, I rebuilt the jar with 1.17.0 since EMR only supports 1.17.0. Only dependency package changes), it reports following error:

Copy code

2023-08-07 01:52:24,074 WARN  org.apache.flink.runtime.taskmanager.Task                    [] - Source: Custom Source (4/4)#4 (8f758eac625e1ced5221cdb24c916c35_bc764cd8ddf7a0cff126f51c16239658_3_4) switched from RUNNING to FAILED with failure cause:
java.lang.NoSuchFieldError: ENABLE_LEGACYSOURCE_EMIT_IDLETIME
        at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:161) ~[flink-dist-1.17.0.jar:1.17.0]
        at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:74) ~[flink-dist-1.17.0.jar:1.17.0]
        at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:335) ~[flink-dist-1.17.0.jar:1.17.0]

any one has seen this issue before? I'm guessing maybe the kinesis source package (flink-connector-kinesis: 4.1.0-1.17) has some conflict with the jar at EMR with Flink 1.17.0?

Xiaobing Xia

08/07/2023, 4:04 AM

Update: I ssh into the EMR master node, and replace the flink-dist-1.17.0.jar with flink-dist-1.17.1.jar from online. It works.

Xiaobing Xia

08/07/2023, 4:05 AM

Does that mean, this is a bug for 1.17.0?

Prathit Malik

08/07/2023, 7:04 AM

This is bug/issue in flink distro installed on EMR. we faced similar issue (with flink sql) with 1.16 on EMR (confirmed after we extracted emr flink jar) Have reported it to AWS support 😄

Samrat Deb

08/07/2023, 7:21 AM

Can you share your pom. I will look into the issue from EMR end.

Samrat Deb

08/07/2023, 7:23 AM

can you check if

flink-streaming-java

is having the scope provided.

Copy code

<!-- <https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-java> -->
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-streaming-java</artifactId>
    <version>1.17.0</version>
    <scope>provided</scope>
</dependency>

Also , can you try <dependency><flink-kinesis><exclude><flink-core> and <flink-runtime> and <flink-streaming-java>

Xiaobing Xia

08/07/2023, 3:32 PM

Yes, it has the "provided" scope.

Xiaobing Xia

08/07/2023, 3:33 PM

@Prathit Malik thank you so much!

Xiaobing Xia

08/07/2023, 3:33 PM

The kinesis related dependency is:

Copy code

<dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kinesis</artifactId>
            <version>4.1.0-${flink.major.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-aws-kinesis-streams</artifactId>
            <version>4.1.0-${flink.major.version}</version>
        </dependency>

👍 1

3 Views

Open in Slack

Previous Next