Hi all I am currently looking at Flink deployment high avail Apache Flink #troubleshooting

Hi all, I am currently looking at Flink deployment...

Simon Lawrence

03/21/2023, 9:58 AM

Hi all, I am currently looking at Flink deployment, high availability using zookeeper being the goal. The documentation provides examples for deploying a flink cluster running in session mode, however I am wondering if it is possible to deploy in application mode using zookeeper for high availability? If anyone could point me in the right direction that would be amazing. Thank you!

kingsathurthi

03/21/2023, 10:17 AM

there is no much difference between application mode and session mode. use below example to have application mode(just use zoopkeer HA detail instead of kubernetes HA mentioned in the example) https://github.com/apache/flink-kubernetes-operator/blob/main/examples/basic-checkpoint-ha.yaml

Simon Lawrence

03/21/2023, 10:31 AM

Hmm, so that suggests to me that it’s just a case of the configuration. Would the same thing work for standalone mode? ie. Just set up the configuration files for zookeeper with HA then run the application following the guidelines in https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/overview/#application-mode

kingsathurthi

03/21/2023, 12:32 PM

Yeah, you can, but I recommend you to use native mode with flink kubernetes operator

Simon Lawrence

03/21/2023, 12:41 PM

Unfortunately for our deployment kubernetes is not an option currently

kingsathurthi

03/21/2023, 2:53 PM

Oh okay

Simon Lawrence

03/21/2023, 4:24 PM

To answer this question for myself and for future people looking for this: Yes, it is possible to run using Application mode with zookeeper for high availability. If you configure your flink-conf.yaml to use zookeeper high availaibility, then run a zookeeper quorum, it will take care of ensuring that there is only a single JobManager actually running the job from application mode.

👍 1

Simon Lawrence

03/21/2023, 4:52 PM

The following is a worked example as per the flink documentation, it uses the same principles as this example and First, amend the flink-conf.yaml file as per here. Ensure your zoo.cfg file is setup as per Then start the zookeeper quorum:

Copy code

$ ./bin/start-zookeeper-quorum.sh
Starting zookeeper daemon on host localhost.

Make sure the application jar file is on the classpath.

Copy code

$ cp ./examples/streaming/TopSpeedWindowing.jar lib/

Then, we can launch the first JobManager (Do this in foreground mode for visibility and ease of killing it) :

Copy code

$ ./bin/standalone-job.sh start-foreground --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing

This JobManager will be launched in application mode, and will start the job straight away. Start a taskmanager to make sure the job actually gets run.

Copy code

$ ./bin/taskmanager.sh start-foreground

We can launch a second JobManager which will act as a failover JobManager as follows:

Copy code

$ ./bin/standalone-job.sh start-foreground --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing -D rest.port=8082

You should now be able to access the web ui on both localhost:8081 and localhost:8082 You should see that the first job manager is running the job due to the line of logging containing:

Starting execution of job 'CarTopSpeedWindowingExample'

You shouldn’t see that logging in the second job manager, as it should be in a standby state. You should see the output in the taskmanager to prove that the job is flowing. Kill the first JobManager using ^+C. You should see that the second JobManager now logs out

Starting execution of job 'CarTopSpeedWindowingExample'

On localhost:8082 you should see that the job has restarted. It should continue to run using the same taskmanager which you can check in stdout in the console.

👍 1

Simon Lawrence

03/21/2023, 4:54 PM

The main downside of this, as mentioned at the end of https://apache-flink.slack.com/archives/C03G7LJTS2G/p1671623836414569 is that zookeeper can’t automatically restart the failed jobmanager. However this is just a different problem to solve, that shouldn’t be too hard.

2 Views

Open in Slack

Previous Next