Hi all, has anyone successfully run apache beam application in python (utilising cross-language pipeline) on a dockerized flink cluster?

👋 I personally didn’t but I believe it was more or less common set up, which is supported for multiple years

There’s not so many resources (at least not well described) as it might seem.

apache/beam_flink1.*_job_server

images somehow didn’t work for me and apache beam sdk harnessing spins up docker images on

host

docker network, which you can’t use e.g. in your docker-compose.yml (where you might have your flink cluster), because docker complains about ports. Long story short i created dockerfile which installs everything required (flink, java, python, docker …) into ubuntu image and runs pipelines from there.

@Viktor Hrtanek did you get this working? We're bringing up a Flink session cluster using the Flink K8s Operator and we got a python beam job submitted to the cluster. Now just working out issues that we're hitting along the way

@Stephen Chu yes i did, as i mentioned i created a standalone dockerfile with all in place

@Viktor Hrtanek - thanks, if I understood you correctly, stitching together all the job server images + sdk hardnessing across multiple containers was painful and so now you just have everything all together in one place

@Stephen Chu I assume networking caused me troubles while using multiple containers, but to be fair I don’t have much experience with docker/ kubernetes, so it might work for you. What you might try is to change apache beam source code to spin sdk harnessing containers on different than

host

network to make see all containers each other. In my case i had one docker compose file with all services like flink job/ task servers/ rocksdb etc. but it didn’t see containers spinned up by beam during sdk harnessing (in my case due to kafka connection from python), so i decided rather put everything into one dockerfile managed by me than to change beam source code. Maybe there are other options too.

Thanks a bunch, @Viktor Hrtanek. This gives me a lot of good stuff to think about