What is the simplest way to install RC1 for 1.16.1 on a Mac with M1 with both PyFlink and Java/Scala Flink?

Download the binary from https://dist.apache.org/repos/dist/dev/flink/flink-1.16.1-rc1/flink-1.16.1-bin-scala_2.12.tgz ?

super and what do I do next?

I'm not sure what you want to achieve

in my project

You can unzip the file and you'll end up with a cluster that you could run

this looks like the cluster code

Like I said, I'm not sure what you want to achieve but "install RC1" for me implies that you want an RC1 cluster 🙂

I am working on a mixed project with both Python and Java code now. Import needed python libs into a virtual environment:

CONDA_SUBDIR=osx-64 conda create -n pystream39 python=3.9
conda activate pystream39
pip install apache-flink
pip install httpx

Code setup

all_files = [
    "flink-connector-jdbc-1.16.0.jar",
    "flink-sql-connector-postgres-cdc-2.3.0.jar",
    "postgresql-9.4.1212.jar"]

    jar_path = make_jar_paths(jar_filenames)

    print("JAR PATH: ", jar_path)
    t_env.get_config().set("pipeline.jars", jar_path)

That’s a tricky one because I have 0 knowledge on PyFlink 😅

Well PyFlink is a complicated construction. Maybe it is better to wait for final release. Thanks

@Sami Badawi Download packages from https://dist.apache.org/repos/dist/dev/flink/flink-1.16.1-rc1/python/ pip uninstall apache-flink-libraries pip uninstall apache-flink pip install apache-flink-libraries-1.16.1.tar.gz pip install apache_flink-1.16.1-xxx.whl (depends on your local environment)

Super thanks! I got this error when I tried:

virtualenv -p /opt/homebrew/Frameworks/Python.framework/Versions/3.9/bin/python3.9 flink1161
source flink1161/bin/activate
pip install apache-flink-libraries-1.16.1.tar.gz
pip install apache_flink-1.16.1-cp39-cp39-macosx_11_0_arm64.whl
python -m pystreaming.event_files_read
Running event_files_read
<frozen importlib._bootstrap>:228: RuntimeWarning: apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate binary incompatibility. Expected 24 from C header, got 32 from PyObject
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/homebrew/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/sami/code/pystreaming/pystreaming/event_files_read.py", line 55, in <module>
    t_env = make_env()
  File "/Users/sami/code/pystreaming/pystreaming/env_setup.py", line 37, in make_env
    envs = make_envs()
  File "/Users/sami/code/pystreaming/pystreaming/env_setup.py", line 26, in make_envs
    env: StreamExecutionEnvironment = StreamExecutionEnvironment.get_execution_environment()
  File "/Users/sami/script/pyflink/1.16.1-rc1/flink1161/lib/python3.9/site-packages/pyflink/datastream/stream_execution_environment.py", line 837, in get_execution_environment
    return StreamExecutionEnvironment(j_stream_exection_environment)
  File "/Users/sami/script/pyflink/1.16.1-rc1/flink1161/lib/python3.9/site-packages/pyflink/datastream/stream_execution_environment.py", line 65, in __init__
    self._open()
  File "/Users/sami/script/pyflink/1.16.1-rc1/flink1161/lib/python3.9/site-packages/pyflink/datastream/stream_execution_environment.py", line 1033, in _open
    startup_loopback_server()
  File "/Users/sami/script/pyflink/1.16.1-rc1/flink1161/lib/python3.9/site-packages/pyflink/datastream/stream_execution_environment.py", line 1023, in startup_loopback_server
    from pyflink.fn_execution.beam.beam_worker_pool_service import \
  File "/Users/sami/script/pyflink/1.16.1-rc1/flink1161/lib/python3.9/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py", line 44, in <module>
    from pyflink.fn_execution.beam import beam_sdk_worker_main  # noqa # pylint: disable=unused-import
  File "/Users/sami/script/pyflink/1.16.1-rc1/flink1161/lib/python3.9/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py", line 21, in <module>
    import pyflink.fn_execution.beam.beam_operations # noqa # pylint: disable=unused-import
  File "/Users/sami/script/pyflink/1.16.1-rc1/flink1161/lib/python3.9/site-packages/pyflink/fn_execution/beam/beam_operations.py", line 27, in <module>
    from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend, RemoteOperatorStateBackend
  File "/Users/sami/script/pyflink/1.16.1-rc1/flink1161/lib/python3.9/site-packages/pyflink/fn_execution/state_impl.py", line 33, in <module>
    from pyflink.fn_execution.beam.beam_coders import FlinkCoder
  File "/Users/sami/script/pyflink/1.16.1-rc1/flink1161/lib/python3.9/site-packages/pyflink/fn_execution/beam/beam_coders.py", line 27, in <module>
    from pyflink.fn_execution.beam import beam_coder_impl_fast as beam_coder_impl
  File "pyflink/fn_execution/beam/beam_coder_impl_fast.pyx", line 1, in init pyflink.fn_execution.beam.beam_coder_impl_fast
KeyError: '__pyx_vtable__'

@Sami Badawi Is the python you use pre-installed by the system, or the venv created by conda, I suspect it has something to do with this. There seems to be no this problem with the python env created by conda

Thanks @Xingbo Huang. I created my venv using either Conda or Virtualenv, I got the same error in both cases. And the same happened when I used the final 1.16.1 release.

@Sami Badawi I have found the root cause. You can execute the following command to solve the problem currently.

pip install cython==0.29.24
brew install gcc
pip uninstall apache-flink
pip uninstall apache-beam
pip install apache-flink

Thanks @Xingbo Huang! I will git that a try.