Hi All! I have a problem with PyFlink(1.17.0) -> NumPy(1.21.6) with M1 processor while running any of PyFlink Examples.

Original error was: dlopen(/Users/ysmirnov/Documents/Repos/flink-python-jobs/venv/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so, 0x0002): tried: '/Users/ysmirnov/Documents/Repos/flink-python-jobs/venv/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/ysmirnov/Documents/Repos/flink-python-jobs/venv/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so' (no such file), '/Users/ysmirnov/Documents/Repos/flink-python-jobs/venv/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

I've tried to use conda, reinstalling numpy and pyflink, set "-arch arm64" in pip install, tried all pythons 3.7-3.10, tried to use rosetta. Nothing helps. If anyone knows how to solve this issue, please help.

@Юрий Смирнов Could you prepare a clean virtual environment and just install NumPy(1.21.6) and see if it works?

@Dian Fu this way it works

OK. Then could you install PyFlink 1.17.0 in this virtual environment(with NumPy already installed) and check if it works?

For above example it still works, but for pyflink job (this one I'm trying to implement) it is the same error

@Luning (Lucas) Wang @Yun Tang @Sebastian Mattheis hello guys! Maybe you can help somehow with this issue? We were expecting that with 17th version there won’t be troubles of working with PyFlink on M1, but for now we can’t run even an example..

@Юрий Смирнов Could you post the full exception stack?

I had ever worked on RocksDB with M1 chips, however, I did not have such experience with Python on M1 machines.

@Dian Fu Here it is

@Юрий Смирнов This is a little wired. The exception occurs at the line

import numpy as np

. To me, there should be no difference between calling

import numpy as np

in PyFlink and calling it manually. Could you double check whether

import numpy as np

works by manually executing it using the same virtual environment?

Checked. It works on this example, with same venv >

@Юрий Смирнов Could you add the following configuration:

python.execution-mode: process

to your PyFlink job and check if it works?

Same result 😞

@Юрий Смирнов Could you execute it as following? That’s source the virtual environment before executing PyFlink.

1) remove `python.execution-mode: process`
2) source /Users/ysmirnov/Documents/Repos/flink-python-jobs/venv/bin/activate
3) python /Users/ysmirnov/Documents/Repos/flink-python-jobs/word_count/word_count.py

Same from PyCharm terminal, and common macos terminal

@Юрий Смирнов I have no idea any more.😅

Nope 😞 Already tried. And now again - nothing changes.

I guess you could try to reinstall the whole virtual environment.

Unfortunately, same result

@Юрий Смирнов I guess we should raise an issue for Flink, because even with all improvements in the 17th version, working with PyFlink on M1 silicon is impossible