Hey team how are you guys managing multiple python versions

Question

Hey team how are you guys managing multiple python versions on your local I m trying to use `pyenv` however the gradle plugin seems to be picking up my system python instead I ve confirmed pyenv works on the terminal

user · Answer

I m running into this because `pyarrow` is failing to install for me and I suspect it s because Gradle is ignoring pyenv and picking up my Macs system s version of `3 8` while our current is `3 7 9`

user · Answer

does seem our python plugin looks at the system python <https github com xvik gradle use python plugin blob master src main groovy ru vyarus gradle plugin python PythonExtension groovy L19> although I haven t seen a good reason why how it s ignoring my `pyenv` shim

user · Answer

personally I do use pyenv on my mac

user · Answer

Good to know Is it picking up the right version Also have you seen this PyArrow error before Chris ``` cOS version 11 0 than being linked 10 9 ld warning dylib usr local Cellar apache arrow 3 0 0 3 lib libarrow dylib was built for newer macOS version 11 0 than being linked 10 9 100 Built target fs Finished cmake build for pyarrow Bundling includes release include Moving built C extension release lib cpython 38 darwin so to build path private var folders 4h k452x8ns06q7w w7wkb3j16h0000gn T pip install t1413mpq pyarrow build lib macosx 10 14 6 x86 64 3 8 pyarrow lib cpython 38 darwin so Moving built C extension release fs cpython 38 darwin so to build path private var folders 4h k452x8ns06q7w w7wkb3j16h0000gn T pip install t1413mpq pyarrow build lib macosx 10 14 6 x86 64 3 8 pyarrow fs cpython 38 darwin so Moving built C extension release csv cpython 38 darwin so to build path private var folders 4h k452x8ns06q7w w7wkb3j16h0000gn T pip install t1413mpq pyarrow build lib macosx 10 14 6 x86 64 3 8 pyarrow csv cpython 38 darwin so Moving built C extension release json cpython 38 darwin so to build path private var folders 4h k452x8ns06q7w w7wkb3j16h0000gn T pip install t1413mpq pyarrow build lib macosx 10 14 6 x86 64 3 8 pyarrow json cpython 38 darwin so Moving built C extension release compute cpython 38 darwin so to build path private var folders 4h k452x8ns06q7w w7wkb3j16h0000gn T pip install t1413mpq pyarrow build lib macosx 10 14 6 x86 64 3 8 pyarrow compute cpython 38 darwin so Did not find release cuda cpython 38 darwin so Cython module cuda failure permitted Did not find release flight cpython 38 darwin so Cython module flight failure permitted Did not find release dataset cpython 38 darwin so Cython module dataset failure permitted Did not find release parquet cpython 38 darwin so Cython module parquet failure permitted Did not find release orc cpython 38 darwin so Cython module orc failure permitted Did not find release plasma cpython 38 darwin so Cython module plasma failure permitted Did not find release s3fs cpython 38 darwin so Cython module s3fs failure permitted Did not find release hdfs cpython 38 darwin so Cython module hdfs failure permitted Did not find release gandiva cpython 38 darwin so Cython module gandiva failure permitted installing to build bdist macosx 10 14 6 x86 64 wheel running install running install lib creating build bdist macosx 10 14 6 x86 64 creating build bdist macosx 10 14 6 x86 64 wheel creating build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow public api pxi gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow io pxi gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow serialization pxi gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow orc py gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow compute pyx gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow orc pxd gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow builder pxi gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow array pxi gt build bdist macosx 10 14 6 x86 64 wheel pyarrow ld warning dylib usr local Cellar apache arrow 3 0 0 3 lib libarrow dylib was built for newer macOS version 11 0 than being linked 10 9 Building wheel for bcrypt PEP 517 started copying build lib macosx 10 14 6 x86 64 3 8 pyarrow generated version py gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow compat py gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow cuda pyx gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow benchmark py gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow parquet py gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow csv pyx gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow config pxi gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow dataset pyx gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow flight pyx gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow ipc py gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow util py gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow flight py gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow tensor pxi gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow init pxd gt build bdist macosx 10 14 6 x86 64 wheel pyarrow copying build lib macosx 10 14 6 x86 64 3 8 pyarrow cffi py gt build bdist macosx 10 14 6 x86 64 wheel pyarrow creating build bdist macosx 10 14 6 x86 64 wheel pyarrow include error can t copy build lib macosx 10 14 6 x86 64 3 8 pyarrow include arrow doesn t exist or not a regular file ERROR Failed building wheel for pyarrow```

user · Answer

i didn t personally run into it but it does remind me of others encountering that <https airbytehq slack com archives C01AHCDHCKE p1612277222072500>

user · Answer

Installing your python amp java on a fresh mac is quite difficult to get it right it d probably be a good thing to take notes for the next ones going through it

user · Answer

Definitely taking notes

user · Answer

Chris can ya do me a favor and give me a peek into one of your ` venv` folders from a module that contains Python whenever convenient Something like ``` source file git master cd venv |gke causal sky 277804 us central1 autopilot cluster 1 default venv git master ls al total 0 drwxr xr x 6 davinchia staff 192 Mar 2 16 37 drwxr xr x 15 davinchia staff 480 Mar 2 16 37 lrwxr xr x 1 davinchia staff 84 Mar 2 16 37 Python gt usr local Cellar python 3 9 3 9 2 1 Frameworks Python framework Versions 3 9 Python drwxr xr x 5 davinchia staff 160 Mar 2 16 37 bin drwxr xr x 3 davinchia staff 96 Mar 2 16 37 include drwxr xr x 3 davinchia staff 96 Mar 2 16 37 lib``` I m curious where your python is pointed to this is wrong

user · Answer

i have this in my zshrc it might be related ```alias brew= env PATH=$ PATH $ pyenv root shims brew ```

user · Answer

mmmh i think i remember something about not doing symlinks in my venv yes here is my `airbyte airbyte integrations connectors source file venv` ```ls al total 0 drwxr xr x 5 chris staff 160 Feb 25 18 24 drwxr xr x 20 chris staff 640 Mar 1 16 53 drwxr xr x 53 chris staff 1696 Feb 25 18 28 bin drwxr xr x 3 chris staff 96 Feb 25 18 24 include drwxr xr x 3 chris staff 96 Feb 25 18 24 lib```

user · Answer

```ls al bin total 6840 drwxr xr x 53 chris staff 1696 Feb 25 18 28 drwxr xr x 5 chris staff 160 Feb 25 18 24 drwxr xr x 5 chris staff 160 Feb 25 18 28 pycache rw r r 1 chris staff 2269 Feb 25 18 24 activate rw r r 1 chris staff 1488 Feb 25 18 24 activate csh rw r r 1 chris staff 3153 Feb 25 18 24 activate fish rw r r 1 chris staff 1751 Feb 25 18 24 activate ps1 rw r r 1 chris staff 1210 Feb 25 18 24 activate xsh rw r r 1 chris staff 1517 Feb 25 18 24 activate this py rwxr xr x 1 chris staff 1072 Feb 25 18 26 airbyte python test rwxr xr x 1 chris staff 1032 Feb 25 18 26 base python rwxr xr x 1 chris staff 295 Feb 25 18 24 black rwxr xr x 1 chris staff 290 Feb 25 18 24 black primer rwxr xr x 1 chris staff 296 Feb 25 18 24 blackd rwxr xr x 1 chris staff 296 Feb 25 18 26 chardetect rwxr xr x 1 chris staff 280 Feb 25 18 28 distro rwxr xr x 1 chris staff 309 Feb 25 18 24 dmypy rwxr xr x 1 chris staff 290 Feb 25 18 28 docker compose rwxr xr x 1 chris staff 282 Feb 25 18 28 dotenv rwxr xr x 1 chris staff 291 Feb 25 18 26 f2py rwxr xr x 1 chris staff 291 Feb 25 18 26 f2py3 rwxr xr x 1 chris staff 291 Feb 25 18 26 f2py3 7 rwxr xr x 1 chris staff 289 Feb 25 18 24 flake8 rwxr xr x 1 chris staff 284 Feb 25 18 26 genson rwxr xr x 1 chris staff 308 Feb 25 18 26 google oauthlib tool rwxr xr x 1 chris staff 284 Feb 25 18 24 isort rwxr xr x 1 chris staff 1770 Feb 25 18 26 jp py rwxr xr x 1 chris staff 288 Feb 25 18 26 jsonschema rwxr xr x 1 chris staff 305 Feb 25 18 24 mypy rwxr xr x 1 chris staff 1506 Feb 25 18 24 mypyc rwxr xr x 1 chris staff 296 Feb 25 18 25 pip rwxr xr x 1 chris staff 296 Feb 25 18 25 pip3 rwxr xr x 1 chris staff 296 Feb 25 18 25 pip3 7 rwxr xr x 1 chris staff 323 Feb 25 18 26 plasma store rwxr xr x 1 chris staff 296 Feb 25 18 25 py test rwxr xr x 1 chris staff 287 Feb 25 18 24 pycodestyle rwxr xr x 1 chris staff 286 Feb 25 18 24 pyflakes rwxr xr x 1 chris staff 287 Feb 25 18 26 pyrsa decrypt rwxr xr x 1 chris staff 287 Feb 25 18 26 pyrsa encrypt rwxr xr x 1 chris staff 285 Feb 25 18 26 pyrsa keygen rwxr xr x 1 chris staff 308 Feb 25 18 26 pyrsa priv2pub rwxr xr x 1 chris staff 281 Feb 25 18 26 pyrsa sign rwxr xr x 1 chris staff 285 Feb 25 18 26 pyrsa verify rwxr xr x 1 chris staff 296 Feb 25 18 25 pytest lrwxr xr x 1 chris staff 7 Feb 25 18 24 python gt python3 rwxr xr x 1 chris staff 2398 Feb 25 18 24 python config rwxr xr x 1 chris staff 3292600 Feb 25 18 24 python3 lrwxr xr x 1 chris staff 7 Feb 25 18 24 python3 7 gt python3 rwxr xr x 1 chris staff 16145 Feb 25 18 26 runxlrd py rwxr xr x 1 chris staff 286 Feb 25 18 24 stubgen rwxr xr x 1 chris staff 287 Feb 25 18 24 stubtest rwxr xr x 1 chris staff 284 Feb 25 18 24 wheel rwxr xr x 1 chris staff 6488 Feb 25 18 28 wsdump py```

user · Answer

gt i remember something about not doing symlinks in my venv yes promising looks like you are on python 3 7 yes

user · Answer

yes

user · Answer

gt alias brew= env PATH=$ PATH $ pyenv root shims brew probably coming from <https github com pyenv pyenv issues 106>

user · Answer

awesome you have given me better leads to chase down

user · Answer

the most annoying thing is how long each install attempt takes

user · Answer

indeed

user · Answer

our build is horrible

user · Answer

need an excuse to play ping pong stuck out tongue

user · Answer

do you have `apache arrow` and or `apache arrow glib` installed if so what versions are they

user · Answer

is that installed through pip

user · Answer

I believe it s through brew

user · Answer

not installed no

user · Answer

huh I thought I needed to in order to install the Arrow C++ libraries <https arrow apache org install >

user · Answer

i do get pyarrow through pip though `pyarrow==3 0 0`

user · Answer

seems that s only the binary so probably easier to get it working

user · Answer

my `pip install pyarrow` runs into the following error without `brew install apache arrow` ``` Found PkgConfig usr local bin pkg config found version 0 29 2 Could NOT find Arrow missing Arrow DIR Checking for module arrow No package arrow found CMake Error at usr local Cellar cmake 3 19 6 share cmake Modules FindPackageHandleStandardArgs cmake 218 message Could NOT find Arrow missing ARROW INCLUDE DIR ARROW LIB DIR ARROW FULL SO VERSION ARROW SO VERSION Call Stack most recent call first usr local Cellar cmake 3 19 6 share cmake Modules FindPackageHandleStandardArgs cmake 582 FPHSA FAILURE MESSAGE cmake modules FindArrow cmake 419 find package handle standard args cmake modules FindArrowPython cmake 46 find package CMakeLists txt 214 find package ```

user · Answer

i think i might have avoided having python through brew

user · Answer

was everything via pyenv and pip including numpy

user · Answer

yes i installed pyenv with brew and then have everything managed through my pyenv

user · Answer

Did you manage to get it working slightly smiling face

user · Answer

Sadly no since it s not really blocking for now I threw it over the fence to Arrow <https issues apache org jira browse ARROW 11835>

user · Answer

cheating and am excluding that module from my gradle for the time being see no evil

user · Answer

this makes me wonder if figuring out a way to exclude pyarrow is worth it

user · Answer

I m curious how did you install java 14 on your mac btw

user · Answer

I used `jenv` How did you do so

user · Answer

I used <https sdkman io >

user · Answer

oops I wasn t clear in my answer I used `brew` to install then `jenv` to switch versions and manage paths

user · Answer

first time coming across sdkman seems cool

user · Answer

i guess they are both related sdkman jenv and do the same thing