https://linen.dev logo
Join Slack
Powered by
Hey team, how are you guys managing multiple pytho...
# contributing-to-airbyte
d
Hey team, how are you guys managing multiple python versions on your local? I’m trying to use 
pyenv
, however the gradle plugin seems to be picking up my system python instead (I’ve confirmed pyenv works on the terminal)
u
I’m running into this because 
pyarrow
is failing to install for me, and I suspect it’s because Gradle is ignoring pyenv and picking up my Macs system’s version of 
3.8
while our current is 
3.7.9
u
does seem our python plugin looks at the system python - https://github.com/xvik/gradle-use-python-plugin/blob/master/src/main/groovy/ru/vyarus/gradle/plugin/python/PythonExtension.groovy#L19 although I haven’t seen a good reason why/how it’s ignoring my 
pyenv
shim
u
personally, I do use pyenv on my mac
u
Good to know. Is it picking up the right version? Also, have you seen this PyArrow error before Chris?
Copy code
cOS version (11.0) than being linked (10.9)
           ld: warning: dylib (/usr/local/Cellar/apache-arrow/3.0.0_3/lib/libarrow.dylib) was built for newer macOS version (11.0) than being linked (10.9)
           [100%] Built target _fs
           -- Finished cmake --build for pyarrow
           Bundling includes: release/include
           Moving built C-extension release/lib.cpython-38-darwin.so to build path /private/var/folders/4h/k452x8ns06q7w_w7wkb3j16h0000gn/T/pip-install-t1413mpq/pyarrow/build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/lib.cpython-38-darwin.so
           Moving built C-extension release/_fs.cpython-38-darwin.so to build path /private/var/folders/4h/k452x8ns06q7w_w7wkb3j16h0000gn/T/pip-install-t1413mpq/pyarrow/build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_fs.cpython-38-darwin.so
           Moving built C-extension release/_csv.cpython-38-darwin.so to build path /private/var/folders/4h/k452x8ns06q7w_w7wkb3j16h0000gn/T/pip-install-t1413mpq/pyarrow/build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_csv.cpython-38-darwin.so
           Moving built C-extension release/_json.cpython-38-darwin.so to build path /private/var/folders/4h/k452x8ns06q7w_w7wkb3j16h0000gn/T/pip-install-t1413mpq/pyarrow/build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_json.cpython-38-darwin.so
           Moving built C-extension release/_compute.cpython-38-darwin.so to build path /private/var/folders/4h/k452x8ns06q7w_w7wkb3j16h0000gn/T/pip-install-t1413mpq/pyarrow/build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_compute.cpython-38-darwin.so
           Did not find release/_cuda.cpython-38-darwin.so
           Cython module _cuda failure permitted
           Did not find release/_flight.cpython-38-darwin.so
           Cython module _flight failure permitted
           Did not find release/_dataset.cpython-38-darwin.so
           Cython module _dataset failure permitted
           Did not find release/_parquet.cpython-38-darwin.so
           Cython module _parquet failure permitted
           Did not find release/_orc.cpython-38-darwin.so
           Cython module _orc failure permitted
           Did not find release/_plasma.cpython-38-darwin.so
           Cython module _plasma failure permitted
           Did not find release/_s3fs.cpython-38-darwin.so
           Cython module _s3fs failure permitted
           Did not find release/_hdfs.cpython-38-darwin.so
           Cython module _hdfs failure permitted
           Did not find release/gandiva.cpython-38-darwin.so
           Cython module gandiva failure permitted
           installing to build/bdist.macosx-10.14.6-x86_64/wheel
           running install
           running install_lib
           creating build/bdist.macosx-10.14.6-x86_64
           creating build/bdist.macosx-10.14.6-x86_64/wheel
           creating build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/public-api.pxi -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/io.pxi -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/serialization.pxi -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/orc.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_compute.pyx -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_orc.pxd -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/builder.pxi -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/array.pxi -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           ld: warning: dylib (/usr/local/Cellar/apache-arrow/3.0.0_3/lib/libarrow.dylib) was built for newer macOS version (11.0) than being linked (10.9)  Building wheel for bcrypt (PEP 517): started
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_generated_version.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/compat.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_cuda.pyx -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/benchmark.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/parquet.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_csv.pyx -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/config.pxi -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_dataset.pyx -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_flight.pyx -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/ipc.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/util.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/flight.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/tensor.pxi -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/__init__.pxd -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/cffi.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
           creating build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow/include
           error: can't copy 'build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/include/arrow': doesn't exist or not a regular file
           ----------------------------------------
           ERROR: Failed building wheel for pyarrow
u
i didn’t personally run into it but it does remind me of others encountering that…. https://airbytehq.slack.com/archives/C01AHCDHCKE/p1612277222072500
u
Installing your python & java on a fresh mac is quite difficult to get it right… it’d probably be a good thing to take notes for the next ones going through it…
u
Definitely taking notes!
u
Chris can ya do me a favor and give me a peek into one of your 
.venv
folders from a module that contains Python whenever convenient? Something like
Copy code
➜  source-file git:(master) cd .venv
(⎈ |gke_causal-sky-277804_us-central1_autopilot-cluster-1:default)
➜  .venv git:(master) ls -al
total 0
drwxr-xr-x   6 davinchia  staff  192 Mar  2 16:37 .
drwxr-xr-x  15 davinchia  staff  480 Mar  2 16:37 ..
lrwxr-xr-x   1 davinchia  staff   84 Mar  2 16:37 .Python -> /usr/local/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/Python
drwxr-xr-x   5 davinchia  staff  160 Mar  2 16:37 bin
drwxr-xr-x   3 davinchia  staff   96 Mar  2 16:37 include
drwxr-xr-x   3 davinchia  staff   96 Mar  2 16:37 lib
I’m curious where your python is pointed to (this is wrong)
u
i have this in my .zshrc: it might be related
Copy code
alias brew='env PATH=${PATH//$(pyenv root)\/shims:/} brew'
u
mmmh i think i remember something about not doing symlinks in my venv yes… here is my `airbyte/airbyte-integrations/connectors/source-file/.venv`:
Copy code
ls -al
total 0
drwxr-xr-x   5 chris  staff   160 Feb 25 18:24 .
drwxr-xr-x  20 chris  staff   640 Mar  1 16:53 ..
drwxr-xr-x  53 chris  staff  1696 Feb 25 18:28 bin
drwxr-xr-x   3 chris  staff    96 Feb 25 18:24 include
drwxr-xr-x   3 chris  staff    96 Feb 25 18:24 lib
u
Copy code
ls -al bin 
total 6840
drwxr-xr-x  53 chris  staff     1696 Feb 25 18:28 .
drwxr-xr-x   5 chris  staff      160 Feb 25 18:24 ..
drwxr-xr-x   5 chris  staff      160 Feb 25 18:28 __pycache__
-rw-r--r--   1 chris  staff     2269 Feb 25 18:24 activate
-rw-r--r--   1 chris  staff     1488 Feb 25 18:24 activate.csh
-rw-r--r--   1 chris  staff     3153 Feb 25 18:24 activate.fish
-rw-r--r--   1 chris  staff     1751 Feb 25 18:24 activate.ps1
-rw-r--r--   1 chris  staff     1210 Feb 25 18:24 activate.xsh
-rw-r--r--   1 chris  staff     1517 Feb 25 18:24 activate_this.py
-rwxr-xr-x   1 chris  staff     1072 Feb 25 18:26 airbyte-python-test
-rwxr-xr-x   1 chris  staff     1032 Feb 25 18:26 base-python
-rwxr-xr-x   1 chris  staff      295 Feb 25 18:24 black
-rwxr-xr-x   1 chris  staff      290 Feb 25 18:24 black-primer
-rwxr-xr-x   1 chris  staff      296 Feb 25 18:24 blackd
-rwxr-xr-x   1 chris  staff      296 Feb 25 18:26 chardetect
-rwxr-xr-x   1 chris  staff      280 Feb 25 18:28 distro
-rwxr-xr-x   1 chris  staff      309 Feb 25 18:24 dmypy
-rwxr-xr-x   1 chris  staff      290 Feb 25 18:28 docker-compose
-rwxr-xr-x   1 chris  staff      282 Feb 25 18:28 dotenv
-rwxr-xr-x   1 chris  staff      291 Feb 25 18:26 f2py
-rwxr-xr-x   1 chris  staff      291 Feb 25 18:26 f2py3
-rwxr-xr-x   1 chris  staff      291 Feb 25 18:26 f2py3.7
-rwxr-xr-x   1 chris  staff      289 Feb 25 18:24 flake8
-rwxr-xr-x   1 chris  staff      284 Feb 25 18:26 genson
-rwxr-xr-x   1 chris  staff      308 Feb 25 18:26 google-oauthlib-tool
-rwxr-xr-x   1 chris  staff      284 Feb 25 18:24 isort
-rwxr-xr-x   1 chris  staff     1770 Feb 25 18:26 jp.py
-rwxr-xr-x   1 chris  staff      288 Feb 25 18:26 jsonschema
-rwxr-xr-x   1 chris  staff      305 Feb 25 18:24 mypy
-rwxr-xr-x   1 chris  staff     1506 Feb 25 18:24 mypyc
-rwxr-xr-x   1 chris  staff      296 Feb 25 18:25 pip
-rwxr-xr-x   1 chris  staff      296 Feb 25 18:25 pip3
-rwxr-xr-x   1 chris  staff      296 Feb 25 18:25 pip3.7
-rwxr-xr-x   1 chris  staff      323 Feb 25 18:26 plasma_store
-rwxr-xr-x   1 chris  staff      296 Feb 25 18:25 py.test
-rwxr-xr-x   1 chris  staff      287 Feb 25 18:24 pycodestyle
-rwxr-xr-x   1 chris  staff      286 Feb 25 18:24 pyflakes
-rwxr-xr-x   1 chris  staff      287 Feb 25 18:26 pyrsa-decrypt
-rwxr-xr-x   1 chris  staff      287 Feb 25 18:26 pyrsa-encrypt
-rwxr-xr-x   1 chris  staff      285 Feb 25 18:26 pyrsa-keygen
-rwxr-xr-x   1 chris  staff      308 Feb 25 18:26 pyrsa-priv2pub
-rwxr-xr-x   1 chris  staff      281 Feb 25 18:26 pyrsa-sign
-rwxr-xr-x   1 chris  staff      285 Feb 25 18:26 pyrsa-verify
-rwxr-xr-x   1 chris  staff      296 Feb 25 18:25 pytest
lrwxr-xr-x   1 chris  staff        7 Feb 25 18:24 python -> python3
-rwxr-xr-x   1 chris  staff     2398 Feb 25 18:24 python-config
-rwxr-xr-x   1 chris  staff  3292600 Feb 25 18:24 python3
lrwxr-xr-x   1 chris  staff        7 Feb 25 18:24 python3.7 -> python3
-rwxr-xr-x   1 chris  staff    16145 Feb 25 18:26 runxlrd.py
-rwxr-xr-x   1 chris  staff      286 Feb 25 18:24 stubgen
-rwxr-xr-x   1 chris  staff      287 Feb 25 18:24 stubtest
-rwxr-xr-x   1 chris  staff      284 Feb 25 18:24 wheel
-rwxr-xr-x   1 chris  staff     6488 Feb 25 18:28 wsdump.py
u
i remember something about not doing symlinks in my venv yes
promising! looks like you are on python 3.7 yes?
u
yes
u
alias brew=‘env PATH=${PATH//$(pyenv root)\/shims:/} brew’
probably coming from https://github.com/pyenv/pyenv/issues/106
u
awesome. you have given me better leads to chase down
u
the most annoying thing is how long each install attempt takes..
u
indeed
u
our build is horrible
u
need an excuse to play ping pong 😛
u
do you have 
apache-arrow
and/or 
apache-arrow-glib
installed? if so, what versions are they?
u
is that installed through pip?
u
I believe it’s through brew
u
not installed no
u
huh I thought I needed to in order to install the Arrow C++ libraries: https://arrow.apache.org/install/
u
i do get pyarrow through pip though 
pyarrow==3.0.0
u
seems that’s only the binary so probably easier to get it working
u
my 
pip install pyarrow
runs into the following error without 
brew install apache-arrow
Copy code
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2")
  -- Could NOT find Arrow (missing: Arrow_DIR)
  -- Checking for module 'arrow'
  --   No package 'arrow' found
  CMake Error at /usr/local/Cellar/cmake/3.19.6/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:218 (message):
    Could NOT find Arrow (missing: ARROW_INCLUDE_DIR ARROW_LIB_DIR
    ARROW_FULL_SO_VERSION ARROW_SO_VERSION)
  Call Stack (most recent call first):
    /usr/local/Cellar/cmake/3.19.6/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:582 (_FPHSA_FAILURE_MESSAGE)
    cmake_modules/FindArrow.cmake:419 (find_package_handle_standard_args)
    cmake_modules/FindArrowPython.cmake:46 (find_package)
    CMakeLists.txt:214 (find_package)
u
i think i might have avoided having python through brew
u
was everything via pyenv and pip (including numpy)?
u
yes i installed pyenv with brew and then have everything managed through my pyenv
u
Did you manage to get it working? 🙂
u
Sadly no.. since it’s not really blocking for now, I threw it over the fence to Arrow: https://issues.apache.org/jira/browse/ARROW-11835
u
(cheating and am excluding that module from my gradle for the time being 🙈)
u
this makes me wonder if figuring out a way to exclude pyarrow is worth it
u
I’m curious, how did you install java 14 on your mac btw?
u
I used 
jenv
. How did you do so?
u
u
oops I wasn’t clear in my answer. I used 
brew
to install, then 
jenv
to switch versions and manage paths
u
first time coming across sdkman; seems cool
u
i guess they are both related (sdkman, jenv) and do the same thing
7 Views
Open in Slack
PreviousNext
Powered by