Davin Chia (Airbyte)
pyenv
, however the gradle plugin seems to be picking up my system python instead (I’ve confirmed pyenv works on the terminal)user
03/02/2021, 8:21 AMpyarrow
is failing to install for me, and I suspect it’s because Gradle is ignoring pyenv and picking up my Macs system’s version of 3.8
while our current is 3.7.9
user
03/02/2021, 8:24 AMpyenv
shimuser
03/02/2021, 8:43 AMuser
03/02/2021, 8:45 AMcOS version (11.0) than being linked (10.9)
ld: warning: dylib (/usr/local/Cellar/apache-arrow/3.0.0_3/lib/libarrow.dylib) was built for newer macOS version (11.0) than being linked (10.9)
[100%] Built target _fs
-- Finished cmake --build for pyarrow
Bundling includes: release/include
Moving built C-extension release/lib.cpython-38-darwin.so to build path /private/var/folders/4h/k452x8ns06q7w_w7wkb3j16h0000gn/T/pip-install-t1413mpq/pyarrow/build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/lib.cpython-38-darwin.so
Moving built C-extension release/_fs.cpython-38-darwin.so to build path /private/var/folders/4h/k452x8ns06q7w_w7wkb3j16h0000gn/T/pip-install-t1413mpq/pyarrow/build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_fs.cpython-38-darwin.so
Moving built C-extension release/_csv.cpython-38-darwin.so to build path /private/var/folders/4h/k452x8ns06q7w_w7wkb3j16h0000gn/T/pip-install-t1413mpq/pyarrow/build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_csv.cpython-38-darwin.so
Moving built C-extension release/_json.cpython-38-darwin.so to build path /private/var/folders/4h/k452x8ns06q7w_w7wkb3j16h0000gn/T/pip-install-t1413mpq/pyarrow/build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_json.cpython-38-darwin.so
Moving built C-extension release/_compute.cpython-38-darwin.so to build path /private/var/folders/4h/k452x8ns06q7w_w7wkb3j16h0000gn/T/pip-install-t1413mpq/pyarrow/build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_compute.cpython-38-darwin.so
Did not find release/_cuda.cpython-38-darwin.so
Cython module _cuda failure permitted
Did not find release/_flight.cpython-38-darwin.so
Cython module _flight failure permitted
Did not find release/_dataset.cpython-38-darwin.so
Cython module _dataset failure permitted
Did not find release/_parquet.cpython-38-darwin.so
Cython module _parquet failure permitted
Did not find release/_orc.cpython-38-darwin.so
Cython module _orc failure permitted
Did not find release/_plasma.cpython-38-darwin.so
Cython module _plasma failure permitted
Did not find release/_s3fs.cpython-38-darwin.so
Cython module _s3fs failure permitted
Did not find release/_hdfs.cpython-38-darwin.so
Cython module _hdfs failure permitted
Did not find release/gandiva.cpython-38-darwin.so
Cython module gandiva failure permitted
installing to build/bdist.macosx-10.14.6-x86_64/wheel
running install
running install_lib
creating build/bdist.macosx-10.14.6-x86_64
creating build/bdist.macosx-10.14.6-x86_64/wheel
creating build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/public-api.pxi -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/io.pxi -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/serialization.pxi -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/orc.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_compute.pyx -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_orc.pxd -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/builder.pxi -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/array.pxi -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
ld: warning: dylib (/usr/local/Cellar/apache-arrow/3.0.0_3/lib/libarrow.dylib) was built for newer macOS version (11.0) than being linked (10.9) Building wheel for bcrypt (PEP 517): started
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_generated_version.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/compat.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_cuda.pyx -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/benchmark.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/parquet.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_csv.pyx -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/config.pxi -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_dataset.pyx -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/_flight.pyx -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/ipc.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/util.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/flight.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/tensor.pxi -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/__init__.pxd -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
copying build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/cffi.py -> build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow
creating build/bdist.macosx-10.14.6-x86_64/wheel/pyarrow/include
error: can't copy 'build/lib.macosx-10.14.6-x86_64-3.8/pyarrow/include/arrow': doesn't exist or not a regular file
----------------------------------------
ERROR: Failed building wheel for pyarrow
user
03/02/2021, 8:49 AMuser
03/02/2021, 8:50 AMuser
03/02/2021, 8:50 AMuser
03/02/2021, 8:53 AM.venv
folders from a module that contains Python whenever convenient? Something like
➜ source-file git:(master) cd .venv
(⎈ |gke_causal-sky-277804_us-central1_autopilot-cluster-1:default)
➜ .venv git:(master) ls -al
total 0
drwxr-xr-x 6 davinchia staff 192 Mar 2 16:37 .
drwxr-xr-x 15 davinchia staff 480 Mar 2 16:37 ..
lrwxr-xr-x 1 davinchia staff 84 Mar 2 16:37 .Python -> /usr/local/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/Python
drwxr-xr-x 5 davinchia staff 160 Mar 2 16:37 bin
drwxr-xr-x 3 davinchia staff 96 Mar 2 16:37 include
drwxr-xr-x 3 davinchia staff 96 Mar 2 16:37 lib
I’m curious where your python is pointed to (this is wrong)user
03/02/2021, 8:53 AMalias brew='env PATH=${PATH//$(pyenv root)\/shims:/} brew'
user
03/02/2021, 8:56 AMls -al
total 0
drwxr-xr-x 5 chris staff 160 Feb 25 18:24 .
drwxr-xr-x 20 chris staff 640 Mar 1 16:53 ..
drwxr-xr-x 53 chris staff 1696 Feb 25 18:28 bin
drwxr-xr-x 3 chris staff 96 Feb 25 18:24 include
drwxr-xr-x 3 chris staff 96 Feb 25 18:24 lib
user
03/02/2021, 8:57 AMls -al bin
total 6840
drwxr-xr-x 53 chris staff 1696 Feb 25 18:28 .
drwxr-xr-x 5 chris staff 160 Feb 25 18:24 ..
drwxr-xr-x 5 chris staff 160 Feb 25 18:28 __pycache__
-rw-r--r-- 1 chris staff 2269 Feb 25 18:24 activate
-rw-r--r-- 1 chris staff 1488 Feb 25 18:24 activate.csh
-rw-r--r-- 1 chris staff 3153 Feb 25 18:24 activate.fish
-rw-r--r-- 1 chris staff 1751 Feb 25 18:24 activate.ps1
-rw-r--r-- 1 chris staff 1210 Feb 25 18:24 activate.xsh
-rw-r--r-- 1 chris staff 1517 Feb 25 18:24 activate_this.py
-rwxr-xr-x 1 chris staff 1072 Feb 25 18:26 airbyte-python-test
-rwxr-xr-x 1 chris staff 1032 Feb 25 18:26 base-python
-rwxr-xr-x 1 chris staff 295 Feb 25 18:24 black
-rwxr-xr-x 1 chris staff 290 Feb 25 18:24 black-primer
-rwxr-xr-x 1 chris staff 296 Feb 25 18:24 blackd
-rwxr-xr-x 1 chris staff 296 Feb 25 18:26 chardetect
-rwxr-xr-x 1 chris staff 280 Feb 25 18:28 distro
-rwxr-xr-x 1 chris staff 309 Feb 25 18:24 dmypy
-rwxr-xr-x 1 chris staff 290 Feb 25 18:28 docker-compose
-rwxr-xr-x 1 chris staff 282 Feb 25 18:28 dotenv
-rwxr-xr-x 1 chris staff 291 Feb 25 18:26 f2py
-rwxr-xr-x 1 chris staff 291 Feb 25 18:26 f2py3
-rwxr-xr-x 1 chris staff 291 Feb 25 18:26 f2py3.7
-rwxr-xr-x 1 chris staff 289 Feb 25 18:24 flake8
-rwxr-xr-x 1 chris staff 284 Feb 25 18:26 genson
-rwxr-xr-x 1 chris staff 308 Feb 25 18:26 google-oauthlib-tool
-rwxr-xr-x 1 chris staff 284 Feb 25 18:24 isort
-rwxr-xr-x 1 chris staff 1770 Feb 25 18:26 jp.py
-rwxr-xr-x 1 chris staff 288 Feb 25 18:26 jsonschema
-rwxr-xr-x 1 chris staff 305 Feb 25 18:24 mypy
-rwxr-xr-x 1 chris staff 1506 Feb 25 18:24 mypyc
-rwxr-xr-x 1 chris staff 296 Feb 25 18:25 pip
-rwxr-xr-x 1 chris staff 296 Feb 25 18:25 pip3
-rwxr-xr-x 1 chris staff 296 Feb 25 18:25 pip3.7
-rwxr-xr-x 1 chris staff 323 Feb 25 18:26 plasma_store
-rwxr-xr-x 1 chris staff 296 Feb 25 18:25 py.test
-rwxr-xr-x 1 chris staff 287 Feb 25 18:24 pycodestyle
-rwxr-xr-x 1 chris staff 286 Feb 25 18:24 pyflakes
-rwxr-xr-x 1 chris staff 287 Feb 25 18:26 pyrsa-decrypt
-rwxr-xr-x 1 chris staff 287 Feb 25 18:26 pyrsa-encrypt
-rwxr-xr-x 1 chris staff 285 Feb 25 18:26 pyrsa-keygen
-rwxr-xr-x 1 chris staff 308 Feb 25 18:26 pyrsa-priv2pub
-rwxr-xr-x 1 chris staff 281 Feb 25 18:26 pyrsa-sign
-rwxr-xr-x 1 chris staff 285 Feb 25 18:26 pyrsa-verify
-rwxr-xr-x 1 chris staff 296 Feb 25 18:25 pytest
lrwxr-xr-x 1 chris staff 7 Feb 25 18:24 python -> python3
-rwxr-xr-x 1 chris staff 2398 Feb 25 18:24 python-config
-rwxr-xr-x 1 chris staff 3292600 Feb 25 18:24 python3
lrwxr-xr-x 1 chris staff 7 Feb 25 18:24 python3.7 -> python3
-rwxr-xr-x 1 chris staff 16145 Feb 25 18:26 runxlrd.py
-rwxr-xr-x 1 chris staff 286 Feb 25 18:24 stubgen
-rwxr-xr-x 1 chris staff 287 Feb 25 18:24 stubtest
-rwxr-xr-x 1 chris staff 284 Feb 25 18:24 wheel
-rwxr-xr-x 1 chris staff 6488 Feb 25 18:28 wsdump.py
user
03/02/2021, 8:58 AMi remember something about not doing symlinks in my venv yespromising! looks like you are on python 3.7 yes?
user
03/02/2021, 8:58 AMuser
03/02/2021, 8:58 AMalias brew=‘env PATH=${PATH//$(pyenv root)\/shims:/} brew’probably coming from https://github.com/pyenv/pyenv/issues/106
user
03/02/2021, 9:01 AMuser
03/02/2021, 9:01 AMuser
03/02/2021, 9:01 AMuser
03/02/2021, 9:01 AMuser
03/02/2021, 9:03 AMuser
03/02/2021, 9:03 AMapache-arrow
and/or apache-arrow-glib
installed? if so, what versions are they?user
03/02/2021, 9:05 AMuser
03/02/2021, 9:06 AMuser
03/02/2021, 9:07 AMuser
03/02/2021, 9:08 AMuser
03/02/2021, 9:15 AMpyarrow==3.0.0
user
03/02/2021, 9:16 AMuser
03/02/2021, 9:17 AMpip install pyarrow
runs into the following error without brew install apache-arrow
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2")
-- Could NOT find Arrow (missing: Arrow_DIR)
-- Checking for module 'arrow'
-- No package 'arrow' found
CMake Error at /usr/local/Cellar/cmake/3.19.6/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:218 (message):
Could NOT find Arrow (missing: ARROW_INCLUDE_DIR ARROW_LIB_DIR
ARROW_FULL_SO_VERSION ARROW_SO_VERSION)
Call Stack (most recent call first):
/usr/local/Cellar/cmake/3.19.6/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:582 (_FPHSA_FAILURE_MESSAGE)
cmake_modules/FindArrow.cmake:419 (find_package_handle_standard_args)
cmake_modules/FindArrowPython.cmake:46 (find_package)
CMakeLists.txt:214 (find_package)
user
03/02/2021, 9:22 AMuser
03/02/2021, 9:28 AMuser
03/02/2021, 9:29 AMuser
03/03/2021, 9:10 AMuser
03/03/2021, 9:12 AMuser
03/03/2021, 9:13 AMuser
03/03/2021, 9:13 AMuser
03/03/2021, 9:25 AMuser
03/03/2021, 9:27 AMjenv
. How did you do so?user
03/03/2021, 9:28 AMuser
03/03/2021, 9:31 AMbrew
to install, then jenv
to switch versions and manage pathsuser
03/03/2021, 9:32 AMuser
03/03/2021, 9:43 AM