I m trying to build the core project for the first time on O

Question

I m trying to build the core project for the first time on OSX 10 14 6 I have openjdk 14 and I think brew gave me python 3 9 when I upgraded other libraries I also have conda python 3 8 available and in my user PATH as my preferred python However gradle isn t finding a usable python during the `checkPython` step and it s unclear how to control which python it tries to use It seems unable to see my conda python at all despite it being in my path and when it uses the brew python virtualenv chokes with the following error ```1 Task failed with an exception What went wrong Execution failed for task airbyte integrations bases base normalization checkPython gt Python not found venv bin python This must be a bug of virtualenv support please report it <https github com xvik gradle use python plugin issues> You can disable virtualenv usage with python scope = USER Try Run with stacktrace option to get the stack trace Run with info or debug option to get more log output Run with scan to get full insights ==============================================================================``` Is there a way I can control what python is used for building I m sure I m not the only one with a messy python situation and a preference to control which one it finds and tries to use If this is a version issue I might be able to pin brew s version to 3 7 or 3 8 instead of 3 9 hopefully

user · Answer

I have tried uninstalling all brew packages literally all and then retest the gradle run it fails to find my conda python that s in my PATH It fails to find any python3 at all in that scenario Once I tell brew to `brew install python` it gives me 3 9 immediately which turns up the virtualenv crash documented above So that s a dead end even though it is pretty much a fresh current install Trying to install with `brew install python 3 7` gives me a python install but it isn t found by the gradle plugin as it s not the default python in brew I might be able to forcibly link that one by following brew s manual instructions

user · Answer

I think `brew install python 3 7` has worked for most of us I think that s what we ve had to do

user · Answer

< s> do you know if anyone with zazmic or global logic or otherwise has been able to use python from conda to run develop on airbyte

user · Answer

< Jenny Brown> if you uninstall all python versions maybe even verify it by running gradle afterwards install `pyenv` then install python 3 7 does this problem persist

user · Answer

and AFAIK no one is using conda

user · Answer

should we be supporting conda

user · Answer

I will try pyenv

user · Answer

I do think we should support conda because it s used heavily in data science and data engineering teams who are using machine learning tools and these are also often the same teams responsible for data movement and data engineering Conda does not play nicely with virtualenv though it s one or the other not both on the same system

user · Answer

I want to say it shouldn t matter whether someone is using python 3 7 from conda or pyenv meaning we should support that

user · Answer

but I don t know enough about conda yet stuck out tongue to tell you whether that s a desirable sane end state

user · Answer

but I agree with your intuition Jenny

user · Answer

The gradle plugin that checks for python was clearly ignoring my user PATH setting entirely which is super unintuitive for doing pretty much anything Then it didn t make its search logic transparent and visible on top of that That s a nightmare for any devops person trying to set up environments or developer workstations it s friction in the adoption process

user · Answer

```brew install pyenv pyenv install 3 7 10 fresh terminal window in case of path adjustments CORE ONLY=1 gradlew build x airbyte webapp test``` still results in python3 not being found at all by the gradle plugin

user · Answer

```java io IOException Cannot run program python3 in directory Users jennybrown dev airbyte airbyte airbyte integrations bases base normalization error=2 No such file or directory```

user · Answer

Oh I missed the `pyenv init` stuff With that fixed it s found My build appears to be progressing now

user · Answer

Glad it s fixed but this is not a great experience Created issue <https github com airbytehq airbyte issues 3970>

user · Answer

the python gradle plugin allows specifiying minimum python version but not maximum man facepalming skin tone 3 ``` minPythonVersion = 3 7 ```

user · Answer

Yeah And 3 9 is just plain broken with virtualenv so I would definitely want to do maxPythonVersion= 3 7 or 3 8 to avoid that

user · Answer

I shouldn t need to use pyenv at all if I have a python in my path of a suitable version no matter whether it s conda s python system python linux brew python or pyenv or virtualenv python I would expect the gradle plugin to trust whatever python I handed it and just try to use that

user · Answer

Or if it has to be opinionated about python environment management it should be super transparent about what it s doing and why so I can make my environment match its expectations

user · Answer

I got farther in the build npm complained about a very outdated version of node and quit I upgraded node to latest Then I reran the build and it started crashing on being unable to find python3 again Baffling

user · Answer

it s not something fun like the second time you ran it from a bash terminal that hasn t been restarted since you installed pyenv mentioning this because this is 100 something i would do by accident

user · Answer

I checked that It turns out that the issue is something being exceptionally specific about patch version ```$ python version pyenv version `3 7 9 is not installed set by Users jennybrown dev airbyte airbyte python version ``` I never mentioned 3 7 9 even once in my manual installs I either typed `3 7` or I typed `3 7 10` but never 9

user · Answer

So something is pinning it

user · Answer

It s correct that 3 7 9 isn t installed because I installed 3 7 10

user · Answer

<https github com airbytehq airbyte blob master python version> must be because of this

user · Answer

That seems like it

user · Answer

jenny if you run the build can you run it with the ` scan` flag to upload the build report

user · Answer

Do you want that before or after I have python 3 7 9 installed I just told it to install that so I could keep progressing

user · Answer

after is fine

user · Answer

okay

user · Answer

preemptively running with ` scan` is useful in case anything goes sideways

user · Answer

user · Answer

It s still failing at finding python correctly although I didn t redo pyenv init after the version update Yeah redoing the pyenv init didn t solve it

user · Answer

can you change <https github com airbytehq airbyte blob master buildSrc src main groovy airbyte python gradle L18> in your local and add ```pythonBinary = python ```

user · Answer

ok

user · Answer

```FAILURE Build failed with an exception What went wrong Execution failed for task airbyte cdk python checkPython gt Python not found venv bin python This must be a bug of virtualenv support please report it You can disable virtualenv usage with python scope = USER Try Run with stacktrace option to get the stack trace Run with info or debug option to get more log output Run with scan to get full insights ```

user · Answer

Also just a warning but in case it s relevant ```Deprecated Gradle features were used in this build making it incompatible with Gradle 7 0 Use warning mode all to show the individual deprecation warnings See <https docs gradle org 6 7 1 userguide command line interface html sec command line warnings>```

user · Answer

good call but that warning shouldn t be relevant

user · Answer

does `airbyte cdk venv ` exist

user · Answer

No but `airbyte cdk python venv ` does exist It contains a ` venv lib python3 9 ` folder in it which means we re mixing python versions and picking up the broken 3 9 stuff

user · Answer

Is there a build clean I can use to drop all the old artifacts including venv stuff

user · Answer

If not I ll reclone from scratch and retry

user · Answer

from plugin docs <https github com xvik gradle use python plugin > gt IMPORTANT virtualenv creates local python copy by default in ` gradle python` Copy is created from global python and later used instead of global python If you want to change used python version in the environment then manually remove ` gradle python` so it could be created again from global python

user · Answer

I think this means we need to manually remove it

user · Answer

Yeah it doesn t tell me which directory to find gradle in though I ll have to dig a bit

user · Answer

does ` gradlew clean` remove ` gradle`

user · Answer

yeah so i suspect killing `airbyte cdk python venv ` manually and then retrying will help

user · Answer

i don t think it does

user · Answer

It didn t

user · Answer

you can `rm gradle` in the root of the project

user · Answer

But that gradle has no python directory in it or binary

user · Answer

I would have to check the venv directories inside every subfolder project find might help me here

user · Answer

``` base Jennys MacBook Pro airbyte jennybrown$ rm rf airbyte integrations bases base python venv base Jennys MacBook Pro airbyte jennybrown$ rm rf airbyte integrations bases airbyte protocol venv base Jennys MacBook Pro airbyte jennybrown$ rm rf airbyte integrations bases base normalization venv base Jennys MacBook Pro airbyte jennybrown$ rm rf airbyte cdk python venv ``` ```$ CORE ONLY=1 gradlew build x airbyte webapp test scan Python System Library Frameworks Python framework Versions 2 7 verion 2 7 16 does not match minimal required version 3 7``` So now it s finding the system python but not the pyenv one

user · Answer

from the root of the project i think you can do `rm rf venv`

user · Answer

for what you were saying about finding venv dirs

user · Answer

It s okay I got them all Find confirms it

user · Answer

and you `rm rf gradle` in the root of the project too

user · Answer

Tried that just now same result

user · Answer

It s end of work day for me I m happy to continue this tomorrow when I get back in

user · Answer

+1

user · Answer

if by the time i m online tomorrow it s still not working let s hop on a zoom call and debug together

user · Answer

I m all for that Zoom allows remote control too if you want to try stuff hands on Finding python shouldn t be this difficult

user · Answer

user · Answer

added a gradle task to remove this automatically for all projets if needed

user · Answer

Jenny when you re back tomorrow try running ` gradlew cleanPythonVenvs` to nuke all of them