https://linen.dev logo
Join Slack
Powered by
I’m trying to build the core project for the first...
# contributing-to-airbyte
j
I’m trying to build the core project for the first time, on OSX 10.14.6. I have openjdk 14, and I think brew gave me python 3.9 when I upgraded other libraries. I also have conda python 3.8 available and in my user PATH as my preferred python. However, gradle isn’t finding a usable python during the 
checkPython
step, and it’s unclear how to control which python it tries to use. It seems unable to see my conda python at all (despite it being in my path), and when it uses the brew python, virtualenv chokes with the following error.
Copy code
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':airbyte-integrations:bases:base-normalization:checkPython'.
> Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (<https://github.com/xvik/gradle-use-python-plugin/issues>). You can disable virtualenv usage with 'python.scope = USER'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================
Is there a way I can control what python is used for building? I’m sure I’m not the only one with a messy python situation and a preference to control which one it finds and tries to use. If this is a version issue, I might be able to pin brew’s version to 3.7 or 3.8 instead of 3.9 (hopefully).
u
I have tried uninstalling all brew packages (literally all) and then retest the gradle run; it fails to find my conda python that’s in my PATH. It fails to find any ‘python3’ at all in that scenario. Once I tell brew to 
brew install python
, it gives me 3.9 immediately, which turns up the virtualenv crash documented above. So that’s a dead end, even though it is pretty much a fresh current install. Trying to install with 
brew install python@3.7
gives me a python install but it isn’t found by the gradle plugin as it’s not the default python in brew. I might be able to forcibly link that one by following brew’s manual instructions.
u
I think 
brew install python@3.7
has worked for most of us? I think that's what we've had to do.
u
@s do you know if anyone with zazmic or global logic (or otherwise) has been able to use python from conda to run / develop on airbyte.
u
@Jenny Brown if you uninstall all python versions (maybe even verify it by running gradle afterwards), install 
pyenv
then install python 3.7 does this problem persist?
u
and AFAIK no one is using conda
u
should we be supporting conda?
u
I will try pyenv.
u
I do think we should support conda, because it’s used heavily in data science and data engineering teams who are using machine learning tools, and these are also often the same teams responsible for data movement and data engineering. Conda does not play nicely with virtualenv though; it’s one or the other, not both on the same system.
u
I want to say it shouldn’t matter whether someone is using python 3.7 from conda or pyenv, meaning we should support that
u
but I don’t know enough about conda (yet 😛 ) to tell you whether that’s a desirable/sane end state
u
but I agree with your intuition Jenny
u
The gradle plugin that checks for python was clearly ignoring my user PATH setting entirely, which is super-unintuitive for doing pretty much anything. Then it didn’t make its search logic transparent and visible on top of that. That’s a nightmare for any devops person trying to set up environments or developer workstations… it’s friction in the adoption process.
u
Copy code
brew install pyenv
pyenv install 3.7.10
# (fresh terminal window in case of path adjustments)
CORE_ONLY=1 ./gradlew build -x :airbyte-webapp:test
still results in python3 not being found at all by the gradle plugin.
u
Copy code
java.io.IOException: Cannot run program "python3" (in directory "/Users/jennybrown/dev/airbyte/airbyte/airbyte-integrations/bases/base-normalization"): error=2, No such file or directory
u
Oh, I missed the 
pyenv init
stuff. With that fixed, it’s found. My build appears to be progressing, now.
u
Glad it’s fixed but this is not a great experience. Created issue https://github.com/airbytehq/airbyte/issues/3970
u
the python gradle plugin allows specifiying minimum python version but not maximum… 🤦🏼‍♂️
Copy code
minPythonVersion = '3.7'
u
Yeah. And 3.9 is just plain broken with virtualenv, so I would definitely want to do maxPythonVersion=‘3.7’ or ‘3.8’, to avoid that.
u
I shouldn’t need to use pyenv at all, if I have a python in my path of a suitable version, no matter whether it’s conda’s python, system python (linux), brew python, or pyenv or virtualenv python… I would expect the gradle plugin to “trust” whatever python I handed it and just try to use that.
u
Or if it has to be opinionated about python environment management, it should be super transparent about what it’s doing and why, so I can make my environment match its expectations.
u
… I got farther in the build. npm complained about a very outdated version of node, and quit. I upgraded node to latest. Then I reran the build and it started crashing on being unable to find python3 again. Baffling.
u
it's not something fun like the second time you ran it from a bash terminal that hasn't been restarted since you installed pyenv. mentioning this because this is 100% something i would do by accident.
u
I checked that. It turns out that the issue is something being exceptionally specific about patch version:
Copy code
$ python --version
pyenv: version `3.7.9' is not installed (set by /Users/jennybrown/dev/airbyte/airbyte/.python-version)
I never mentioned 3.7.9 even once in my manual installs. I either typed 
3.7
or I typed 
3.7.10
but never 9.
u
So something is pinning it.
u
It’s correct that 3.7.9 isn’t installed, because I installed 3.7.10.
u
u
That seems like it.
u
jenny, if you run the build, can you run it with the 
--scan
flag to upload the build report?
u
Do you want that before or after I have python 3.7.9 installed? I just told it to install that so I could keep progressing.
u
after is fine
u
okay.
u
preemptively running with 
--scan
is useful in case anything goes sideways
u
u
It’s still failing at finding python correctly, although I didn’t redo pyenv init after the version update. (Yeah, redoing the pyenv init didn’t solve it.)
u
can you change https://github.com/airbytehq/airbyte/blob/master/buildSrc/src/main/groovy/airbyte-python.gradle#L18 in your local and add
Copy code
pythonBinary = 'python'
u
ok
u
https://gradle.com/s/tj24eptcisi5y
Copy code
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-cdk:python:checkPython'.
> Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (<https://github.com/xvik/gradle-use-python-plugin/issues>). You can disable virtualenv usage with 'python.scope = USER'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
u
Also, just a warning, but in case it’s relevant:
Copy code
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See <https://docs.gradle.org/6.7.1/userguide/command_line_interface.html#sec:command_line_warnings>
u
good call. but that warning shouldn't be relevant.
u
does 
airbyte-cdk/.venv/
exist?
u
No, but 
airbyte-cdk/python/.venv/
does exist. It contains a 
.venv/lib/python3.9/
folder in it, which means we’re mixing python versions and picking up the broken 3.9 stuff.
u
Is there a “build clean” I can use to drop all the old artifacts including venv stuff?
u
If not, I’ll reclone from scratch and retry.
u
from plugin docs: https://github.com/xvik/gradle-use-python-plugin/
IMPORTANT: virtualenv creates local python copy (by default in 
.gradle/python
). Copy is created from global python and later used instead of global python. If you want to change used python version in the environment, then manually remove 
.gradle/python
 so it could be created again (from global python).
u
I think this means we need to manually remove it
u
Yeah… it doesn’t tell me which directory to find .gradle in, though. I’ll have to dig a bit.
u
does 
./gradlew clean
remove 
.gradle
?
u
yeah. so i suspect killing 
airbyte-cdk/python/.venv/
manually and then retrying will help.
u
i don't think it does.
u
It didn’t.
u
you can 
rm .gradle
in the root of the project.
u
But that .gradle has no python directory in it (or binary).
u
I would have to check the .venv directories inside every subfolder-project… find might help me here.
u
Copy code
(base) Jennys-MacBook-Pro:airbyte jennybrown$ rm -rf airbyte-integrations/bases/base-python/.venv/
(base) Jennys-MacBook-Pro:airbyte jennybrown$ rm -rf airbyte-integrations/bases/airbyte-protocol/.venv/
(base) Jennys-MacBook-Pro:airbyte jennybrown$ rm -rf airbyte-integrations/bases/base-normalization/.venv/
(base) Jennys-MacBook-Pro:airbyte jennybrown$ rm -rf airbyte-cdk/python/.venv/
Copy code
$ CORE_ONLY=1 ./gradlew build -x :airbyte-webapp:test --scan
Python (/System/Library/Frameworks/Python.framework/Versions/2.7) verion 2.7.16 does not match minimal required version: 3.7
So now it’s finding the system python but not the pyenv one.
u
from the root of the project, i think you can do 
rm -rf ./**/.venv
u
(for what you were saying about finding .venv dirs)
u
It’s okay, I got them all. Find confirms it.
u
and you 
rm -rf .gradle
in the root of the project too?
u
Tried that just now; same result.
u
It’s end of work day for me. I’m happy to continue this tomorrow when I get back in.
u
👍
u
if by the time i'm online tomorrow it's still not working, let's hop on a zoom call and debug together?
u
I’m all for that. Zoom allows remote control too if you want to try stuff hands-on. Finding python shouldn’t be this difficult.
u
u
added a gradle task to remove this automatically for all projets if needed
u
Jenny when you’re back tomorrow, try running 
./gradlew cleanPythonVenvs
to nuke all of them
92 Views
Open in Slack
PreviousNext
Powered by