Hi team. I try to build the project based on commit

commit 451aa97bfa6406883ca59fb5632d3e4f9cf2cb72 (HEAD -> master, origin/master, origin/HEAD)
Author: Charles <giardina.charles@gmail.com>
Date:   Fri Apr 16 17:13:08 2021 -0700

Java14

siamakhaschemi@localhost airbyte % java -version
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment (build 14.0.1+7)
OpenJDK 64-Bit Server VM (build 14.0.1+7, mixed mode, sharing)

Python 3.7.9 managed by pyenv

siamakhaschemi@localhost airbyte % pyenv version
3.7.9 (set by /Users/siamakhaschemi/Development/ing/airbyte/.python-version)

virtualenv

siamakhaschemi@localhost airbyte % virtualenv --version
13.1.2

When I run gradlew on the top-level of the project, I get an error for pythonenv

siamakhaschemi@localhost airbyte % ./gradlew clean build --stacktrace
[...]
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-integrations:bases:airbyte-protocol:checkPython'.
> Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (<https://github.com/xvik/gradle-use-python-plugin/issues>). You can disable virtualenv usage with 'python.scope = USER'.

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':airbyte-integrations:bases:airbyte-protocol:checkPython'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:200)
        at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:263)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:198)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:179)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:109)
[...]
Caused by: org.gradle.api.GradleException: Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (<https://github.com/xvik/gradle-use-python-plugin/issues>). You can disable virtualenv usage with 'python.scope = USER'.
        at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.checkPython(CheckPythonTask.groovy:82)
        at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.switchEnvironment(CheckPythonTask.groovy:156)
        at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.run(CheckPythonTask.groovy:61)
        at jdk.internal.reflect.GeneratedMethodAccessor1043.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.run(ExecuteActionsTaskExecuter.java:555)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:75)
[...]
Caused by: org.gradle.process.internal.ExecException: A problem occurred starting process 'command '.venv/bin/python''
        at org.gradle.process.internal.DefaultExecHandle.execExceptionFor(DefaultExecHandle.java:241)
        at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:218)
        at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:369)
        at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:87)
        at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
        ... 3 more
Caused by: net.rubygrapefruit.platform.NativeException: Could not start '.venv/bin/python'
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
        at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
        at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:98)
        at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:71)
        ... 4 more
Caused by: java.io.IOException: Cannot run program ".venv/bin/python" (in directory "/Users/siamakhaschemi/Development/ing/airbyte/airbyte-integrations/bases/airbyte-protocol"): error=2, No such file or directory
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
        ... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
        ... 8 more

siamakhaschemi@localhost airbyte % ls airbyte-integrations/bases/airbyte-protocol/.venv
lib

siamakhaschemi@localhost airbyte % ls airbyte-integrations/bases/airbyte-protocol/.venv/lib
python3.9

siamakhaschemi@localhost airbyte % ls airbyte-integrations/bases/airbyte-protocol/.venv/lib/python3.9
config-3.9-darwin lib-dynload

Seems that the virtualenv is not created in a way that it contains a python binary. Do I need to install

pipenv

also? I am new to python, maybe it’s a trivial problem.

After deleting all the

.venv

directories and run again, it seems like the plugin is using another python version on my computer:

> Task :airbyte-integrations:bases:base-python-test:checkPython FAILED
Using python 3.9.4 from /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9 (python3)
Using pip 21.0.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)
Using virtualenv 16.7.9 (in '.venv')
[python] python3 -m virtualenv .venv
         Traceback (most recent call last):
           File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
             return _run_code(code, main_globals, None,
           File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
             exec(code, run_globals)
           File "/Users/siamakhaschemi/Library/Python/3.9/lib/python/site-packages/virtualenv.py", line 2634, in <module>
             main()
           File "/Users/siamakhaschemi/Library/Python/3.9/lib/python/site-packages/virtualenv.py", line 860, in main
             create_environment(
           File "/Users/siamakhaschemi/Library/Python/3.9/lib/python/site-packages/virtualenv.py", line 1162, in create_environment
             install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages=site_packages, clear=clear, symlink=symlink)
           File "/Users/siamakhaschemi/Library/Python/3.9/lib/python/site-packages/virtualenv.py", line 1451, in install_python
             copy_required_modules(home_dir, symlink)
           File "/Users/siamakhaschemi/Library/Python/3.9/lib/python/site-packages/virtualenv.py", line 1341, in copy_required_modules
             dst_filename = change_prefix(filename, dst_prefix)
           File "/Users/siamakhaschemi/Library/Python/3.9/lib/python/site-packages/virtualenv.py", line 1273, in change_prefix
             raise AssertionError("Filename {} does not start with any of these prefixes: {}".format(filename, prefixes))
         Using base prefix '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9'
         Using base prefix '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9'AssertionError: Filename /usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py does not start with any of these prefixes: ['/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/Extras/lib/python', '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9', '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9', '/Users/siamakhaschemi/Library/Python/3.9/lib/python/site-packages', '/Users/siamakhaschemi/.local/lib/python/3.9/site-packages', '/Users/siamakhaschemi/Library/Python/3.9/site-packages', '/Library/Python/3.9/site-packages']

siamakhaschemi@localhost airbyte % python --version
Python 3.7.9

siamakhaschemi@localhost airbyte % pip --version
pip 21.0.1 from /Users/siamakhaschemi/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pip (python 3.7)

siamakhaschemi@localhost airbyte % python3 --version
Python 3.7.9

siamakhaschemi@localhost airbyte % pip3 --version
pip 21.0.1 from /Users/siamakhaschemi/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pip (python 3.7)

Ok, Python is complicated for a Java-guy 🙂 …

Yeap!

It’s because your previous venv was created using a diff python version

Once you reset your venv, the newly created venv matched your current Python version

For some reason the

gradle-use-python-plugin

plugin picks up the wrong python version when it creates the virtual-env. I now created the virtualenvs manually, which seems to solve the issue. So after forking, I had to do:

find . -name ".venv" -exec rm -r "{}" \;

cd airbyte-integrations/connectors/source-google-workspace-admin-reports && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-slack-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-github-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-quickbooks-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-appstore-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-greenhouse && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-zoom-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-salesforce-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-stripe-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-jira && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-google-sheets && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-braintree-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-zendesk-support-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-marketo-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-intercom-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-google-directory && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-instagram && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-scaffold-source-python && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-looker && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-mixpanel-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-googleanalytics-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-exchangeratesapi-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-sendgrid && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-file && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-twilio-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-microsoft-teams && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-google-adwords-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-http-request && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-appsflyer-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-smartsheets && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-mailchimp && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-tempo && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-recurly && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-zendesk-talk && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-freshdesk && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-drift && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-gitlab-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-hubspot && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-facebook-marketing && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-shopify-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/bases/base-python && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/bases/base-python-test && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/bases/airbyte-protocol && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/bases/base-normalization && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/bases/base-singer && python3 -m virtualenv .venv && cd -

It now works (not very elegant, but I want to move forward), except failing tests in airbyte-webapp, which is unrelated to this python problem.

Is your pyenv shim loaded into your path?

If it works it works!

Curious, what are you trying to do?

I plan to write a Java-based connector for data from these two sources: • https://awin.com (HTTP API with 3-4 endpoints, XML response) • https://www.financeads.com (HTTP API returning two csv datasets) We plan to use Airbyte to push this data to Bigquery, where we continue using dbt for further transformation. We initially want to keep the connectors for private use, until we share them. I hope there is a way in Airbyte to specify our own Docker-Repository (GCP) for our custom connector.

Is your pyenv shim loaded into your path?

Let me check what that means. Sorry, first Python day …

That’s alright. If you are planning to write a java based connector you don’t need to build the python modules

Yes there is a way to add your own connectors!

I’m on my phone now. I will share this later

We welcome your contributions 😊

Ok, thank you for the hint (no python needed).