Hi team. I try to build the project based on commi...
# contributing-to-airbyte
Hi team. I try to build the project based on commit
Copy code
commit 451aa97bfa6406883ca59fb5632d3e4f9cf2cb72 (HEAD -> master, origin/master, origin/HEAD)
Author: Charles <giardina.charles@gmail.com>
Date:   Fri Apr 16 17:13:08 2021 -0700
Copy code
siamakhaschemi@localhost airbyte % java -version
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment (build 14.0.1+7)
OpenJDK 64-Bit Server VM (build 14.0.1+7, mixed mode, sharing)
Python 3.7.9 managed by pyenv
Copy code
siamakhaschemi@localhost airbyte % pyenv version
3.7.9 (set by /Users/siamakhaschemi/Development/ing/airbyte/.python-version)
Copy code
siamakhaschemi@localhost airbyte % virtualenv --version
When I run gradlew on the top-level of the project, I get an error for pythonenv
Copy code
siamakhaschemi@localhost airbyte % ./gradlew clean build --stacktrace
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-integrations:bases:airbyte-protocol:checkPython'.
> Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (<https://github.com/xvik/gradle-use-python-plugin/issues>). You can disable virtualenv usage with 'python.scope = USER'.

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':airbyte-integrations:bases:airbyte-protocol:checkPython'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:200)
        at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:263)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:198)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:179)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:109)
Caused by: org.gradle.api.GradleException: Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (<https://github.com/xvik/gradle-use-python-plugin/issues>). You can disable virtualenv usage with 'python.scope = USER'.
        at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.checkPython(CheckPythonTask.groovy:82)
        at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.switchEnvironment(CheckPythonTask.groovy:156)
        at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.run(CheckPythonTask.groovy:61)
        at jdk.internal.reflect.GeneratedMethodAccessor1043.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.run(ExecuteActionsTaskExecuter.java:555)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:75)
Caused by: org.gradle.process.internal.ExecException: A problem occurred starting process 'command '.venv/bin/python''
        at org.gradle.process.internal.DefaultExecHandle.execExceptionFor(DefaultExecHandle.java:241)
        at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:218)
        at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:369)
        at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:87)
        at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
        ... 3 more
Caused by: net.rubygrapefruit.platform.NativeException: Could not start '.venv/bin/python'
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
        at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
        at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:98)
        at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:71)
        ... 4 more
Caused by: java.io.IOException: Cannot run program ".venv/bin/python" (in directory "/Users/siamakhaschemi/Development/ing/airbyte/airbyte-integrations/bases/airbyte-protocol"): error=2, No such file or directory
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
        ... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
        ... 8 more
Copy code
siamakhaschemi@localhost airbyte % ls airbyte-integrations/bases/airbyte-protocol/.venv

siamakhaschemi@localhost airbyte % ls airbyte-integrations/bases/airbyte-protocol/.venv/lib

siamakhaschemi@localhost airbyte % ls airbyte-integrations/bases/airbyte-protocol/.venv/lib/python3.9
config-3.9-darwin lib-dynload
Seems that the virtualenv is not created in a way that it contains a python binary. Do I need to install
also? I am new to python, maybe it’s a trivial problem.
After deleting all the
directories and run again, it seems like the plugin is using another python version on my computer:
Copy code
> Task :airbyte-integrations:bases:base-python-test:checkPython FAILED
Using python 3.9.4 from /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9 (python3)
Using pip 21.0.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)
Using virtualenv 16.7.9 (in '.venv')
[python] python3 -m virtualenv .venv
         Traceback (most recent call last):
           File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
             return _run_code(code, main_globals, None,
           File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
             exec(code, run_globals)
           File "/Users/siamakhaschemi/Library/Python/3.9/lib/python/site-packages/virtualenv.py", line 2634, in <module>
           File "/Users/siamakhaschemi/Library/Python/3.9/lib/python/site-packages/virtualenv.py", line 860, in main
           File "/Users/siamakhaschemi/Library/Python/3.9/lib/python/site-packages/virtualenv.py", line 1162, in create_environment
             install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages=site_packages, clear=clear, symlink=symlink)
           File "/Users/siamakhaschemi/Library/Python/3.9/lib/python/site-packages/virtualenv.py", line 1451, in install_python
             copy_required_modules(home_dir, symlink)
           File "/Users/siamakhaschemi/Library/Python/3.9/lib/python/site-packages/virtualenv.py", line 1341, in copy_required_modules
             dst_filename = change_prefix(filename, dst_prefix)
           File "/Users/siamakhaschemi/Library/Python/3.9/lib/python/site-packages/virtualenv.py", line 1273, in change_prefix
             raise AssertionError("Filename {} does not start with any of these prefixes: {}".format(filename, prefixes))
         Using base prefix '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9'
         Using base prefix '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9'AssertionError: Filename /usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py does not start with any of these prefixes: ['/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/Extras/lib/python', '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9', '/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9', '/Users/siamakhaschemi/Library/Python/3.9/lib/python/site-packages', '/Users/siamakhaschemi/.local/lib/python/3.9/site-packages', '/Users/siamakhaschemi/Library/Python/3.9/site-packages', '/Library/Python/3.9/site-packages']
Copy code
siamakhaschemi@localhost airbyte % python --version
Python 3.7.9

siamakhaschemi@localhost airbyte % pip --version
pip 21.0.1 from /Users/siamakhaschemi/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pip (python 3.7)

siamakhaschemi@localhost airbyte % python3 --version
Python 3.7.9

siamakhaschemi@localhost airbyte % pip3 --version
pip 21.0.1 from /Users/siamakhaschemi/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pip (python 3.7)
Ok, Python is complicated for a Java-guy 🙂
It’s because your previous venv was created using a diff python version
Once you reset your venv, the newly created venv matched your current Python version
For some reason the
plugin picks up the wrong python version when it creates the virtual-env. I now created the virtualenvs manually, which seems to solve the issue. So after forking, I had to do:
Copy code
find . -name ".venv" -exec rm -r "{}" \;

cd airbyte-integrations/connectors/source-google-workspace-admin-reports && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-slack-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-github-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-quickbooks-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-appstore-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-greenhouse && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-zoom-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-salesforce-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-stripe-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-jira && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-google-sheets && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-braintree-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-zendesk-support-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-marketo-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-intercom-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-google-directory && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-instagram && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-scaffold-source-python && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-looker && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-mixpanel-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-googleanalytics-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-exchangeratesapi-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-sendgrid && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-file && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-twilio-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-microsoft-teams && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-google-adwords-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-http-request && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-appsflyer-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-smartsheets && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-mailchimp && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-tempo && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-recurly && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-zendesk-talk && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-freshdesk && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-drift && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-gitlab-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-hubspot && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-facebook-marketing && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/connectors/source-shopify-singer && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/bases/base-python && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/bases/base-python-test && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/bases/airbyte-protocol && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/bases/base-normalization && python3 -m virtualenv .venv && cd -
cd airbyte-integrations/bases/base-singer && python3 -m virtualenv .venv && cd -
It now works (not very elegant, but I want to move forward), except failing tests in airbyte-webapp, which is unrelated to this python problem.
Is your pyenv shim loaded into your path?
If it works it works!
Curious, what are you trying to do?
I plan to write a Java-based connector for data from these two sources: • https://awin.com (HTTP API with 3-4 endpoints, XML response) • https://www.financeads.com (HTTP API returning two csv datasets) We plan to use Airbyte to push this data to Bigquery, where we continue using dbt for further transformation. We initially want to keep the connectors for private use, until we share them. I hope there is a way in Airbyte to specify our own Docker-Repository (GCP) for our custom connector.
Is your pyenv shim loaded into your path?
Let me check what that means. Sorry, first Python day …
That’s alright. If you are planning to write a java based connector you don’t need to build the python modules
Yes there is a way to add your own connectors!
I’m on my phone now. I will share this later
We welcome your contributions 😊
Ok, thank you for the hint (no python needed).