https://linen.dev logo
Join Slack
Powered by
Guys, what is the best way to manage python versio...
# contributing-to-airbyte
a
Guys, what is the best way to manage python versions? Default version for 
python
command in macos is 2.7, however according the docs:
Copy code
We assume `python` points to python >=3.7.
as a result build is failing ( however it was always built fine before ). What is a proper way to make it to be built with python3? • just create alias in .zshrc • pyenv • some other tool?
u
the simplest for me was to uninstall all the default python binaries and use pyenv
u
it just started failing today?
u
well probably I did not rebuild it for a quite long period 🙂
u
i think i am the reason
a
u
this assumption started recently
u
lmk if you’re still having issues artem
u
sherif what about this commit should have caused this?
a
is it because we added 
pythonBinary = 'python'
?
u
or some other line in there.
u
ah. i'm getting a python version issue now too.
u
Copy code
* What went wrong:
Execution failed for task ':airbyte-integrations:bases:base-normalization:checkPython'.
> Python (/System/Library/Frameworks/Python.framework/Versions/2.7) verion 2.7.16 does not match minimal required version: 3.7
u
pythonBinary
u
removing
a
well yeah, it’s not working for me 🙂 I tried to alias it via .zshrc and 
python --version
prints 
Python 3.9.0
but when I run it - it still displays issue with version
u
u
well my guess is that 
python
on my computer is python 2.whatever but python3 is python 3.7 so adding the pyton binary points it to 2 by accident?
a
Its the default for all macos - python is 2 and it is in 
usr/bin/python
and the one installed via brew is in 
usr/local/bin
but I do not think that removing the built-in one is a good idea
u
i'm seeing a new error now
u
Copy code
> Task :airbyte-integrations:bases:base-normalization:unitTest FAILED
[python] .venv/bin/python -m pytest -s unit_tests
         ============================= test session starts ==============================
         platform darwin -- Python 3.7.9, pytest-6.1.2, py-1.10.0, pluggy-0.13.1
         rootdir: /Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization
         collected 0 items / 4 errors

         ==================================== ERRORS ====================================
         _______ ERROR collecting unit_tests/test_destination_name_transformer.py _______
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_destination_name_transformer.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:151: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_destination_name_transformer.py:29: in <module>
             from normalization.destination_type import DestinationType
         normalization/__init__.py:2: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/__init__.py:1: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/transform.py:30: in <module>
             import yaml
         E   ModuleNotFoundError: No module named 'yaml'
         _____________ ERROR collecting unit_tests/test_stream_processor.py _____________
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_stream_processor.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:151: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_stream_processor.py:30: in <module>
             from airbyte_protocol.models.airbyte_protocol import DestinationSyncMode, SyncMode
         E   ModuleNotFoundError: No module named 'airbyte_protocol'
         ___________ ERROR collecting unit_tests/test_table_name_registry.py ____________
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_table_name_registry.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:151: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_table_name_registry.py:32: in <module>
             from normalization.transform_catalog.catalog_processor import CatalogProcessor
         normalization/__init__.py:2: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/__init__.py:1: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/transform.py:30: in <module>
             import yaml
         E   ModuleNotFoundError: No module named 'yaml'
         _____________ ERROR collecting unit_tests/test_transform_config.py _____________
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_transform_config.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:151: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_transform_config.py:30: in <module>
             from normalization.transform_catalog.transform import extract_schema
         normalization/__init__.py:2: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/__init__.py:1: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/transform.py:30: in <module>
             import yaml
         E   ModuleNotFoundError: No module named 'yaml'
         =========================== short test summary info ============================
         ERROR unit_tests/test_destination_name_transformer.py
         ERROR unit_tests/test_stream_processor.py
         ERROR unit_tests/test_table_name_registry.py
         ERROR unit_tests/test_transform_config.py
         !!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!
         ============================== 4 errors in 0.19s ===============================

FAILURE: Build failed with an exception.
u
i think i've diagnosed it.
a
Copy code
./gradlew :airbyte-integrations:bases:base-normalization:build
./gradlew :airbyte-integrations:bases:base-normalization:cleanPythonVenv
./gradlew :airbyte-integrations:bases:base-normalization:build
this sequence of commands fails on the second build.
u
this sequence is okay.
Copy code
./gradlew :airbyte-integrations:bases:base-normalization:build
./gradlew :airbyte-integrations:bases:base-normalization:cleanPythonVenv
./gradlew :airbyte-integrations:bases:base-normalization:clean
./gradlew :airbyte-integrations:bases:base-normalization:build
u
my guess is that running 
cleanPythonVenv
by itself is not really a safe operation and it always has to be paired with a clean. @s do you think we should codify that in the gradle setup? or should we just be documenting it that they have to be used together.
u
I’d prefer that if they’re strictly required to happen together, codify it. Don’t rely on human variance in accurate steps if automation can make it right.
a
I think it’s because the clean venv task has to also get rid of other secondary python artifacts like the .eggs and what not
u
clean is very sane to do here
u
Fix worked - @s thank you for it!
u
now I have another error:
Copy code
> Task :airbyte-integrations:connectors:source-amazon-seller-partner:unitTest FAILED
         unit_tests/test_source.py {"type": "LOG", "log": {"level": "INFO", "message": "Checking access to Amazon SP-API"}}
         .

         =================================== FAILURES ===================================
         ____________________________ test_get_cursor_state _____________________________

             def test_get_cursor_state():
                 cursor_value = "2021-05-02"
                 end_date = "2021-06-02"

                 base_client = get_base_client(SP_CREDENTIALS)

         >       assert "2021-05-03" == base_client._get_cursor_state(cursor_value=cursor_value, end_date=end_date)
         E       AssertionError: assert '2021-05-03' == '2021-06-03'
Am I the only 1 who has this error
p
even though I am running ./gradlew clean build -x test
u
this seems like a legitimate error
u
can you build with 
CORE_ONLY=1
u
this is a problem with a connector which I don’t think you’re trying to build
u
I tried to rebuild it without core_only because airbyte_temporal was not starting with 60sec timeout exception, so I decided that total rebuilt could help
u
building with core only should still rebuild temporal
p
but this is the error I get with temporal
Copy code
ERROR: for airbyte-temporal  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for airbyte-temporal  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
u
the command I am using is 
CORE_ONLY=true ./gradlew --no-daemon build --scan -x test
u
worked now. Strange…
u
is it possible that it was not starting due to some non up-to-date info stored in volumes?
3 Views
Open in Slack
PreviousNext
Powered by