https://linen.dev logo
a

Artem Astapenko

06/09/2021, 2:36 PM
Guys, what is the best way to manage python versions? Default version for
python
command in macos is 2.7, however according the docs:
Copy code
We assume `python` points to python >=3.7.
as a result build is failing ( however it was always built fine before ). What is a proper way to make it to be built with python3? • just create alias in .zshrc • pyenv • some other tool?
u

user

06/09/2021, 2:53 PM
the simplest for me was to uninstall all the default python binaries and use pyenv
u

user

06/09/2021, 2:53 PM
it just started failing today?
u

user

06/09/2021, 2:54 PM
well probably I did not rebuild it for a quite long period 🙂
u

user

06/09/2021, 3:06 PM
i think i am the reason
u

user

06/09/2021, 3:07 PM
this assumption started recently
u

user

06/09/2021, 3:07 PM
lmk if you’re still having issues artem
u

user

06/09/2021, 3:42 PM
sherif what about this commit should have caused this?
a

Artem Astapenko

06/09/2021, 3:42 PM
is it because we added
pythonBinary = 'python'
?
u

user

06/09/2021, 3:43 PM
or some other line in there.
u

user

06/09/2021, 3:45 PM
ah. i'm getting a python version issue now too.
u

user

06/09/2021, 3:45 PM
Copy code
* What went wrong:
Execution failed for task ':airbyte-integrations:bases:base-normalization:checkPython'.
> Python (/System/Library/Frameworks/Python.framework/Versions/2.7) verion 2.7.16 does not match minimal required version: 3.7
u

user

06/09/2021, 3:45 PM
pythonBinary
u

user

06/09/2021, 3:45 PM
removing
a

Artem Astapenko

06/09/2021, 3:46 PM
well yeah, it’s not working for me 🙂 I tried to alias it via .zshrc and
python --version
prints
Python 3.9.0
but when I run it - it still displays issue with version
u

user

06/09/2021, 3:47 PM
well my guess is that
python
on my computer is python 2.whatever but python3 is python 3.7 so adding the pyton binary points it to 2 by accident?
a

Artem Astapenko

06/09/2021, 3:48 PM
Its the default for all macos - python is 2 and it is in
usr/bin/python
and the one installed via brew is in
usr/local/bin
but I do not think that removing the built-in one is a good idea
u

user

06/09/2021, 3:49 PM
i'm seeing a new error now
u

user

06/09/2021, 3:49 PM
Copy code
> Task :airbyte-integrations:bases:base-normalization:unitTest FAILED
[python] .venv/bin/python -m pytest -s unit_tests
         ============================= test session starts ==============================
         platform darwin -- Python 3.7.9, pytest-6.1.2, py-1.10.0, pluggy-0.13.1
         rootdir: /Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization
         collected 0 items / 4 errors

         ==================================== ERRORS ====================================
         _______ ERROR collecting unit_tests/test_destination_name_transformer.py _______
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_destination_name_transformer.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:151: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_destination_name_transformer.py:29: in <module>
             from normalization.destination_type import DestinationType
         normalization/__init__.py:2: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/__init__.py:1: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/transform.py:30: in <module>
             import yaml
         E   ModuleNotFoundError: No module named 'yaml'
         _____________ ERROR collecting unit_tests/test_stream_processor.py _____________
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_stream_processor.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:151: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_stream_processor.py:30: in <module>
             from airbyte_protocol.models.airbyte_protocol import DestinationSyncMode, SyncMode
         E   ModuleNotFoundError: No module named 'airbyte_protocol'
         ___________ ERROR collecting unit_tests/test_table_name_registry.py ____________
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_table_name_registry.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:151: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_table_name_registry.py:32: in <module>
             from normalization.transform_catalog.catalog_processor import CatalogProcessor
         normalization/__init__.py:2: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/__init__.py:1: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/transform.py:30: in <module>
             import yaml
         E   ModuleNotFoundError: No module named 'yaml'
         _____________ ERROR collecting unit_tests/test_transform_config.py _____________
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_transform_config.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:151: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_transform_config.py:30: in <module>
             from normalization.transform_catalog.transform import extract_schema
         normalization/__init__.py:2: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/__init__.py:1: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/transform.py:30: in <module>
             import yaml
         E   ModuleNotFoundError: No module named 'yaml'
         =========================== short test summary info ============================
         ERROR unit_tests/test_destination_name_transformer.py
         ERROR unit_tests/test_stream_processor.py
         ERROR unit_tests/test_table_name_registry.py
         ERROR unit_tests/test_transform_config.py
         !!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!
         ============================== 4 errors in 0.19s ===============================

FAILURE: Build failed with an exception.
u

user

06/09/2021, 3:51 PM
i think i've diagnosed it.
a

Artem Astapenko

06/09/2021, 3:52 PM
Copy code
./gradlew :airbyte-integrations:bases:base-normalization:build
./gradlew :airbyte-integrations:bases:base-normalization:cleanPythonVenv
./gradlew :airbyte-integrations:bases:base-normalization:build
this sequence of commands fails on the second build.
u

user

06/09/2021, 3:52 PM
this sequence is okay.
Copy code
./gradlew :airbyte-integrations:bases:base-normalization:build
./gradlew :airbyte-integrations:bases:base-normalization:cleanPythonVenv
./gradlew :airbyte-integrations:bases:base-normalization:clean
./gradlew :airbyte-integrations:bases:base-normalization:build
u

user

06/09/2021, 3:53 PM
my guess is that running
cleanPythonVenv
by itself is not really a safe operation and it always has to be paired with a clean. @s do you think we should codify that in the gradle setup? or should we just be documenting it that they have to be used together.
u

user

06/09/2021, 3:56 PM
I’d prefer that if they’re strictly required to happen together, codify it. Don’t rely on human variance in accurate steps if automation can make it right.
a

Artem Astapenko

06/09/2021, 3:58 PM
I think it’s because the clean venv task has to also get rid of other secondary python artifacts like the .eggs and what not
u

user

06/09/2021, 3:58 PM
clean is very sane to do here
u

user

06/09/2021, 4:23 PM
Fix worked - @s thank you for it!
u

user

06/09/2021, 5:55 PM
now I have another error:
Copy code
> Task :airbyte-integrations:connectors:source-amazon-seller-partner:unitTest FAILED
         unit_tests/test_source.py {"type": "LOG", "log": {"level": "INFO", "message": "Checking access to Amazon SP-API"}}
         .

         =================================== FAILURES ===================================
         ____________________________ test_get_cursor_state _____________________________

             def test_get_cursor_state():
                 cursor_value = "2021-05-02"
                 end_date = "2021-06-02"

                 base_client = get_base_client(SP_CREDENTIALS)

         >       assert "2021-05-03" == base_client._get_cursor_state(cursor_value=cursor_value, end_date=end_date)
         E       AssertionError: assert '2021-05-03' == '2021-06-03'
Am I the only 1 who has this error
p

Ping

06/09/2021, 5:55 PM
even though I am running ./gradlew clean build -x test
u

user

06/09/2021, 5:55 PM
this seems like a legitimate error
u

user

06/09/2021, 5:56 PM
can you build with
CORE_ONLY=1
u

user

06/09/2021, 5:56 PM
this is a problem with a connector which I don’t think you’re trying to build
u

user

06/09/2021, 5:57 PM
I tried to rebuild it without core_only because airbyte_temporal was not starting with 60sec timeout exception, so I decided that total rebuilt could help
u

user

06/09/2021, 5:57 PM
building with core only should still rebuild temporal
p

Ping

06/09/2021, 5:58 PM
but this is the error I get with temporal
Copy code
ERROR: for airbyte-temporal  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for airbyte-temporal  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
u

user

06/09/2021, 5:59 PM
the command I am using is
CORE_ONLY=true ./gradlew --no-daemon build --scan -x test
u

user

06/09/2021, 6:06 PM
worked now. Strange…
u

user

06/09/2021, 6:07 PM
is it possible that it was not starting due to some non up-to-date info stored in volumes?
3 Views