https://linen.dev logo
Join Slack
Powered by
anyone else having trouble compiling from a clean ...
# contributing-to-airbyte
c
anyone else having trouble compiling from a clean install (make sure that your pyenv is starting from scratch, not just 
./gradlew clean
)? failing on task 
:airbyte-integrations:bases:base-normalization:installLocalReqs
. it's a bunch of C level errors when trying to compile pyarrow. I feel like I've seen this before and the answer was just to downgrade the version of the python library we were using. I know something in normalization got upgraded again. Being kinda lazy and asking this question, because last time I dug into this, I lost like a day and a half of my time.
u
a user from community contributed the PR to upgrade it back to .0.19 yes
u
okay. but we haven't fixed the issue that made it not work on our machines?
u
i believe @Davin Chia (Airbyte) was having the same issue.
u
normally you don’t need to compile pyarrow (with the C compiler etc) and just use pyarrow binary from pypi
u
can we revert this until we can make this run our machines?
u
let's revert this until we have a documented workaround.
u
do you mind doing this chris?
u
I would like to try something for this
u
the latest Pip actually builds fine and we've held off upgrading to it because of long resolving time - I want to give upgrading a shot based on the comments in this issue. TLDR: seemed like it's decently stable for now
u
let's do it on a branch.
u
sorry. my tolerance for debugging this again is super low.
u
yeah ofc
u
is Marcos running into this?
u
dunno.
u
i'll revert it.
u
wait to spend a few mins and see if the latest pip version helps? I'm throwing up a branch now
u
i can't merge code right now...
u
!!! this is the highest priority ticket for me
u
ok
u
u
running it now
u
I'm trying and it's going to fail on other stuff because our python deps are a mess
u
ugh. i can't test it because this airbyte-cli stuff is failing.
u
let me kill it
u
Charles can you pull once more before testing. Added some stuff that will help with failures
u
IIRC upgrading PIP made some dependency resolution stuff break
u
never fear Davin is here
u
It looks like the hash for airbyte-cli was deleted from their repo?
u
I’ll fix the airbyte-cli thing
u
yeaH
u
thanks, jared
u
jk. I did run into some errors and it was because we actually had incompatible python versions specified
u
this is why go is awful @Davin Chia (Airbyte) 🙂
u
the old Pip resolver just discarded it
u
davin. your branch seems to build successfully on my machine.
u
I have come to the conclusion that everything (except this awesome team) is awful, and some things are less awful than others 😢
u
davin. your branch seems to build successfully on my machine.
was this with the latest pull?
u
no.
u
i'll pull again.
u
Jared, what's this CLI thing? tag me on the PR
u
nope. newlly pulled fails
u
Copy code
> Task :airbyte-integrations:bases:base-normalization:unitTest FAILED
[python] .venv/bin/python -m pytest unit_tests
         ============================= test session starts ==============================
         platform darwin -- Python 3.7.9, pytest-6.1.2, py-1.10.0, pluggy-0.13.1
         rootdir: /Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization
         collected 0 items / 3 errors

         ==================================== ERRORS ====================================
         _______ ERROR collecting unit_tests/test_destination_name_transformer.py _______
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_destination_name_transformer.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:127: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_destination_name_transformer.py:25: in <module>
             from normalization.destination_type import DestinationType
         normalization/__init__.py:2: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/__init__.py:1: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/transform.py:28: in <module>
             import yaml
         E   ModuleNotFoundError: No module named 'yaml'
         _____________ ERROR collecting unit_tests/test_stream_processor.py _____________
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_stream_processor.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:127: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_stream_processor.py:29: in <module>
             from airbyte_protocol.models.airbyte_protocol import DestinationSyncMode, SyncMode
         E   ModuleNotFoundError: No module named 'airbyte_protocol'
         _____________ ERROR collecting unit_tests/test_transform_config.py _____________
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_transform_config.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:127: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_transform_config.py:27: in <module>
             from normalization.transform_catalog.transform import extract_schema
         normalization/__init__.py:2: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/__init__.py:1: in <module>
             from normalization.transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/transform.py:28: in <module>
             import yaml
         E   ModuleNotFoundError: No module named 'yaml'
         =========================== short test summary info ============================
         ERROR unit_tests/test_destination_name_transformer.py
         ERROR unit_tests/test_stream_processor.py
         ERROR unit_tests/test_transform_config.py
         !!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!!
         ============================== 3 errors in 0.30s ===============================
u
hmm that passes for me - you don't have other changes in your local?
u
no, i do not.
u
my only local change is comment out airbyte-cli from settings.gradle.
u
are you removing all of you pyenvs before running the command?
u
nope - did you?
u
yes.
u
that is how you have to test this, because 
./gradlew clean
does not clear pyenvs.
u
got it. do you have a handy command to do so?
u
here's what i'm running to clear out the pyenvs in bases:
Copy code
git checkout davinchia/upgrade-pip
./gradlew clean
rm -rf airbyte-integrations/bases
git checkout .
u
i should probably do that for 
connectors/
too but i'm only worried about the core build for now.
u
makes sense
u
should 
./gradlew clean
handle this in the future? I'm trying this locally too
u
We should probably bite the bullet and do it.
u
yeap some error as you charles
u
We held off in the past because if you accidentally do this you have to wait like 40 min for all of the venvs
u
oh. i guess there's this command for killing all venvs 
rm -rf airbyte-integrations/**/.venv/
u
yeah. we probably should.
u
actually i have no opinion on whether we should or shouldn't
u
seems like there's more to debug here. can i push the dbt downgrade for now and we can keep working on this off of master?
u
@Jared Rhizor (Airbyte) tag me on the cli thing when it's ready i can rubberstamp
u
yeah. let's unblock you for now
u
u
done
u
thanks
Open in Slack
PreviousNext
Powered by