https://datahubproject.io logo
Join Slack
Powered by
Hello DataHub team! Can anyone help me to troubles...
# ingestion
c
Hello DataHub team! Can anyone help me to troubleshoot an error I'm getting when running metadata ingestion unit tests? (I'd like to set up our CI/CD to run the tests automatically.) Details in thread 🧵
here's what I ran - so I am starting with a fresh venv
Copy code
python3 -m venv venv
venv/bin/python -m pip install --upgrade pip wheel setuptools
source venv/bin/activate
cd metadata-ingestion
pip install -e .
./scripts/codegen.sh
pip install -e '.[dev]'
pytest -m 'not slow' -vv --cov-fail-under 0
here's the stack trace from the error I see, 
No module named 'sqlalchemy.sql.roles'
I get the same error just running 
../gradlew :metadata-ingestion:check
pip show sqlalchemy
gives version 1.3.24
b
Hmm I see the same exact version
But not seeing this issue
always surprised by how finicky python deps can be
➕ 1
Is there an easy way to clear pip's cache?
maybe 
pip cache purge
but no pip expert here
m
Hey @colossal-account-65055, on a fresh checkout (
rm -rf metadata-ingest/venv
), I just did
Copy code
./gradlew :metadata-ingestion:installDev

followed by

./gradlew :metadata-ingestion:check
and it works for me.
We do have CI runs on this on the 
datahub
repo, you can see them here e.g.: (https://github.com/linkedin/datahub/runs/3302754637)
🙌 1
oh interesting @colossal-account-65055: looks like this affected others as well (https://githubmemory.com/repo/snowflakedb/snowflake-sqlalchemy/issues/235)
@colossal-account-65055: can you check 
pip freeze | grep snowflake
?
@colossal-account-65055 we found the underlying problem and just fixed it! Looks like the default 
dev_requirements
were not constraining the snowflake dependency.
🙌 1
can you refresh your dev env and recheck?
👀 1
c
it works!! Thank you very much @mammoth-bear-12532 and @big-carpet-38439 for your expert troubleshooting assist!
🙌 1
😅 1
Is a gradle build needed for running the python unit tests?
b
i usually just run
Copy code
pytest -m 'not integration'

# Run Docker-based integration tests.
pytest -m 'integration'
✅ 1
m
from within the venv ^
2 Views
Open in Slack
PreviousNext
Powered by