Has anyone found a faster way to run unit tests Running test DataHub #ingestion

Has anyone found a faster way to run unit tests? R...

fancy-sundown-164

11/30/2021, 5:25 PM

Has anyone found a faster way to run unit tests? Running testQuick (

../gradlew :metadata-ingestion:testQuick

) with a modified

build.grade

file to only run the tests I want takes 3-4 minutes, but probably 99% of this is just loading dependencies. Is there some way to cache this step or something between runs?

👀 1

mammoth-bear-12532

11/30/2021, 6:00 PM

if you’ve got a

venv

set up inside

metadata-ingestion

then the fastest way I’ve found is to run

pytest tests/…/test_file.py

from within the venv

mammoth-bear-12532

11/30/2021, 6:00 PM

you do need to run

../gradlew installDevTest

once to make sure you have the right dependencies installed ahead of time otherwise pytest will complain

mammoth-bear-12532

11/30/2021, 6:02 PM

If you have suggestions on how we can use gradle’s inbuilt caching to skip a few of the install steps reliably, that would be awesome.

mammoth-bear-12532

12/01/2021, 2:07 AM

@fancy-sundown-164: we have an improvement in the build system for you to try. If you pull latest, you should be able to give this a try.

mammoth-bear-12532

12/01/2021, 2:07 AM

Copy code

# Run all tests in a single file
../gradlew :metadata-ingestion:testSingle -PtestFile=tests/unit/test_airflow.py
# Run all tests under tests/unit
../gradlew :metadata-ingestion:testSingle -PtestFile=tests/unit

https://datahubproject.io/docs/metadata-ingestion/developing/

Open in Slack

Previous Next