has anyone figured out writing tests and handling ...
# contributing-to-airbytec
charles
10/27/2020, 5:08 PM
has anyone figured out writing tests and handling test dependencies in python already in a way that is graceful?
u
user
10/27/2020, 5:08 PMI would be interested too!
u
user
10/27/2020, 5:09 PMwhat about handling test deps are you looking for? just importing them for testing only but not package them when shipping the package?
u
user
10/27/2020, 5:09 PMi have dependencies that are only releveant to the tests, not the main package.
u
user
10/27/2020, 5:09 PMlet me try to say that clearer.
u
user
10/27/2020, 5:09 PMthe tests depend on a library that the main program does not depend on.
u
user
10/27/2020, 5:10 PMi want to be able to run
pip install just-my-*******-test-dependenciesu
user
10/27/2020, 5:10 PMsetup.py looks something like this.
u
user
10/27/2020, 5:10 PM Copy code
setup(
...
install_requires=["tap-github==1.9.0", "requests", "base_singer", "airbyte_protocol"],
tests_require=["base_python_test"],
)u
user
10/27/2020, 5:10 PMbut i think tests_require may have been deprecated.
u
user
10/27/2020, 5:13 PMgoing to try to use extras_require
u
user
10/27/2020, 5:15 PMhttps://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html#optional-dependencies that seems to be the recommended method for tests
u
user
10/27/2020, 5:15 PMthey also recommend that we move away from
setup.pysetup.cfgu
user
10/27/2020, 5:18 PMlol yup.
u
user
10/27/2020, 5:21 PMi’m still not entirely sure from a DX point of view how it separates what files are allowed to use test deps versus main deps.
u
user
10/27/2020, 5:21 PMmaybe you just find out that you screwed up at runtime?
u
user
10/27/2020, 5:21 PMdoes it do any “interesting” separation in the .venv/libs directory?
u
user
10/27/2020, 5:22 PMnot that i’ve seen so far.
u
user
10/27/2020, 5:22 PMbut chances are i’m still doing 5 or 6 things wrong.
u
user
10/27/2020, 5:27 PMyou can use a separate requirements file for dev dependencies
u
user
10/27/2020, 5:27 PMthat's basically what requirements.txt is already
u
user
10/27/2020, 5:27 PMyeah. so i’m doing that.
u
user
10/27/2020, 5:27 PMonly for testing, because it's doing the editable installs
u
user
10/27/2020, 5:28 PMi’m definitely trying to do a weird thing where i want to expose a test module to be run externally.
u
user
10/27/2020, 5:28 PMso that’s my bad for not being cookiecutter.
u
user
10/27/2020, 5:29 PMI think the module depending on that test module should just link to it using an editable dependency
u
user
10/27/2020, 5:29 PMin requirements.txt
u
user
10/27/2020, 5:31 PMkk
u
user
10/27/2020, 5:31 PMwhen in the dokcerfile for an integration we run
RUN pip install .u
user
10/27/2020, 5:31 PMthat is using the install_reqs or setup right?
u
user
10/27/2020, 5:32 PMyeah setup.py
u
user
10/27/2020, 5:32 PMi guess that has to be true because we don’t copy of requirements.txt
u
user
10/27/2020, 5:34 PMso i think i can do this:
Copy code
extras_require={"standardtest": ["base_python_test", "tap-exchangeratesapi==0.1.1"]},pip install ".[standardtest]"u
user
10/27/2020, 5:34 PMor something like it
u
user
10/27/2020, 5:34 PMbut haven’t actually gotten it to work yet
u
user
10/27/2020, 5:36 PMI don't know if you can have an editable install within
extras_requireu
user
10/27/2020, 5:37 PMi don’t think i’m trying to though.
u
user
10/27/2020, 5:37 PMI doubt
base_python_testu
user
10/27/2020, 5:37 PMWhich is probably why it's failing
u
user
10/27/2020, 5:37 PMisn’t it the same as
install_requires=["tap-github==1.9.0", "requests", "base_singer", "airbyte_protocol"],u
user
10/27/2020, 5:37 PMbase_singer and airbyte_protocol are made available as part of the docker setup
u
user
10/27/2020, 5:37 PMthey are installed in parent images into the global python scope
u
user
10/27/2020, 5:38 PMyeah. i’m doing the same thing with base_python_test
u
user
10/27/2020, 5:40 PMdoes it show up under
pip listu
user
10/27/2020, 5:42 PMworking on it.
u
user
10/27/2020, 5:42 PMalso happy to look at the source
u
user
10/27/2020, 5:43 PMi’m confused right now, because i was trying to get this out in the virtualenv on my local machine
u
user
10/27/2020, 5:43 PMbut i think that’s in a dumb state for some reason.
u
user
10/27/2020, 5:43 PMi’ll try to build the docker image.
u
user
10/27/2020, 5:45 PMif you're trying to get this into your local venv you'll need to install as an editable dependency outside of setup.py
u
user
10/27/2020, 5:47 PMkk. dunno what that means tbh. but going to focus on the docker part for now and will try to decipher that later.
u
user
10/27/2020, 5:48 PMare you using unittest or pytest?
u
user
10/27/2020, 5:49 PMneither
u
user
10/27/2020, 5:49 PMno test framework
u
user
10/27/2020, 5:49 PMbecause it’s just passing an interface to the standard tests.
u
user
10/27/2020, 5:49 PMah
u
user
10/27/2020, 6:00 PMokay @Jared Rhizor (Airbyte) i think i’ve hit a wall. would you be willing to take a couple minutes to look with me before sync?
u
user
10/27/2020, 6:03 PMin a 1:1
u
user
10/27/2020, 6:03 PMah. oaky
u
user
10/27/2020, 6:24 PMis there a reason we use _ instead of - here? shouldn’t this refer to the package name? it seems like both resolve, so maybe it doesn’t matter.
Copy code
install_requires=["airbyte_protocol"],u
user
10/27/2020, 7:04 PMpackage names can't include
- Copy code
Package and Module Names
Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.u
user
10/27/2020, 7:05 PM Copy code
➜ airbyte git:(master) ✗ pip list
Package Version Location
------------------ ---------- ------------------------------
...
importlib-metadata 2.0.0
...
psycopg2-binary 2.8.5
python-dateutil 2.8.1u
user
10/27/2020, 7:06 PMit seems like a lot of them use dashes.
u
user
10/27/2020, 7:09 PMah sorry you are talking about the Pypi package
the pep is about the pure python package (that ends up after the
importu
user
10/27/2020, 7:09 PMoh
u
user
10/27/2020, 7:09 PMthis is why in requirements.txt you might have
psycopg2-binarypsycopg2_binaryu
user
10/27/2020, 7:09 PMthat confused the hell out of me the first time.