https://linen.dev logo
c

charles

10/27/2020, 5:08 PM
has anyone figured out writing tests and handling test dependencies in python already in a way that is graceful?
u

user

10/27/2020, 5:08 PM
I would be interested too!
u

user

10/27/2020, 5:09 PM
what about handling test deps are you looking for? just importing them for testing only but not package them when shipping the package?
u

user

10/27/2020, 5:09 PM
i have dependencies that are only releveant to the tests, not the main package.
u

user

10/27/2020, 5:09 PM
let me try to say that clearer.
u

user

10/27/2020, 5:09 PM
the tests depend on a library that the main program does not depend on.
u

user

10/27/2020, 5:10 PM
i want to be able to run
pip install just-my-*******-test-dependencies
u

user

10/27/2020, 5:10 PM
setup.py looks something like this.
u

user

10/27/2020, 5:10 PM
Copy code
setup(
    ...
    install_requires=["tap-github==1.9.0", "requests", "base_singer", "airbyte_protocol"],
    tests_require=["base_python_test"],
)
u

user

10/27/2020, 5:10 PM
but i think tests_require may have been deprecated.
u

user

10/27/2020, 5:13 PM
going to try to use extras_require
u

user

10/27/2020, 5:15 PM
u

user

10/27/2020, 5:15 PM
they also recommend that we move away from
setup.py
to
setup.cfg
u

user

10/27/2020, 5:18 PM
lol yup.
u

user

10/27/2020, 5:21 PM
i’m still not entirely sure from a DX point of view how it separates what files are allowed to use test deps versus main deps.
u

user

10/27/2020, 5:21 PM
maybe you just find out that you screwed up at runtime?
u

user

10/27/2020, 5:21 PM
does it do any “interesting” separation in the .venv/libs directory?
u

user

10/27/2020, 5:22 PM
not that i’ve seen so far.
u

user

10/27/2020, 5:22 PM
but chances are i’m still doing 5 or 6 things wrong.
u

user

10/27/2020, 5:27 PM
you can use a separate requirements file for dev dependencies
u

user

10/27/2020, 5:27 PM
that's basically what requirements.txt is already
u

user

10/27/2020, 5:27 PM
yeah. so i’m doing that.
u

user

10/27/2020, 5:27 PM
only for testing, because it's doing the editable installs
u

user

10/27/2020, 5:28 PM
i’m definitely trying to do a weird thing where i want to expose a test module to be run externally.
u

user

10/27/2020, 5:28 PM
so that’s my bad for not being cookiecutter.
u

user

10/27/2020, 5:29 PM
I think the module depending on that test module should just link to it using an editable dependency
u

user

10/27/2020, 5:29 PM
in requirements.txt
u

user

10/27/2020, 5:31 PM
kk
u

user

10/27/2020, 5:31 PM
when in the dokcerfile for an integration we run
RUN pip install .
u

user

10/27/2020, 5:31 PM
that is using the install_reqs or setup right?
u

user

10/27/2020, 5:32 PM
yeah setup.py
u

user

10/27/2020, 5:32 PM
i guess that has to be true because we don’t copy of requirements.txt
u

user

10/27/2020, 5:34 PM
so i think i can do this:
Copy code
extras_require={"standardtest": ["base_python_test", "tap-exchangeratesapi==0.1.1"]},
pip install ".[standardtest]"
u

user

10/27/2020, 5:34 PM
or something like it
u

user

10/27/2020, 5:34 PM
but haven’t actually gotten it to work yet
u

user

10/27/2020, 5:36 PM
I don't know if you can have an editable install within
extras_require
u

user

10/27/2020, 5:37 PM
i don’t think i’m trying to though.
u

user

10/27/2020, 5:37 PM
I doubt
base_python_test
is actually installed within the venv you are using
u

user

10/27/2020, 5:37 PM
Which is probably why it's failing
u

user

10/27/2020, 5:37 PM
isn’t it the same as
install_requires=["tap-github==1.9.0", "requests", "base_singer", "airbyte_protocol"],
u

user

10/27/2020, 5:37 PM
base_singer and airbyte_protocol are made available as part of the docker setup
u

user

10/27/2020, 5:37 PM
they are installed in parent images into the global python scope
u

user

10/27/2020, 5:38 PM
yeah. i’m doing the same thing with base_python_test
u

user

10/27/2020, 5:40 PM
does it show up under
pip list
on the image?
u

user

10/27/2020, 5:42 PM
working on it.
u

user

10/27/2020, 5:42 PM
also happy to look at the source
u

user

10/27/2020, 5:43 PM
i’m confused right now, because i was trying to get this out in the virtualenv on my local machine
u

user

10/27/2020, 5:43 PM
but i think that’s in a dumb state for some reason.
u

user

10/27/2020, 5:43 PM
i’ll try to build the docker image.
u

user

10/27/2020, 5:45 PM
if you're trying to get this into your local venv you'll need to install as an editable dependency outside of setup.py
u

user

10/27/2020, 5:47 PM
kk. dunno what that means tbh. but going to focus on the docker part for now and will try to decipher that later.
u

user

10/27/2020, 5:48 PM
are you using unittest or pytest?
u

user

10/27/2020, 5:49 PM
neither
u

user

10/27/2020, 5:49 PM
no test framework
u

user

10/27/2020, 5:49 PM
because it’s just passing an interface to the standard tests.
u

user

10/27/2020, 5:49 PM
ah
u

user

10/27/2020, 6:00 PM
okay @Jared Rhizor (Airbyte) i think i’ve hit a wall. would you be willing to take a couple minutes to look with me before sync?
u

user

10/27/2020, 6:03 PM
in a 1:1
u

user

10/27/2020, 6:03 PM
ah. oaky
u

user

10/27/2020, 6:24 PM
is there a reason we use _ instead of - here? shouldn’t this refer to the package name? it seems like both resolve, so maybe it doesn’t matter.
Copy code
install_requires=["airbyte_protocol"],
u

user

10/27/2020, 7:04 PM
package names can't include
-
if i'm not wrong? From PEP:
Copy code
Package and Module Names

Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.
https://www.python.org/dev/peps/pep-0008/#prescriptive-naming-conventions
u

user

10/27/2020, 7:05 PM
Copy code
➜  airbyte git:(master) ✗ pip list
Package            Version    Location
------------------ ---------- ------------------------------
...
importlib-metadata 2.0.0
...
psycopg2-binary    2.8.5
python-dateutil    2.8.1
u

user

10/27/2020, 7:06 PM
it seems like a lot of them use dashes.
u

user

10/27/2020, 7:09 PM
ah sorry you are talking about the Pypi package the pep is about the pure python package (that ends up after the
import
directives)
u

user

10/27/2020, 7:09 PM
oh
u

user

10/27/2020, 7:09 PM
this is why in requirements.txt you might have
psycopg2-binary
but you actually import
psycopg2_binary
?
u

user

10/27/2020, 7:09 PM
that confused the hell out of me the first time.