https://datahubproject.io logo
Join Slack
Powered by
Hi there! I'm developing a new ingestion source f...
# contribute-code
a
Hi there! I'm developing a new ingestion source for Sisense, almost there, i'm on the final piece of writing tests, I do have a few questions about this...(i'm not an expert)...i kinda followed the metabase test.. • created a setup folder with 3 json files (dataset.json, dashboards.json, charts.json) each with the rest api output from the sisense endpoints that i'm consuming • created 
sisense_mces_golden.json
with the output of a dashboard + charts in the test_sisense.py file i've replaced the mapping json variable as well as changing all the metabase reference for sisense directories, values when running pytest i've got the following error
Copy code
File "/Users/edgar.valdez/Documents/datahub/metadata-ingestion/venv/lib/python3.8/site-packages/coverage/config.py", line 563, in read_coverage_config
    raise ConfigError(f"Couldn't read {fname!r} as a config file")
coverage.exceptions.ConfigError: Couldn't read 'setup.cfg' as a config file
could anybody please guide me through this last bit 🙂
oh, seems that i forgot to install
Copy code
pip install -e '.[integration-tests]'
after running
Copy code
pytest -m 'not integration and not slow_integration'
i've got
I'd appreciate a bit of guidance in this topic! 🙂
d
Are you using intellij? There you need to specify the 
--no-cov
option when you are running the tests as it can’t work well with setup.py
a
Hi @dazzling-judge-80093, I'm using vs code
d
oh, I use intellij, so I know that one but there the --no-cov parameter help in the test runs
a
In terms is the setup.py that was solve installing the integration-test. The “issue” now is that (if I read the screenshot properly) the test are not passsing? But tbh I have worked with test before, so I'm a bit lost in this whole matter
See that 3rd column in the screenshot (0%)
d
Can you run your test directly? Like:
Copy code
pytest -v tests/unit/test_bigquery_source.py
a
oh, i only wrote a test under 
metadata-ingestion/tests/integration/
none under 
metadata-ingestion/tests/unit
d
you should be able to run those tests similar way as well -> 
pytest -v tests/integration/hive/test_hive.py
a
yup, i can
as I mentioned as i don't have experience building test i used the metabase test but replacing everything related to metabase to my source...files, variables, json mapping url's
d
your test method names should start with 
test_
prefix otherwise it won’t run
a
yup, the methods are called: • _test_mode_ingest_failure_ • _test_mode_ingest_success_
d
Can you try to run the Metabase tests from the command line similar way? Are you in the virtualenv when you run the tests?
I just tried to copy metabase under sisense directory and run the tests and for me it works fine
a
running the metabase works
i'm running from this path 
metadata-ingestion
with the venv activated
d
it looks ok
a
ok, so might be something i changed along the way to point to my .json files
ie i've changed this:
Copy code
JSON_RESPONSE_MAP = {
    "<http://localhost:3000/api/v1/elasticubes/getElasticubes>": "datasets.json",
    "<http://localhost:3000/api/v1/dashboards>": "dashboards.json",
    "<http://localhost:3000/api/v1/dashboard/1/widgets>": "charts.json",
}
to point to my files
and the test structure looks like this
d
but it seems like the tests method was not called at all
a
i renamed everything with metabase like this 
"datahub.ingestion.source.sisense.requests.session",
"<http://datahub.ingestion.source.sisense.requests.post|datahub.ingestion.source.sisense.requests.post>",
"datahub.ingestion.source.sisense.requests.delete",
i think the problem is here: 
test_resources_dir _=_ pytestconfig.rootpath _/_ "tests/integration/sisense"
somehow that 
pytestconfig.rootpath
is not showing anything if i do a print(pytestconfig.rootpath)
then the pipeline looks like this
Copy code
pipeline = Pipeline.create(
            {
                "run_id": "sisense-test",
                "source": {
                    "type": "sisense",
                    "config": {
                        "username": "xxxx",
                        "password": "xxxx",
                        "connect_uri": "<http://localhost:3000/>",
                    },
                },
                "sink": {
                    "type": "file",
                    "config": {"filename": f"{tmp_path}/sisense_mces.json",},
                },
            }
        )
oh, i just changed the port to match my SisenseConfig object
now i can see something
d
oh, cool
a
dumb...quick question...is does the url has to be the "real" url? for testing? not that localhost
so i need to input the real url & credentials in my test file?
i would have thought that's why i have hardcoded .json outputs...to not make those api calls and to safely commit the test
d
I guess this depends on your test
Normally you should mock service calls or if you bring up a service from docker the run it from there and connect with dummy secrets
a
🤔 i guess might be something else...metabase works well and it has the same structure & pattern
36 Views
Open in Slack
PreviousNext
Powered by