Hello community!
Creating a PR for contribution f...
# troubleshootb
best-umbrella-88325
02/06/2023, 10:44 AMHello community!
Creating a PR for contribution for the very first time, so new to running the unit tests on the DataHub project.
I've made a few changes and am running the unit test as highlighted in https://datahubproject.io/docs/metadata-ingestion/developing/#testing.
When I run these tests, 5 of the tests fail with errors.. Not sure what's happening or happens under the hood. Can someone please help me out here?
Test output in 🧵
best-umbrella-88325
02/06/2023, 10:45 AM Copy code
-----------------------------------------------------------------------------------------------------------------------
TOTAL 33591 12031 64%
========================================================================================================= short test summary info ==========================================================================================================
FAILED tests/unit/test_pipeline.py::TestPipeline::test_configure_without_sink - AssertionError: assert {'server': 'h...RE-uPOh9GoV4'} == {'server': 'h...'token': None}
FAILED tests/integration/kafka-connect/test_kafka_connect.py::test_kafka_connect_ingest - Exception: Command docker-compose -f "/mnt/c/XXX/XXX/open-datahub/datahub/metadata-ingestion/tests/integration/kafka/docker-compose.yml" -f "/mnt/c/XXX/XXX/open-datahub/datahub/metadata-ingestion/tests/integration/kaf...
FAILED tests/integration/kafka-connect/test_kafka_connect.py::test_kafka_connect_mongosourceconnect_ingest - Exception: Command docker-compose -f "/mnt/c/XXX/XXX/open-datahub/datahub/metadata-ingestion/tests/integration/kafka/docker-compose.yml" -f "/mnt/c/XXX/XXX/open-datahub/datahub/metadata-ingestion/tests/integration/kaf...
FAILED tests/integration/ldap/test_ldap_stateful.py::test_ldap_stateful - Exception: Command docker-compose -f "/mnt/c/XXX/XXX/open-datahub/datahub/metadata-ingestion/tests/integration/ldap/docker-compose.yml" -p "pytest287-ldap" up --build -d returned 1: """openldap Pulling
FAILED tests/integration/s3/test_s3.py::test_data_lake_s3_ingest[file_without_extension.json] - AssertionError: MCEs differ
FAILED tests/integration/s3/test_s3.py::test_data_lake_s3_ingest[multiple_files.json] - AssertionError: MCEs differ
FAILED tests/integration/s3/test_s3.py::test_data_lake_local_ingest[file_without_extension.json] - datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (s3): [Errno 2] No such file or directory
FAILED tests/integration/s3/test_s3.py::test_data_lake_local_ingest[folder_no_partition.json] - datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (s3): [Errno 2] No such file or directory
FAILED tests/integration/s3/test_s3.py::test_data_lake_local_ingest[folder_no_partition_exclude.json] - datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (s3): [Errno 2] No such file or directory
FAILED tests/integration/s3/test_s3.py::test_data_lake_local_ingest[folder_no_partition_filename.json] - datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (s3): [Errno 2] No such file or directory
FAILED tests/integration/s3/test_s3.py::test_data_lake_local_ingest[folder_no_partition_glob.json] - datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (s3): [Errno 2] No such file or directory
FAILED tests/integration/s3/test_s3.py::test_data_lake_local_ingest[folder_partition_basic.json] - datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (s3): [Errno 2] No such file or directory
FAILED tests/integration/s3/test_s3.py::test_data_lake_local_ingest[folder_partition_keyval.json] - datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (s3): [Errno 2] No such file or directory
FAILED tests/integration/s3/test_s3.py::test_data_lake_local_ingest[multiple_files.json] - datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (s3): [Errno 2] No such file or directory
FAILED tests/integration/s3/test_s3.py::test_data_lake_local_ingest[multiple_spec_for_files.json] - datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (s3): [Errno 2] No such file or directory
FAILED tests/integration/s3/test_s3.py::test_data_lake_local_ingest[single_file.json] - datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (s3): [Errno 2] No such file or directory
FAILED tests/integration/snowflake/test_snowflake.py::test_snowflake_basic - AssertionError: MCEs differ
ERROR tests/integration/presto-on-hive/test_presto_on_hive.py::test_presto_on_hive_ingest[hive-False-False-False-_1] - Exception: Command docker-compose -f "/mnt/c/XXX/XXX/open-datahub/datahub/metadata-ingestion/tests/integration/presto-on-hive/docker-compose.yml" -p "pytest287-presto-on-hive" up --build -d returned 1: """presto-cli Pulling
ERROR tests/integration/presto-on-hive/test_presto_on_hive.py::test_presto_on_hive_ingest[presto-on-hive-True-True-False-_2] - Exception: Command docker-compose -f "/mnt/c/XXX/XXX/open-datahub/datahub/metadata-ingestion/tests/integration/presto-on-hive/docker-compose.yml" -p "pytest287-presto-on-hive" up --build -d returned 1: """presto-cli Pulling
ERROR tests/integration/presto-on-hive/test_presto_on_hive.py::test_presto_on_hive_ingest[hive-False-False-True-_3] - Exception: Command docker-compose -f "/mnt/c/XXX/XXX/open-datahub/datahub/metadata-ingestion/tests/integration/presto-on-hive/docker-compose.yml" -p "pytest287-presto-on-hive" up --build -d returned 1: """presto-cli Pulling
ERROR tests/integration/presto-on-hive/test_presto_on_hive.py::test_presto_on_hive_ingest[presto-on-hive-True-True-True-_4] - Exception: Command docker-compose -f "/mnt/c/XXX/XXX/open-datahub/datahub/metadata-ingestion/tests/integration/presto-on-hive/docker-compose.yml" -p "pytest287-presto-on-hive" up --build -d returned 1: """presto-cli Pulling
ERROR tests/integration/presto-on-hive/test_presto_on_hive.py::test_presto_on_hive_instance_ingest - Exception: Command docker-compose -f "/mnt/c/XXX/XXX/open-datahub/datahub/metadata-ingestion/tests/integration/presto-on-hive/docker-compose.yml" -p "pytest287-presto-on-hive" up --build -d returned 1: """presto-cli Pulling
======================================================================= 17 failed, 754 passed, 1 skipped, 47 deselected, 68 warnings, 5 errors in 1093.11s (0:18:13) =======================================================================
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.9/logging/__init__.py", line 1082, in emit
stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
File "/mnt/c/XXX/XXX/open-datahub/datahub/metadata-ingestion/venv/lib/python3.9/site-packages/airflow/settings.py", line 407, in dispose_ormg
gentle-hamburger-31302
02/07/2023, 12:11 PMHi @best-umbrella-88325
Are you writing new ingestion source ?
b
best-umbrella-88325
02/07/2023, 12:11 PMHey!
Nope, making a few modifications to the existing S3 source
g
gentle-hamburger-31302
02/07/2023, 12:12 PMparent directory of source ?
b
best-umbrella-88325
02/07/2023, 12:12 PMWould you please elaborate as to what you are referring to?
best-umbrella-88325
02/07/2023, 12:14 PMI've made a change in a couple of files in metadata-ingestion/src/datahub/ingestion/source/s3/
g
gentle-hamburger-31302
02/07/2023, 12:14 PMExample
gentle-hamburger-31302
02/07/2023, 12:14 PMPowerBI source parent directory is datahub/metadata-ingestion/src/datahub/ingestion/source/powerbi
b
best-umbrella-88325
02/07/2023, 12:14 PMdatahub/metadata-ingestion/src/datahub/ingestion/source/s3/
b
better-orange-49102
02/07/2023, 12:20 PMA suggestion is to check master branch to see if the tests have been since updated... It changes pretty rapidly
g
gentle-hamburger-31302
02/07/2023, 12:31 PMPlease follow below steps
1. clean the ingestion: ./gradlew metadata ingestionclean
2. Install dev dependency: ./gradlew metadata ingestioninstallDev
3. Check if there any lint error got introduced because of your changes: ./gradlew metadata ingestionlintFix
4. Change current directory: cd datahub/metadata-ingestion
5. Activate virtual environment: source venv/bin/activate
6. Execute S3 test: pytest tests/integration/s3
The test in step 6 should pass , if you get error
AssertionError: MCEs differpytest tests/integration/s3 --update-golden-filesgentle-hamburger-31302
02/07/2023, 12:31 PM^ @best-umbrella-88325
b
best-umbrella-88325
02/07/2023, 12:41 PMSure, will give this a try
best-umbrella-88325
02/08/2023, 10:33 AM@gentle-hamburger-31302 Thanks this worked! The S3 tests ran successfully. Although I have one question. Will running only the integration tests be enough to make it eligible to create a PR?
g
gentle-hamburger-31302
02/08/2023, 10:39 AMYou open PR , once you raise PR it will trigger complete test cycle and in that cycle the build should pass, lint fixes and source existing and new integration test passing is must to open PR
b
best-umbrella-88325
02/08/2023, 10:40 AMNoted. Thank you!
2 Views