https://linen.dev logo
Join Slack
Powered by
Why is develop locally removed from tutorials? <ht...
# help-connector-development
a
Why is develop locally removed from tutorials? https://github.com/airbytehq/airbyte/blob/master/docs/contributing-to-airbyte/developing-locally.md I am trying to write simple jdbc java source connector for databricks. I was following above document. Are there any updates in it which we should be aware of? I am facing difficulties in running gradle build. Some acceptance tests are failing which i have not even touched. Log of gradle build:
Copy code
E               requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
         
         .venv/lib/python3.9/site-packages/requests/adapters.py:578: ReadTimeout
         
         
          test_utils.py::test_not_found_container ???                        98% ??????????????????????????????
          test_utils.py::TestLoadYamlOrJsonPath.test_load_json ???           98% ??????????????????????????????
          test_utils.py::TestLoadYamlOrJsonPath.test_load_yaml ???           99% ??????????????????????????????
          test_utils.py::TestLoadYamlOrJsonPath.test_load_other ???          99% ??????????????????????????????
          test_utils.py::test_find_all_values_for_key_in_schema[schema0-looking_for_this_key-expected_values0] ???99% ??????????????????????????????
          test_utils.py::test_find_all_values_for_key_in_schema[schema1-looking_for_this_key-expected_values1] ???99% ??????????????????????????????
          test_utils.py::test_find_all_values_for_key_in_schema[schema2-looking_for_this_key-expected_values2] ???99% ??????????????????????????????
          test_utils.py::test_build_configured_catalog_from_discovered_catalog_and_empty_streams[discovered_catalog0-empty_streams0] ???99% ??????????????????????????????
          test_utils.py::test_build_configured_catalog_from_discovered_catalog_and_empty_streams[discovered_catalog1-empty_streams1] ???99% ??????????????????????????????
          test_utils.py::test_build_configured_catalog_from_custom_catalog[custom_configured_catalog0-False] ???99% ??????????????????????????????
          test_utils.py::test_build_configured_catalog_from_custom_catalog[custom_configured_catalog1-True] ???100% ??????????????????????????????=============================== warnings summary ===============================
         test_config.py::TestConfig::test_config_parsing[Legacy config should be parsed without error.]
         test_config.py::TestConfig::test_legacy_config_migration[A legacy raw config is parsed into a new config structure without error.]
         test_config.py::TestConfig::test_legacy_config_migration[A legacy raw config, with a test_strictness_level defined, is parsed into a new config structure without error.]
           /Users/ameya/Workspace/airbyte/airbyte-integrations/bases/connector-acceptance-test/connector_acceptance_test/config.py:285: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
             logging.warn("The acceptance-test-config.yml file is in a legacy format. Please migrate to the latest format.")
         
         -- Docs: <https://docs.pytest.org/en/stable/warnings.html>
         =========================== short test summary info ============================
         FAILED unit_tests/test_utils.py::test_docker_runner[standard] - requests.exce...
         FAILED unit_tests/test_utils.py::test_docker_runner[waiting] - requests.excep...
         FAILED unit_tests/test_utils.py::test_not_found_container - requests.exceptio...
         
         Results (624.33s (0:10:24)):
              543 passed
                3 failed
                  - test_utils.py:268 test_docker_runner[standard]
                  - test_utils.py:268 test_docker_runner[waiting]
                  - test_utils.py:308 test_not_found_container
         /Users/ameya/Workspace/airbyte/airbyte-integrations/bases/connector-acceptance-test/.venv/lib/python3.9/site-packages/coverage/control.py:788: CoverageWarning: No data was collected. (no-data-collected)
           self._warn("No data was collected.", slug="no-data-collected")

> Task :airbyte-integrations:bases:connector-acceptance-test:_unitTestCoverage FAILED
k
A message from kapa.ai
Based on the information provided in the extracts, it seems like you're facing issues with some acceptance tests failing while trying to develop a JDBC Java source connector for Databricks. The document you're referring to, [Developing Locally](https://github.com/airbytehq/airbyte/blob/master/docs/contributing-to-airbyte/developing-locally.md), is part of Airbyte's documentation and not a tutorial. It provides guidance on how to set up your local environment for Airbyte development. There's no information in the extracts indicating that this document has been removed from the tutorials. Regarding the failing tests, Airbyte runs all connectors against the same set of integration tests to ensure a minimum quality bar. These tests ensure that each connector adheres to the [Airbyte Specification](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol) and responds correctly to Airbyte commands when provided valid (or invalid) inputs. If some tests are failing, it could be due to some issues with your connector's implementation or configuration. In the 
high
test strictness level, the 
basic_read
test is run on a configured catalog created from the discovered catalog from which declared empty streams are removed. If you have declared 
configured_catalog_path
in the test configuration, it could lead to test failures as it's not allowed ([source](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference)). If you're facing issues with the Gradle build, it could be due to various reasons such as incorrect Java version, issues with the environment setup, etc. One user faced a similar issue and was able to resolve it by setting the 
JAVA_HOME
environment to point to the JDK folder ([source](https://docs.airbyte.com/contributing-to-airbyte/developing-locally)). If the issue persists, I would recommend reaching out to the Airbyte community on their [Slack channel](https://airbytehq.slack.com) or [forum](https://discuss.airbyte.io) for more specific guidance.
a
The documentation link for Developing Locally no longer exists. It was present in last week.
@kapa.ai
k
A message from kapa.ai
Sorry @Ameya Bapat, I seem to be having a bit of trouble. Don't worry though, I've pinged my support folks and they will try to fix me up! 🧑‍💻 Perhaps try again in 5mins?
a
@kapa.ai any updates?
k
A message from kapa.ai
Sorry @Ameya Bapat, I seem to be having a bit of trouble. Don't worry though, I've pinged my support folks and they will try to fix me up! 🧑‍💻 Perhaps try again in 5mins?
l
a
it was https://github.com/airbytehq/airbyte/blob/master/docs/contributing-to-airbyte/developing-locally.md and now think it is moved to other place and the documentation link (not git repo page) is throwing page not found.
@Luis Vicente are you able to build via gradle locally?
l
I was 2 weeks ago. I was updating a source and I added a bunch of unit tests and I was able to run them locally
but it was written in java
a
ok. I am trying to create a simple java jdbc source connector for databricks. i ma facing issues in building via gradle hence, not able to run my connector locally from code.
l
let me try to run my tests again just in case they are broken too
a
ok cool
l
./gradlew :airbyte-integrations:connectors:source-kafka:test
worked for me
but I might not have the same kind of tests as you
a
ok. actually i am running complete build 
SUB_BUILD=PLATFORM ./gradlew build
not specific one
l
ah... I don't build the entire thing for my tests. Just the tests for the source
a
in order to run specific builds and test on local, i need to build entire airbyte atleast once. Facing issues in it.
l
maybe my repo was already fully compiled before latest changes. You will have to wait until some airbyte people replies to you
a
yeah, exactly. It would not compile again.
Open in Slack
PreviousNext
Powered by