https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I am developing on source-hubspot c...
# replication-ask-ai
k
@kapa.ai I am developing on source-hubspot connector in local How can I perform sample tests to extract data and visualize the output in local ?
k
You can do this entirely locally by using the existing testing tooling around the connector image. ### 1. Build the 
source-hubspot
image From the Airbyte repo root (after forking/cloning):
Copy code
bash airbyte-ci connectors --name source-hubspot build
This will produce a local image like:
Copy code
text airbyte/source-hubspot:dev
[[Connector contributions](https://docs.airbyte.com/platform/1.6/contributing-to-airbyte/developing-locally#connector-contributions)] ### 2. Run connector tests (sample extraction) From the 
airbyte-integrations/connectors/source-hubspot
directory you have a few options: a) Fast local tests via Poe
Copy code
bash poe test-fast          # quick sanity tests poe test-unit-tests    # all unit tests poe test-integration-tests  # integration tests that hit HubSpot
These commands are the standard way to run connector tests locally for any connector. [[Common tasks](https://docs.airbyte.com/platform/connector-development/local-connector-development#common-development-tasks)] b) Full CI-like suite (includes Acceptance Tests)
Copy code
bash airbyte-ci connectors --name=source-hubspot test
This runs the same suite as CI (QA checks, unit, integration, acceptance tests). [[Testing connectors](https://docs.airbyte.com/platform/connector-development/testing-connectors)] To run Acceptance Tests specifically, ensure 
acceptance-test-config.yml
exists in the connector root and points to your built image:
Copy code
yaml connector_image: airbyte/source-hubspot:dev acceptance-tests:   spec:     tests:       - spec_path: "some_folder/spec.yaml"
Then run via 
airbyte-ci
as above, or directly with pytest (for debugging):
Copy code
bash cd airbyte-integrations/bases/connector-acceptance-test poetry install poetry run pytest -p connector_acceptance_test.plugin \   --acceptance-test-config=../../connectors/source-hubspot --pdb
[[Acceptance tests ref](https://docs.airbyte.com/platform/connector-development/testing-connectors/connector-acceptance-tests-reference#setting-up-standard-acceptance-tests-for-your-connector)] > Note: for tests that hit HubSpot, you must provide a 
config.json
with valid credentials in a 
.secrets
folder under the connector directory. [[Testing connectors](https://docs.airbyte.com/platform/connector-development/testing-connectors)] ### 3. Visualize the output The docs don’t describe a dedicated “pretty printer” for records, but you can: - Inspect test logs from 
poe
/ 
airbyte-ci
runs (they include the connector’s stdout, which is the Airbyte JSON messages). - Or, after loading the built image into a local 
abctl
instance, configure a HubSpot source in the UI and run a sync to a local destination (e.g., a warehouse or file destination) and inspect the resulting tables/files. [[Connector contributions](https://docs.airbyte.com/platform/1.6/contributing-to-airbyte/developing-locally#connector-contributions)] If you clarify whether you prefer running via the UI or purely via CLI/pytest, I can narrow the steps further.
k
@kapa.ai I want to run a single specific stream and check the extracted data
k
A message from kapa.ai
The knowledge sources don’t describe a way to run a single stream of 
source-hubspot
directly from the connector in isolation (e.g., “run only 
contacts
stream via CLI”), so I can’t give you an exact, documented command for that connector. What is documented is how to: 1. Select specific streams when reading via PyAirbyte (shown for a GitHub source, but the pattern is generic):
Copy code
python    source.get_available_streams()    source.set_streams(["issues"])   # choose a subset of streams    read_result = source.read(cache=colab_cache)
This shows that, when using PyAirbyte, you can restrict extraction to a subset of streams before reading. [[GitHub chatbot](https://airbyte.com/tutorials/github-documentation-chatbot-using-airbyte-pgvector)] 2. Run a configured catalog that includes only the streams you want for a custom Python source, then read:
Copy code
bash    poetry run source-survey-monkey-demo read \      --config secrets/config.json \      --catalog integration_tests/configured_catalog.json
In that example, the configured catalog JSON lists multiple streams, and you can control which streams are present/enabled there. [[Reading subresource](https://docs.airbyte.com/platform/connector-development/tutorials/custom-python-connector/reading-from-a-subresource)] Applied to your case, the documented pattern is: - Create a configured catalog that includes only the HubSpot stream you want to test (e.g., 
contacts
). - Run the connector’s 
read
command (or equivalent) with: - 
--config
pointing to your HubSpot credentials JSON. - 
--catalog
pointing to that single‑stream configured catalog. - Inspect the connector’s stdout (Airbyte messages) or send the output to a simple destination (e.g., file) via the Airbyte UI to visualize the extracted records. However, the exact filenames/commands for 
source-hubspot
(e.g., its executable name, sample catalogs) are not specified in the provided knowledge, so I can’t give you a connector‑specific command line with confidence.
k
@kapa.ai error:
Copy code
Poetry could not find a pyproject.toml file in /Users/kevinlefevre/PycharmProjects/airbyte/airbyte-integrations/connectors/source-hubspot or its parents
k
A message from kapa.ai
Poetry is complaining because it expects a 
pyproject.toml
in the current (or parent) directory, but the 
source-hubspot
connector directory doesn’t have one. That usually means you’re either: - In the wrong directory, or - Using the wrong test command for that connector type. From the docs, Python connectors are tested with 
pytest
(often via 
poetry run pytest
), but this assumes the connector is a Python connector with a 
pyproject.toml
in its root. [[Testing connectors](https://docs.airbyte.com/platform/connector-development/testing-connectors)] For local development, the recommended, connector‑agnostic way to run tests is via Poe the Poet tasks from the connector directory:
Copy code
bash cd airbyte-integrations/connectors/source-hubspot  # Fast sanity tests poe test-fast  # All unit tests poe test-unit-tests  # All integration tests poe test-integration-tests
These 
poe
commands abstract away whether the connector is Python, Java, or manifest‑only, and are the documented interface for “Running Tests” locally. [[Common tasks](https://docs.airbyte.com/platform/connector-development/local-connector-development#common-development-tasks)] So to fix your error: 1. Make sure you’re in the 
source-hubspot
connector directory. 2. Use 
poe
test tasks instead of calling 
poetry
directly for this connector. If you still need to use 
pytest
directly, you’ll need to run it in whatever subdirectory actually contains a 
pyproject.toml
, but the exact layout for 
source-hubspot
isn’t described in the provided docs, so I can’t give a connector‑specific path with certainty.
k
@kapa.ai adapt the read command to use poe
k
A message from kapa.ai
The knowledge sources only show running a source’s 
read
command directly with 
poetry
, for example:
Copy code
bash poetry run source-survey-monkey-demo read \   --config secrets/config.json \   --catalog integration_tests/configured_catalog.json
for the SurveyMonkey demo connector.[[Reading page](https://docs.airbyte.com/platform/connector-development/tutorials/custom-python-connector/reading-a-page#write-a-failing-test-that-reads-a-single-page)][[Reading subresource](https://docs.airbyte.com/platform/connector-development/tutorials/custom-python-connector/reading-from-a-subresource)] They also explain that Poe is used as a generic task runner, and that you can run tasks like 
poe test-fast
, 
poe test-unit-tests
, etc., from a connector directory, but they do not document a Poe task that wraps a 
read
command for a source connector.[[Tooling](https://docs.airbyte.com/platform/connector-development/local-connector-development#tooling)] Because there is no example or reference of a 
poe read
(or similar) task for connectors in the provided docs, I can’t reliably tell you the exact Poe command or task name for 
source-hubspot
. It may or may not exist in that connector’s 
pyproject.toml
/ Poe config, but that configuration is not included in the knowledge sources. Given the docs you shared, the only documented way to run a 
read
locally is still the direct pattern:
Copy code
bash <runner> run <connector-executable> read \   --config <path-to-config.json> \   --catalog <path-to-configured_catalog.json>
with 
<runner>
being 
poetry
in the examples above.[[Reading page](https://docs.airbyte.com/platform/connector-development/tutorials/custom-python-connector/reading-a-page#write-a-failing-test-that-reads-a-single-page)] To adapt this to Poe for 
source-hubspot
, you would need to define or inspect a Poe task in that connector’s configuration, but the knowledge sources do not show how that task is defined, so I can’t provide a concrete 
poe ...
command.
2 Views
Open in Slack
PreviousNext
Powered by