Hi, I'm developing a destination connector for a t...
# help-connector-development
r
Hi, I'm developing a destination connector for a to-be-published database. I can locally go through my connector along with a ./run-ab-platform.sh airbyte deployment, however I can not get normalization related option visible. I've tried as many ways as I count found, such as https://airbyte.com/tutorials/creating-duckdb-destination-with-python https://docs.airbyte.com/connector-development/testing-connectors/testing-a-local-catalog-in-development/, with no luck, and it seems some documents is out-of-date and some are rapidly changing. So is it possible to finish a locally end-to-end connector go through (w/ locally deployed airbyte, w/ normalization) before publish anything? Or I have to publish the connector first and then add normalization to it, like mentioned here https://docs.airbyte.com/connector-development/#4-publish-the-connector
k
A message from kapa.ai
Yes, it is possible to test your destination connector with normalization locally before publishing it. To activate normalization for your destination connector, you need to update the
spec.json
file and set
supportsNormalization
and
supportsDBT
to
true
. However, with the newest version, these settings have been moved to
destination_definitions.yaml
and
destination_specs.yaml
[source](https://airbyte.com/tutorials/creating-duckdb-destination-with-python). To build the Docker image and run the Gradle command for DuckDB, you can follow these steps:
Copy code
bash #build normalization cd airbyte-integrations/connectors/destination-duckdb docker build . -t airbyte/normalization-duckdb:0.1.0 -t airbyte/normalization-duckdb:0.2.25 --no-cache  cd ../../.. ./gradlew :airbyte-integrations:bases:base-normalization:airbyteDockerDuckDb && docker tag airbyte/normalization-duckdb:dev airbyte/normalization-duckdb:0.2.25
And to run the integration tests either locally with [PyTest](https://docs.pytest.org/):
Copy code
bash cd airbyte-integrations/bases/base-normalization NORMALIZATION_TEST_TARGET=duckdb  pytest integration_tests/test_normalization.py
Or with Gradle from the root folder:
Copy code
bash NORMALIZATION_TEST_TARGET=duckdb ./gradlew :airbyte-integrations:bases:base-normalization:integrationTest`
[source](https://airbyte.com/tutorials/creating-duckdb-destination-with-python) If you need more help with normalization, you can watch this [Demo Hour: Debugging Normalization](

https://youtu.be/7sKy7VbcNSM

) video for more insights. Remember that some documents might be outdated, and it's always a good idea to check the latest documentation and requirements. If you need further assistance, don't hesitate to reach out to the Airbyte community for support.
e
Hi @Robert! We are in the middle of dramatically changing how normalization works at Airbyte. In the very near future we will be removing support for normalization containers entirely. I know it's not the best answer, but we don't yet have anything so share about what the new typing-and-deduplciation process will look like other than it will be the responsibility of the destination container itself (e.g. the destination can run this method mid-sync, at every checkpoint). We are starting by adding support for our Java destinations this month. In a few weeks we should have a Java example for you, but we have no plans at this time for python support.
(edited the above to add links)
The goal is to eventually remove dbt from our connectors. We want to be able to more tightly control how things work, when typing-and-deduplication runs, and how we handle errors... which all point to the connector building and running that SQL itself. TBD on how soon that happens (answering a now removed question about dbt)
r
@Evan Tahler (Airbyte) Thanks again for quick response! Your information is very important. One more question, will airbyte try to keep compatibility to current connectors/normalizations for a while (emm, if so, how long?), or just push everything to their next generation?
e
There will be some breaking changes, especially as the output table structure is changing. Choosing which version of a connector you are using is how you can decide when to opt-into that change