Hello Guys, does anyone know, How can i replace th...
# help-connector-development
a
Hello Guys, does anyone know, How can i replace the original connector with the custom connector in airbyte. the problem i am facing is that catalog enrichment is different in both the connector. i would really like your help in case i am missing something
k
To replace an original connector with a custom version in Airbyte, you can follow these steps depending on whether you are modifying an existing connector locally or adding a completely new Docker image. ### 1. Modifying an Existing Connector Locally If you have modified the code or the specification (such as the catalog/spec.json) of an existing connector and want to test it: * Build the Image: Use
airbyte-ci
to build your modified connector. This will typically tag the image as
dev
on your local Docker host.
Copy code
bash     airbyte-ci connectors --name source-<source-name> build
[[Developing Locally (2.0)](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-contributions)] * Load the Image (if using abctl): If you are running Airbyte via
abctl
, you must load the local image into the cluster:
Copy code
bash     kind load docker-image airbyte/source-<source-name>:dev -n airbyte-abctl
[[Developing Locally (2.0)](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-contributions)] * Update in UI: Navigate to Workspace settings > Sources (or Destinations), find the original connector, and click the edit icon. Change the version tag to
dev
(or your custom tag) and click Change to save. This action also refreshes the connector specification cache, which is necessary if your catalog enrichment logic has changed. [[Developing Locally (2.0)](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-contributions); [Custom or New Connector](https://docs.airbyte.com/integrations/custom-connectors)] ### 2. Adding a New Custom Docker Connector If you have built a separate image and want to add it as a new option: 1. Go to Workspace settings > Sources (or Destinations). 2. Click New connector and select Add a new Docker connector. 3. Enter a display name, the Docker full image name, and the Docker image tag. 4. Once added, you can select this custom version when setting up a new connection. [[Using Custom Connectors (2.0)](https://docs.airbyte.com/platform/2.0/operator-guides/using-custom-connectors#4-use-your-custom-docker-connector-in-airbyte)] ### Addressing Catalog Enrichment Differences If the catalog enrichment is different between your custom connector and the original, the knowledge sources suggest that Airbyte Server caches these specifications. To ensure your custom catalog logic is recognized: * In the UI settings for that connector, re-saving the version (even if it's still
dev
) triggers a refresh of the connector's specification. [[Developing Locally (2.0)](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-contributions)] * If you are using the Connector Builder, you can implement Custom Components in Python to handle specialized transformations or complex API behaviors that standard components cannot address. [[Custom components for the Connector Builder](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/custom-components)]
a
my custom connector doesnt get sync_id , generation_id fields in the catalog @kapa.ai
k
A message from kapa.ai
I'm sorry, but the provided knowledge sources do not contain information on how to explicitly add
sync_id
or
generation_id
to a connector's catalog definition. In Airbyte, these fields are typically system-level metadata added by the platform at sync time rather than fields defined within the connector's discovered catalog. Based on the documentation provided: *
_airbyte_generation_id
and
_airbyte_meta
(which contains
sync_id
) are described as fields that "Always exist" in the destination output schema (e.g., in S3 CSV destinations) rather than in the source's discovered catalog. [[S3 Destination CSV Schema](https://docs.airbyte.com/integrations/destinations/s3#csv)] * The
stream_generation
table in the internal configuration database tracks these IDs, linking a
connection_id
to a
generation_id
and a
start_job_id
. [[Config Database stream_generation](https://docs.airbyte.com/platform/2.0/understanding-airbyte/database-data-catalog#stream_generation)] * When defining a connector's catalog via the
discover
method, you typically define the
json_schema
for the data and the
supported_sync_modes
. [[Custom Python Connector Discover](https://docs.airbyte.com/platform/1.8/connector-development/tutorials/custom-python-connector/discover)] If your custom connector requires specific sync modes or cursor fields to be present in the catalog for the platform to manage generations and syncs correctly, you may need to ensure your
AirbyteStream
object includes:
Copy code
python supported_sync_modes=['full_refresh', 'incremental'] source_defined_cursor=True default_cursor_field=['your_field']
[[Generated Python Source Connector Error](https://discuss.airbyte.io/t/2940)] If you are looking for these fields during the "Discover" phase in the UI, they generally will not appear because they are internal platform metadata injected during the actual sync process.