https://linen.dev logo
Join Slack
Powered by
how/where is the list of sources that we see in ai...
# ask-ai
j
how/where is the list of sources that we see in airbyte webapp maintained? when a new connector is added in airbytehq/airbyte repo, how does that get added to the list of sources with the right docker image and tags
k
The list of sources that you see in the Airbyte webapp is maintained in a configuration file within the Airbyte repository. When a new connector is added to the airbytehq/airbyte repository, it needs to be properly configured to appear in the list of sources with the correct Docker image and tags. Here's how this process works: 1. The list of sources is primarily maintained in a file called 
source_definitions.yaml
. This file is located in the Airbyte repository at 
airbyte/airbyte-config/init/src/main/resources/seed/source_definitions.yaml
[Forum Post: How do I locally switch to a modified version of a connector?](https://discuss.airbyte.io/t/1812). 2. When a new connector is added to the repository, it needs to be included in this 
source_definitions.yaml
file. This file contains the metadata for each source connector, including its name, Docker image, and version tag. 3. The 
source_definitions.yaml
file is used to seed the Airbyte database with the available connectors when Airbyte is initialized or updated [Forum Post: Airbyte source and destination definitions persistence issue](https://github.com/airbytehq/airbyte/issues/26820). 4. When you update a connector version in the Airbyte UI, these changes are not automatically persisted back to the 
source_definitions.yaml
file. This means that if you redeploy Airbyte, the connector versions will revert to the ones specified in the file [Forum Post: Airbyte source and destination definitions persistence issue](https://github.com/airbytehq/airbyte/issues/26820). 5. For a new connector to appear in the UI, it needs to be added to this file with the correct Docker image and tag information. The Docker image should be built and pushed to the appropriate registry (usually Airbyte's public Docker repository) [Documentation > Deploying-airbyte > Integrations > Custom image registry](https://docs.airbyte.com/deploying-airbyte/integrations/custom-image-registries). 6. After adding a new connector to the 
source_definitions.yaml
file and building/pushing the Docker image, you typically need to redeploy Airbyte for the changes to take effect and for the new connector to appear in the UI. It's important to note that this process may vary slightly depending on your specific Airbyte deployment and version. For custom or private connectors, there are additional steps involved in using custom Docker registries and adding the connector through the UI settings [Documentation > Operator-guides > Uploading Docker-based custom connectors](https://docs.airbyte.com/operator-guides/using-custom-connectors#4-use-your-custom-docker-connector-in-airbyte).
j
@kapa.ai can you provide a link to this source_definitions.yaml file? which repository is it in?
k
A message from kapa.ai
I apologize, but I need to correct my previous statement. The information I provided earlier about the 
source_definitions.yaml
file is outdated. According to more recent documentation, Airbyte has moved away from using 
source_definitions.yaml
and now uses a 
metadata.yaml
file for each connector. Here's the updated information: The list of sources is now maintained in individual 
metadata.yaml
files for each connector in the airbytehq/airbyte repository. These files contain crucial information about each connector, including its type, definition ID, Docker image tag, and more. You can find examples of these 
metadata.yaml
files in the Airbyte GitHub repository. For instance, the Postgres source connector's 
metadata.yaml
file would be located in the 
airbyte-integrations/connectors/source-postgres/
directory. For more details on the structure and purpose of the 
metadata.yaml
file, you can refer to the [Connector Metadata.yaml File documentation](https://docs.airbyte.com/connector-development/connector-metadata-file). This new approach replaces the previous 
source_definitions.yaml
and 
destinations_definitions.yaml
files, providing a more organized and accessible way to manage connector data.
j
@kapa.ai i read that the metadata.yaml files for each connector is loaded into the airbyte metadata service. can you tell me more about that
k
A message from kapa.ai
Certainly! The 
metadata.yaml
files for each connector are indeed loaded into the Airbyte metadata service. Let me provide you with more information about this process based on the knowledge sources available. According to the [Connector Metadata.yaml File documentation](https://docs.airbyte.com/connector-development/connector-metadata-file), when the 
metadata.yaml
file is updated, the data is automatically uploaded to Airbyte's metadata service. This service then uses this information to generate two publicly available registries: 1. 
oss_registry.json
(for open-source connectors) 2. 
cloud_registry.json
(for cloud connectors) The 
registries
section within the 
metadata.yaml
file plays a crucial role in determining the contents of these registry files. Here's how it works: 1. Each 
metadata.yaml
file contains a 
registries
section with two subsections: 
cloud
and 
oss
(Open Source Software). 2. These subsections specify whether the connector should be included in the respective registry and can override certain properties like the Docker repository. 3. If a connector is listed as 
enabled: true
under the 
oss
section, it will be included in the 
oss_registry.json
file. 4. Similarly, if it's listed as 
enabled: true
under the 
cloud
section, it will be included in the 
cloud_registry.json
file. Here's an example of how the 
registries
section might look in a 
metadata.yaml
file:
Copy code
yaml registries:   cloud:     dockerRepository: airbyte/source-postgres-strict-encrypt     enabled: true   oss:     enabled: true
This system provides a flexible and organized way to manage which connectors are included in each registry, allowing Airbyte to maintain separate lists for open-source and cloud deployments. The metadata service and the resulting registry files play a crucial role in how Airbyte handles connector data, improving the overall organization and accessibility of this information across the platform. [Connector Metadata.yaml File documentation](https://docs.airbyte.com/connector-development/connector-metadata-file)
j
@kapa.ai i see I can set the helm value global.image.registry to set the registry for the core airbyte and sidecar images. but how to set the registry for other images like different source and destinatoin connectors?
k
A message from kapa.ai
Based on the information provided in the knowledge sources, I can provide you with an updated answer regarding setting the registry for source and destination connectors in Airbyte. As of Airbyte version 1.4.0, released on January 17, 2025, connectors now support custom image registries. This means that when you configure Airbyte to use a custom image registry, it automatically uses that registry for connector images as well as platform images. This is a significant improvement from previous versions where only platform images supported this feature. To set the registry for source and destination connectors, you can use the same 
global.image.registry
configuration in your 
values.yaml
file that you use for core Airbyte and sidecar images. Here's an example of how to set this in your `values.yaml`:
Copy code
yaml global:   image:     registry: <http://my-registry.foo.com/|my-registry.foo.com/>
This configuration will prefix all docker images, including connector images, with 
<http://my-registry.foo.com/|my-registry.foo.com/>
. However, there are a few important points to keep in mind: 1. You must ensure that copies of both platform and connector images are available in your custom image registry. 2. [Custom Docker connectors](https://docs.airbyte.com/operator-guides/using-custom-connectors/) in your workspace that specify an image using a fully qualified domain name (for example, `example.com/airbyte/your-custom-source`) will ignore your configured custom image registry and pull images from the domain specified by that connector. 3. You need to make sure that your Kubernetes cluster has access to pull images from your custom registry. This may involve setting up appropriate authentication if your registry requires it. For more detailed information on setting up and using custom image registries with Airbyte, you can refer to the [Custom image registry documentation](https://docs.airbyte.com/deploying-airbyte/integrations/custom-image-registries). [Release notes for Airbyte 1.4.0](https://docs.airbyte.com/release_notes/v-1.4)
Open in Slack
PreviousNext
Powered by