@kapa.ai how to add a custom connector in Airbyte Cloud?

To add a custom connector in Airbyte, you can follow these steps: 1. Get the Docker coordinate of a custom connector from Dockerhub (or any image repository that Airbyte can access). 2. In the Airbyte UI, go to the Admin section, and click on

[+ New connector]

on the top right. 3. You will be asked for the display name, the Docker repository name (repository + image name), tag, and documentation URL for that connector. Once this is filled, you will see your connector in the UI and your team will be able to use it, from the UI and Airbyte's API too. Note that this new connector could just be an updated version of an existing connector that you adapted to your specific edge case. Anything is possible! [source](https://docs.airbyte.com/integrations/custom-connectors#adding-your-connectors-in-the-ui) When using Airbyte on Kubernetes, the repository name must be a valid Kubernetes name. That is, it must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name', or '123-abc'). Other names will work locally on Docker but cause an error on Kubernetes (Internal Server Error: Get Spec job failed). [source](https://docs.airbyte.com/integrations/custom-connectors#adding-your-connectors-in-the-ui) Please note that custom connectors are currently exclusive to Airbyte Open-Source deployments. However, there are plans for their release on Airbyte Cloud, scheduled for January 2024. You can track the progress on this development [here](https://github.com/orgs/airbytehq/projects/37?pane=issue&itemId=45471174). [source](https://docs.airbyte.com/integrations/custom-connectors) If you'd like to ask for a new connector, you can request it directly [here](https://github.com/airbytehq/airbyte/discussions/new?category=new-connector-request). [source](https://docs.airbyte.com/integrations/custom-connectors)

I don't see an Admin section in my Airbyte Cloud UI.. @kapa.ai

A message from kapa.ai

I'm sorry for the confusion, but currently, the ability to add custom connectors is not available in Airbyte Cloud. This feature is only available in the open-source version of Airbyte. The Airbyte team is actively working on this feature and it is scheduled for release on Airbyte Cloud in January 2024. You can track the progress of this feature on this [Github project page](https://github.com/orgs/airbytehq/projects/37?pane=issue&itemId=45471174). In the meantime, if you need a new connector that is not currently supported, you can request it directly [here](https://github.com/airbytehq/airbyte/discussions/new?category=new-connector-request). The Airbyte team regularly reviews these requests and adds new connectors based on user demand.

@kapa.ai what all do we need in source.py file for custom connectors?

A message from kapa.ai

The

source.py

file is crucial when creating a custom connector in Airbyte. It contains the implementation of your connector and defines the streams (endpoints) you want to hit. Here's what you typically need: 1. Imports: Import necessary modules and packages. This includes

requests

,

logging

,

AbstractSource

,

Stream

,

HttpStream

, and any other dependencies your connector might need. 2. Connector Class: Define a class for your connector that inherits from

AbstractSource

. This class should implement methods like

check_connection

and

streams

. The

check_connection

method is used to verify if the connection to the data source is successful. The

streams

method returns a list of streams (endpoints) that the connector can pull data from. 3. Stream Classes: For each endpoint you want to hit, define a class that inherits from

HttpStream

or

Stream

. This class should implement methods like

path

,

parse_response

, and

next_page_token

. The

path

method defines the API endpoint to hit,

parse_response

parses the API response, and

next_page_token

handles pagination. Here's a sample code snippet from the knowledge sources:

python from typing import Any, Iterable, List, Mapping, MutableMapping, Optional, Tuple   import requests   import logging   from airbyte_cdk.sources import AbstractSource   from airbyte_cdk.sources.streams import Stream   from airbyte_cdk.sources.streams.http import HttpStream    logger = logging.getLogger("airbyte")    class SourcePythonHttpExample(AbstractSource):       def check_connection(self, logger, config) -> Tuple[bool, any]:           # Implementation of connection check logic         pass      def streams(self, config: Mapping[str, Any]) -> List[Stream]:           # Return list of streams         pass  class Pokemon(HttpStream):       url_base = "<https://pokeapi.co/api/v2/>"       primary_key = None        def __init__(self, pokemon_name: str, **kwargs):           super().__init__(**kwargs)           self.pokemon_name = pokemon_name        def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:           # Handle pagination         pass      def path(self) -> str:           # Define API endpoint         pass      def parse_response(self) -> Iterable[Mapping]:           # Parse API response         pass

[source](https://docs.airbyte.com/connector-development/tutorials/cdk-speedrun#define-connector-inputs) Please note that the actual implementation of these methods will depend on the specifics of the API you're connecting to.