Silvia Giammarinaro
06/03/2024, 10:27 AMkapa.ai
06/03/2024, 10:27 AM<https://api.airbyte.com/v1/destinations>
, you can use Basic Authentication. Here is an example of how you can do this using the RestClient in C#: csharp var client = new RestClient("<https://api.airbyte.com/v1/destinations>"); var request = new RestRequest(); request.AddHeader("accept", "application/json"); request.AddHeader("content-type", "application/json"); request.AddHeader("Authorization", "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes("username:password")));
Replace "username"
and "password"
with your actual Airbyte username and password. For more details on setting up destinations and authentication, you can refer to the [Airbyte API documentation](https://reference.airbyte.com/reference/start) and the [tutorial on creating and monitoring a data pipeline using the Airbyte API](https://airbyte.com/tutorials/create-and-monitor-a-data-pipeline-using-the-airbyte-api).Silvia Giammarinaro
06/03/2024, 10:28 AMkapa.ai
06/03/2024, 10:29 AMbash
cd ../../connectors/source-exchange-rates-tutorial
2. Install dependencies using Poetry:
bash
poetry install
3. Verify the setup by running the spec
operation:
bash
poetry run source-exchange-rates-tutorial spec
You should see an output similar to this:
json
{"type": "SPEC", "spec": {"documentationUrl": "<https://docsurl.com>", "connectionSpecification": {"$schema": "<http://json-schema.org/draft-07/schema#>", "title": "Python Http Tutorial Spec", "type": "object", "required": ["TODO"], "additionalProperties": false, "properties": {"TODO: This schema defines the configuration required for the source. This usually involves metadata such as database and/or authentication information.": {"type": "string", "description": "describe me"}}}}}
[Source](https://docs.airbyte.com/connector-development/config-based/tutorial/install-dependencies)
### Running Commands to Validate Connectors
To ensure your connector is valid and can connect to the source, you can run the spec
and check
commands:
1. Run the spec
command:
bash
python main.py spec
2. Run the check
command with a configuration file:
bash
python main.py check --config secrets/config.json
[Source](https://discuss.airbyte.io/t/2508)
### Example of a Custom Source Connector
Here is an example of a custom source connector in Python that skips the first API call if the pre-fetched data is empty:
```python
class SourceFoo(AbstractSource):
def check_connection(self, logger, config) -> Tuple[bool, any]:
resp = requests.get("FOO_URL", headers=headers)
if resp.status_code == 200:
return True, None
else:
return False, resp.text
def streams(self, config: Mapping[str, Any]) -> List[Stream]:
auth = TokenAuthenticator(token=config["token"], auth_method="Basic")
ok, ids = self.get_ids(config["token"])
return [
Items(ids=ids, authenticator=auth),
]
def get_ids(self, token) -> Tuple[bool, any]:
resp = requests.get("FOO_URL", ...)
if resp.status_code == 200:
ids = resp.json()["data"]
return True, [id for id in ids if condition]
else:
return False, resp.text
class Items(HttpStream):
url_base = "FOO_URL"
primary_key = "id"kapa.ai
06/03/2024, 10:29 AMbash cd ../../connectors/source-exchange-rates-tutorial
2. Install dependencies using Poetry: bash poetry install
3. Verify the setup by running the spec
operation: bash poetry run source-exchange-rates-tutorial spec
You should see an output similar to this: json {"type": "SPEC", "spec": {"documentationUrl": "<https://docsurl.com>", "connectionSpecification": {"$schema": "<http://json-schema.org/draft-07/schema#>", "title": "Python Http Tutorial Spec", "type": "object", "required": ["TODO"], "additionalProperties": false, "properties": {"TODO: This schema defines the configuration required for the source. This usually involves metadata such as database and/or authentication information.": {"type": "string", "description": "describe me"}}}}}
[Source](https://docs.airbyte.com/connector-development/config-based/tutorial/install-dependencies) ### Running Commands to Validate Connectors To ensure your connector is valid and can connect to the source, you can run the spec
and check
commands: 1. Run the spec
command: bash python main.py spec
2. Run the check
command with a configuration file: bash python main.py check --config secrets/config.json
[Source](https://discuss.airbyte.io/t/2508) ### Example of a Custom Source Connector Here is an example of a custom source connector in Python that skips the first API call if the pre-fetched data is empty: python class SourceFoo(AbstractSource): def check_connection(self, logger, config) -> Tuple[bool, any]: resp = requests.get("FOO_URL", headers=headers) if resp.status_code == 200: return True, None else: return False, resp.text def streams(self, config: Mapping[str, Any]) -> List[Stream]: auth = TokenAuthenticator(token=config["token"], auth_method="Basic") ok, ids = self.get_ids(config["token"]) return [ Items(ids=ids, authenticator=auth), ] def get_ids(self, token) -> Tuple[bool, any]: resp = requests.get("FOO_URL", ...) if resp.status_code == 200: ids = resp.json()["data"] return True, [id for id in ids if condition] else: return False, resp.text class Items(HttpStream): url_base = "FOO_URL" primary_key = "id" def __init__(self, ids: list, **kwargs): super().__init__(**kwargs) if not(not ids): self.id = ids.pop(0) else: self.id = None self.ids = ids
[Source](https://discuss.airbyte.io/t/3383) ### Additional Resources - [Python CDK](https://docs.airbyte.com/connector-development/cdk-python/) - [Basic Concepts](https://docs.airbyte.com/connector-development/cdk-python/basic-concepts) - [Defining Stream Schemas](https://docs.airbyte.com/Silvia Giammarinaro
06/03/2024, 10:29 AMkapa.ai
06/03/2024, 10:29 AMkapa.ai
06/03/2024, 10:29 AMSilvia Giammarinaro
06/03/2024, 10:32 AMkapa.ai
06/03/2024, 10:32 AMkapa.ai
06/03/2024, 10:32 AMSilvia Giammarinaro
06/03/2024, 10:39 AMkapa.ai
06/03/2024, 10:40 AMkapa.ai
06/03/2024, 10:40 AMSilvia Giammarinaro
06/03/2024, 12:03 PMkapa.ai
06/03/2024, 12:03 PMkapa.ai
06/03/2024, 12:03 PM