I have the following usecase: I want to move data ...
# replication-ask-ai
m
I have the following usecase: I want to move data from Excel files in SharePoint and a PostgresDB to a MS SQL DB. In MS SQL I need to join everything, make some small transformations and write everything back to SharePoint. Since Airbyte has no SharePoint connector I used the Office 365 API to move the SharePoint data. But I would like to at least move the Postgres data via Airbyte. I have a local installation on one of our servers and have set up the necessary connection. If I run it via the frontend, the data arrives in MS SQL as expected. However, since my orchestrating piece of software for this whole process is my Python script, I would like to call Airbyte from within this script and tell the above described connection to run. This way, I can load the Excel files and Postgres in parallel and then start the transformation once everything was moved. I have a question about this: Is there a standard way how to remotely access Airbyte and tell it to run a sync?
k
Yes, there is a way to programmatically run a sync in Airbyte. You can use the Airbyte API to create connections and run syncs. This can be done using the web UI, octavia-cli, or by calling the public API. Here is the link to the API documentation: [Airbyte API Documentation](https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#servers) You can use this API to integrate Airbyte into your Python script and trigger the sync process programmatically. Please note that the information on how to use the API specifically within a Python script is not provided in the extracts. You might need to refer to the API documentation and Python's requests library documentation to understand how to make API calls from your Python script. Source: [Previous Support Conversation in Slack](https://airbytehq.slack.com/archives/C021JANJ6TY/p1673015450666659)
m
@kapa.ai Don't I need Airbyte cloud to use the airbyte API? I have a local installation on one of our servers and I don't plan to get Airbyte cloud. Does your proposed solution still work without Airbyte cloud?
k
Yes, you can use the Airbyte API with a local installation of Airbyte. You do not need to have Airbyte Cloud to use the API. The API is part of the open-source Airbyte and should work with your local installation. To deploy Airbyte locally, you need to install Docker on your workstation and then run the following commands in your terminal:
Copy code
bash git clone <https://github.com/airbytehq/airbyte.git>   cd airbyte   ./run-ab-platform.sh
Once you see an Airbyte banner, the UI is ready to go at http://localhost:8000. You will be asked for a username and password. By default, that's username
airbyte
and password
password
. Once you deploy Airbyte to your servers, be sure to change these in your
.env
file. For more details, you can refer to the [Local Deployment](https://docs.airbyte.com/deploying-airbyte/local-deployment) guide in the Airbyte documentation. Please note that while the API is available in the open-source version of Airbyte, certain features or support may only be available in the paid version of Airbyte Cloud. If you have specific requirements or questions, you may want to reach out to the Airbyte team for clarification. Sources: - [Deploy Airbyte](https://docs.airbyte.com/quickstart/deploy-airbyte) - [Local Deployment](https://docs.airbyte.com/deploying-airbyte/local-deployment) - [Previous Support Conversation in Slack](https://airbytehq.slack.com/archives/C021JANJ6TY/p1670847154714299)
m
@kapa.ai thanks a lot. I will try this. Currently struggling with the authentication. Is there a code reference for authentication?
k
A message from kapa.ai
Yes, there are several references to authentication in the provided extracts. The method of authentication depends on the specific service you are trying to connect to. Here are a few examples: 1. For AWS - Amazon ECR: You can authenticate to an ECR private registry using the
aws
CLI:
Copy code
bash aws ecr get-login-password --region region | docker login --username AWS --password-stdin <http://aws_account_id.dkr.ecr.region.amazonaws.com|aws_account_id.dkr.ecr.region.amazonaws.com>
You can find more details about this command and other available authentication methods [here](https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html). You will have to authenticate your local/CI environment (where you build your image) and your EC2 instance where your Airbyte instance is running. [source](https://docs.airbyte.com/operator-guides/using-custom-connectors) 2. For Docker-compose Airbyte deployments - Self hosted - Open Source Docker Registry: To authenticate your local/CI environment and Airbyte instance you can use the [
docker login
](https://docs.docker.com/engine/reference/commandline/login/) command. [source](https://docs.airbyte.com/operator-guides/using-custom-connectors) 3. For Docker-compose Airbyte deployments - On DockerHub - Repositories: You can use Docker Desktop to authenticate your local machine to your DockerHub registry by signing in on the desktop application using your DockerID. You need to use a [service account](https://docs.docker.com/docker-hub/service-accounts/) to authenticate your Airbyte instance to your DockerHub registry. [source](https://docs.airbyte.com/operator-guides/using-custom-connectors) 4. For specific connectors like Salesforce, Intercom, Notion: The authentication process is described in the setup guide for each connector. For example, for Salesforce, you need to enter your Client ID, Client Secret, and Refresh Token. [source](https://docs.airbyte.com/integrations/sources/salesforce) Please refer to the specific documentation for the service you are trying to connect to for more detailed instructions.
m
@kapa.ai I actually meant authentication using the Airbyte API 😉
k
A message from kapa.ai
Sure, I understand now. You're asking about authenticating when making requests to the Airbyte API. In the open-source version of Airbyte, you can use Basic Authentication when making API requests. Here's an example of how to do this with a
curl
command:
Copy code
bash curl --location --request POST '<http://localhost:8000/api/v1/connections/list>' \ --header 'Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=' \ --header 'Content-Type: application/json' \ --data-raw '{     "workspaceId": "095869d2-7fee-46bf-936e-300e6e827011" }'
In this example, the
Authorization
header is a base64 encoded
username:password
. You would replace
dXNlcm5hbWU6cGFzc3dvcmQ=
with the base64 encoding of your own
username:password
. Please note that the username and password are the ones you set up during the Airbyte deployment. By default, the username is
airbyte
and the password is
password
. Be sure to change these in your
.env
file when you deploy Airbyte to your servers. You can find more details about this in the [previous support conversation in Slack](https://airbytehq.slack.com/archives/C021JANJ6TY/p1673554354932049).