Hi <@U04S8PLJWMT>, can you help me to setup airbyt...
# ask-ai
k
Hi @kapa.ai, can you help me to setup airbyte setup locally so I test the code changes
k
You can set up a local Airbyte Core environment with
abctl
, then build/test your code changes against it. --- ## 1. Prerequisites for local development For platform development, you need at least: [[Dev locally 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] - Java 21 - Node 20 - Python 3.10 - Docker -
jq
To run Airbyte locally via
abctl
, you also need Docker Desktop installed and running. [[Quickstart](https://docs.airbyte.com/platform/using-airbyte/getting-started/oss-quickstart#part-1-install-docker-desktop)] If you’re building connectors, also install
airbyte-ci
following its README. [[Dev locally 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] --- ## 2. Install and run Airbyte locally with
abctl
Follow the OSS Quickstart to install
abctl
(Part 2 in that guide), then run Airbyte. [[Quickstart](https://docs.airbyte.com/platform/using-airbyte/getting-started/oss-quickstart)] To run locally on `localhost`:
Copy code
bash abctl local install
This deploys Airbyte into a local Kubernetes cluster (kind) created by
abctl
using Docker. [[Quickstart Part 3](https://docs.airbyte.com/platform/using-airbyte/getting-started/oss-quickstart#part-3-run-airbyte)] When installation finishes, Airbyte will be available at: -
<http://localhost:8000>
(or the host you specify with
--host
). [[Quickstart Part 3](https://docs.airbyte.com/platform/using-airbyte/getting-started/oss-quickstart#part-3-run-airbyte)] You can also: - Expose over HTTP with a hostname:
Copy code
bash   abctl local install --host <http://example.com|example.com>
- Expose over HTTP without HTTPS, disabling secure cookies:
Copy code
bash   abctl local install --host <http://example.com|example.com> --insecure-cookies
- Run in low-resource mode (<4 CPUs, disables Connector Builder):
Copy code
bash   abctl local install --low-resource-mode
[[Quickstart Part 3](https://docs.airbyte.com/platform/using-airbyte/getting-started/oss-quickstart#part-3-run-airbyte)] If you see a
Readiness probe failed: HTTP probe failed with statuscode: 503
warning, you can let installation continue and consider allocating more resources. [[Quickstart Part 3](https://docs.airbyte.com/platform/using-airbyte/getting-started/oss-quickstart#part-3-run-airbyte)] --- ## 3. Inspect and debug the local
abctl
cluster
abctl
uses kind (Kubernetes-in-Docker) and Helm to deploy Airbyte. You can use
kubectl
to inspect pods. [[Dev locally 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] Configure
kubectl
for the
abctl
cluster:
Copy code
bash kind export kubeconfig -n airbyte-abctl
List pods:
Copy code
bash kubectl get pods -n airbyte-abctl
View logs for a specific pod (example server pod name):
Copy code
bash kubectl logs -n airbyte-abctl airbyte-abctl-server-74465db7fd-gk25q
[[Dev locally 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] --- ## 4. Building the platform code and using your images with
abctl
If you’re changing Airbyte Platform code (
airbyte-platform
repo): [[Platform contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#platform-contributions)] 1. Fork and clone the platform repo:
Copy code
bash    git clone [email protected]:{YOUR_USERNAME}/airbyte-platform.git    cd airbyte-platform
2. Check
abctl
status:
Copy code
bash    abctl local status
3. Build with Gradle:
Copy code
bash    ./gradlew build
This builds all code and Docker images (JARs + images) so you can run Airbyte locally. [[Platform contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#platform-contributions)] 4. Load your dev image into the
abctl
kind cluster
, for example for the server:
Copy code
bash    kind load docker-image airbyte/server:dev --name airbyte-abctl
5. In your
values.yaml
, point the component to the
dev
tag:
Copy code
yaml    server:      image:        tag: dev
6. Redeploy Airbyte with these values:
Copy code
bash    abctl local install --values values.yaml
Now your running Airbyte instance should be using your locally built images, so you can manually test your code changes via the UI or API. [[Platform contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#platform-contributions)] --- ## 5. (Optional) Run platform acceptance tests To run end-to-end platform tests against a test-container-managed deployment: [[Platform contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#run-platform-acceptance-tests)]
Copy code
bash ./gradlew clean build ./gradlew :oss:airbyte-tests:acceptanceTests
To run them against an existing external (e.g. your
abctl
) deployment:
Copy code
bash USE_EXTERNAL_DEPLOYMENT=true ./gradlew :oss:airbyte-tests:acceptanceTests
--- If you tell me whether you’re modifying platform code, connectors, or docs, I can narrow this down to an exact minimal command sequence for your case.
k
how to debug and test then on local machine the changes
@kapa.ai how to debug and test airbyte code changes on local machine
k
A message from kapa.ai
Here’s how you can debug and test Airbyte code changes locally, depending on what you’re modifying. --- ## 1. Prerequisites Install: Java 21, Node 20, Python 3.10, Docker,
jq
, and
airbyte-ci
(if you’re building connectors). [[Dev prereqs 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] Install and set up
abctl
following the OSS Quickstart;
abctl
uses a local kind cluster + Helm to run Airbyte. [[Dev prereqs 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] --- ## 2. Run Airbyte locally with
abctl
Install Airbyte locally:
Copy code
bash abctl local install
This deploys Airbyte into a local kind cluster; you then access the UI at
<http://localhost:8000>
. [[Dev prereqs 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] To inspect/debug pods:
Copy code
bash kind export kubeconfig -n airbyte-abctl kubectl get pods -n airbyte-abctl kubectl logs -n airbyte-abctl airbyte-abctl-server-<pod-id>
[[Dev prereqs 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] --- ## 3. Debugging JVM Docker containers (platform or JVM connectors) To attach a JVM debugger (e.g., IntelliJ) to a Docker container: 1. In the Airbyte repo, use the debug compose extension. The
docker-compose.debug.yaml
sets:
Copy code
yaml    server:      environment:        - JAVA_TOOL_OPTIONS=${DEBUG_SERVER_JAVA_OPTIONS}
2. Start Docker Compose with debug options:
Copy code
bash    DEBUG_SERVER_JAVA_OPTIONS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005" \    VERSION="dev" \    docker compose -f docker-compose.yaml -f docker-compose.debug.yaml up
3. Configure a Remote JVM Debug in your IDE pointing to port
5005
and attach. [[Debugging Docker](https://docs.airbyte.com/platform/connector-development/debugging-docker)] On macOS you may need Docker Mac Connect to reach containers by IP. [[Debugging Docker](https://docs.airbyte.com/platform/connector-development/debugging-docker)] --- ## 4. If you’re changing connectors ### 4.1 Build the connector image From your
airbyte
repo fork, build the connector:
Copy code
bash airbyte-ci connectors --name source-<source-name> build
This produces a local Docker image:
Copy code
text airbyte/source-<source-name>:dev
Verify it:
Copy code
bash docker images | grep airbyte/source-<source-name>:dev
[[Connector contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#connector-contributions)] ### 4.2 Load into the
abctl
cluster
Copy code
bash kind load docker-image airbyte/source-<source-name>:dev -n airbyte-abctl
[[Connector contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#connector-contributions)] ### 4.3 Use your image from the UI and test In the Airbyte UI: - Go to
<http://localhost:8000>
-
Settings
(gear icon) →
Sources
or
Destinations
- For the connector, set the version to your tag (default
dev
) - Click Change Now any sync using that connector will run your modified image. [[Connector contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#connector-contributions)] #### Refreshing spec changes If you changed
spec.json
, Airbyte caches specs. To refresh: - Same settings screen - Leave version as
dev
- Click Change again to refresh the dev connector spec. [[Connector contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#connector-contributions)] You can also use the same local image as a Custom connector from the UI (build locally, then add as custom); this is confirmed in the community forum. [[Forum connector test](https://discuss.airbyte.io/t/1812)] --- ## 5. If you’re changing platform code (
airbyte-platform
) ### 5.1 Build platform images From your
airbyte-platform
fork:
Copy code
bash git clone [email protected]:{YOUR_USERNAME}/airbyte-platform.git cd airbyte-platform  abctl local status   # ensure abctl is healthy  ./gradlew build
This builds all platform code and Docker images (e.g.,
airbyte/server:dev
). [[Platform contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#platform-contributions)] ### 5.2 Load your images into
abctl
and redeploy Example for the server:
Copy code
bash kind load docker-image airbyte/server:dev --name airbyte-abctl
Adjust your `values.yaml`:
Copy code
yaml server:   image:     tag: dev
Then redeploy:
Copy code
bash abctl local install --values values.yaml
Your running instance will now use your modified platform images, and you can test behavior via the UI/API. [[Platform contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#using-the-images-in-abctl)] --- ## 6. Optional: Run platform acceptance tests To run end‑to‑end platform tests:
Copy code
bash ./gradlew clean build ./gradlew :oss:airbyte-tests:acceptanceTests
To run them against an existing external deployment (e.g., your
abctl
instance):
Copy code
bash USE_EXTERNAL_DEPLOYMENT=true ./gradlew :oss:airbyte-tests:acceptanceTests
[[Platform contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#run-platform-acceptance-tests)] --- If you tell me whether you’re editing a specific connector or the core platform, I can give you a short, step‑by‑step command list tailored to your case.