kanchalkumar karale
12/03/2025, 4:02 PMkapa.ai
12/03/2025, 4:02 PMabctl, then build/test your code changes against it. --- ## 1. Prerequisites for local development For platform development, you need at least: [[Dev locally 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] - Java 21 - Node 20 - Python 3.10 - Docker - jq To run Airbyte locally via abctl, you also need Docker Desktop installed and running. [[Quickstart](https://docs.airbyte.com/platform/using-airbyte/getting-started/oss-quickstart#part-1-install-docker-desktop)] If you’re building connectors, also install airbyte-ci following its README. [[Dev locally 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] --- ## 2. Install and run Airbyte locally with abctl Follow the OSS Quickstart to install abctl (Part 2 in that guide), then run Airbyte. [[Quickstart](https://docs.airbyte.com/platform/using-airbyte/getting-started/oss-quickstart)] To run locally on `localhost`: bash abctl local install This deploys Airbyte into a local Kubernetes cluster (kind) created by abctl using Docker. [[Quickstart Part 3](https://docs.airbyte.com/platform/using-airbyte/getting-started/oss-quickstart#part-3-run-airbyte)] When installation finishes, Airbyte will be available at: - <http://localhost:8000> (or the host you specify with --host). [[Quickstart Part 3](https://docs.airbyte.com/platform/using-airbyte/getting-started/oss-quickstart#part-3-run-airbyte)] You can also: - Expose over HTTP with a hostname: bash abctl local install --host <http://example.com|example.com> - Expose over HTTP without HTTPS, disabling secure cookies: bash abctl local install --host <http://example.com|example.com> --insecure-cookies - Run in low-resource mode (<4 CPUs, disables Connector Builder): bash abctl local install --low-resource-mode [[Quickstart Part 3](https://docs.airbyte.com/platform/using-airbyte/getting-started/oss-quickstart#part-3-run-airbyte)] If you see a Readiness probe failed: HTTP probe failed with statuscode: 503 warning, you can let installation continue and consider allocating more resources. [[Quickstart Part 3](https://docs.airbyte.com/platform/using-airbyte/getting-started/oss-quickstart#part-3-run-airbyte)] --- ## 3. Inspect and debug the local abctl cluster abctl uses kind (Kubernetes-in-Docker) and Helm to deploy Airbyte. You can use kubectl to inspect pods. [[Dev locally 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] Configure kubectl for the abctl cluster: bash kind export kubeconfig -n airbyte-abctl List pods: bash kubectl get pods -n airbyte-abctl View logs for a specific pod (example server pod name): bash kubectl logs -n airbyte-abctl airbyte-abctl-server-74465db7fd-gk25q [[Dev locally 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] --- ## 4. Building the platform code and using your images with abctl If you’re changing Airbyte Platform code (airbyte-platform repo): [[Platform contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#platform-contributions)] 1. Fork and clone the platform repo: bash git clone [email protected]:{YOUR_USERNAME}/airbyte-platform.git cd airbyte-platform 2. Check abctl status: bash abctl local status 3. Build with Gradle: bash ./gradlew build This builds all code and Docker images (JARs + images) so you can run Airbyte locally. [[Platform contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#platform-contributions)] 4. Load your dev image into the abctl kind cluster, for example for the server: bash kind load docker-image airbyte/server:dev --name airbyte-abctl 5. In your values.yaml, point the component to the dev tag: yaml server: image: tag: dev 6. Redeploy Airbyte with these values: bash abctl local install --values values.yaml Now your running Airbyte instance should be using your locally built images, so you can manually test your code changes via the UI or API. [[Platform contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#platform-contributions)] --- ## 5. (Optional) Run platform acceptance tests To run end-to-end platform tests against a test-container-managed deployment: [[Platform contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#run-platform-acceptance-tests)] bash ./gradlew clean build ./gradlew :oss:airbyte-tests:acceptanceTests To run them against an existing external (e.g. your abctl) deployment: bash USE_EXTERNAL_DEPLOYMENT=true ./gradlew :oss:airbyte-tests:acceptanceTests --- If you tell me whether you’re modifying platform code, connectors, or docs, I can narrow this down to an exact minimal command sequence for your case.kanchalkumar karale
12/03/2025, 4:05 PMkanchalkumar karale
12/03/2025, 4:12 PMkapa.ai
12/03/2025, 4:13 PMkapa.ai
12/03/2025, 4:13 PMjq, and airbyte-ci (if you’re building connectors). [[Dev prereqs 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] Install and set up abctl following the OSS Quickstart; abctl uses a local kind cluster + Helm to run Airbyte. [[Dev prereqs 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] --- ## 2. Run Airbyte locally with abctl Install Airbyte locally: bash abctl local install This deploys Airbyte into a local kind cluster; you then access the UI at <http://localhost:8000>. [[Dev prereqs 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] To inspect/debug pods: bash kind export kubeconfig -n airbyte-abctl kubectl get pods -n airbyte-abctl kubectl logs -n airbyte-abctl airbyte-abctl-server-<pod-id> [[Dev prereqs 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] --- ## 3. Debugging JVM Docker containers (platform or JVM connectors) To attach a JVM debugger (e.g., IntelliJ) to a Docker container: 1. In the Airbyte repo, use the debug compose extension. The docker-compose.debug.yaml sets: yaml server: environment: - JAVA_TOOL_OPTIONS=${DEBUG_SERVER_JAVA_OPTIONS} 2. Start Docker Compose with debug options: bash DEBUG_SERVER_JAVA_OPTIONS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005" \ VERSION="dev" \ docker compose -f docker-compose.yaml -f docker-compose.debug.yaml up 3. Configure a Remote JVM Debug in your IDE pointing to port 5005 and attach. [[Debugging Docker](https://docs.airbyte.com/platform/connector-development/debugging-docker)] On macOS you may need Docker Mac Connect to reach containers by IP. [[Debugging Docker](https://docs.airbyte.com/platform/connector-development/debugging-docker)] --- ## 4. If you’re changing connectors ### 4.1 Build the connector image From your airbyte repo fork, build the connector: bash airbyte-ci connectors --name source-<source-name> build This produces a local Docker image: text airbyte/source-<source-name>:dev Verify it: bash docker images | grep airbyte/source-<source-name>:dev [[Connector contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#connector-contributions)] ### 4.2 Load into the abctl cluster bash kind load docker-image airbyte/source-<source-name>:dev -n airbyte-abctl [[Connector contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#connector-contributions)] ### 4.3 Use your image from the UI and test In the Airbyte UI: - Go to <http://localhost:8000> - Settings (gear icon) → Sources or Destinations - For the connector, set the version to your tag (default dev) - Click Change Now any sync using that connector will run your modified image. [[Connector contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#connector-contributions)] #### Refreshing spec changes If you changed spec.json, Airbyte caches specs. To refresh: - Same settings screen - Leave version as dev - Click Change again to refresh the dev connector spec. [[Connector contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#connector-contributions)] You can also use the same local image as a Custom connector from the UI (build locally, then add as custom); this is confirmed in the community forum. [[Forum connector test](https://discuss.airbyte.io/t/1812)] --- ## 5. If you’re changing platform code (airbyte-platform) ### 5.1 Build platform images From your airbyte-platform fork: bash git clone [email protected]:{YOUR_USERNAME}/airbyte-platform.git cd airbyte-platform abctl local status # ensure abctl is healthy ./gradlew build This builds all platform code and Docker images (e.g., airbyte/server:dev). [[Platform contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#platform-contributions)] ### 5.2 Load your images into abctl and redeploy Example for the server: bash kind load docker-image airbyte/server:dev --name airbyte-abctl Adjust your `values.yaml`: yaml server: image: tag: dev Then redeploy: bash abctl local install --values values.yaml Your running instance will now use your modified platform images, and you can test behavior via the UI/API. [[Platform contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#using-the-images-in-abctl)] --- ## 6. Optional: Run platform acceptance tests To run end‑to‑end platform tests: bash ./gradlew clean build ./gradlew :oss:airbyte-tests:acceptanceTests To run them against an existing external deployment (e.g., your abctl instance): bash USE_EXTERNAL_DEPLOYMENT=true ./gradlew :oss:airbyte-tests:acceptanceTests [[Platform contrib 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#run-platform-acceptance-tests)] --- If you tell me whether you’re editing a specific connector or the core platform, I can give you a short, step‑by‑step command list tailored to your case.