<@U04S8PLJWMT> how to work now with custom connect...
# replication-ask-ai
m
@kapa.ai how to work now with custom connectors buit locally and airbyte 2.0? it seems that using the local image name when creating a new source does not work anymore, i guess it's because the kubernetes cluster abctl creates can't resolver the image name to the local docker server instance
k
New message text here
m
@kapa.ai how to have my local abctl instance authenticate against an external docker registry
k
A message from kapa.ai
To have an
abctl
(Airbyte 2.0) instance authenticate against an external/private Docker registry, you configure the Kubernetes cluster Airbyte runs on to use your registry credentials. There are two main patterns in the docs: --- ## 1. Global custom image registry (Helm / values.yaml) If you want Airbyte to pull all platform and connector images from a specific registry: 1. Create a Kubernetes secret with your Docker config:
Copy code
bash    kubectl create secret generic regcred \      --from-file=.dockerconfigjson=<path/to/.docker/config.json> \      --type=<http://kubernetes.io/dockerconfigjson|kubernetes.io/dockerconfigjson>
2. Reference that secret and registry in Airbyte’s `values.yaml`:
Copy code
yaml    global:      image:        registry: <http://ghcr.io/NAMESPACE|ghcr.io/NAMESPACE>      imagePullSecrets:        - name: regcred
This tells the cluster (including the
abctl
-managed Airbyte) to use
<http://ghcr.io/NAMESPACE|ghcr.io/NAMESPACE>
and authenticate with
regcred
when pulling images. [[Custom registry step 1](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/custom-image-registries#step-1-customize-airbyte-to-use-your-image-registry); [Enterprise custom registry](https://docs.airbyte.com/platform/2.0/enterprise-setup/implementation-guide#configure-a-custom-image-registry)] > Note: Custom Docker connectors that specify a fully qualified domain name (e.g. `example.com/airbyte/my-connector`) will bypass this global registry setting and pull directly from the FQDN. [[Custom registry](https://docs.airbyte.com/platform/2.0/deploying-airbyte/integrations/custom-image-registries#custom-connectors-using-fully-qualified-domain-names)] --- ## 2. Authenticate worker pods to a private registry (Kubernetes env var) If you only need Airbyte workers to pull connector images from a private registry: 1. Create a Kubernetes
Secret
with your registry credentials (same as above, or as per Kubernetes docs). [[Docker requirements 2.0](https://docs.airbyte.com/platform/2.0/operator-guides/using-custom-connectors#3-adhere-to-airbytes-docker-image-requirements)] 2. Set the
JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET
environment variable on the
airbyte-worker
pod to the name of that Secret: > “Set the
JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET
environment variable on the
airbyte-worker
pod. The value must be the name of your previously created Kubernetes Secret.” [[Docker requirements 2.0](https://docs.airbyte.com/platform/2.0/operator-guides/using-custom-connectors#3-adhere-to-airbytes-docker-image-requirements)] This makes the worker use that secret when pulling connector images from your external registry. --- The docs do not show an
abctl
-specific flag or command to inject these settings automatically; you need to apply the Kubernetes secret and configuration (values.yaml or env var) to the cluster where your
abctl
instance is running.
m
@kapa.ai i have loaded my image into kind ~ kind load docker-image span/airbyte-source-bitbucket:dev -n airbyte-abctl Image: "span/airbyte-source-bitbucket:dev" with ID "sha256:ecce4354eeb12e521c22cb7489ec8d59d1dae7908907da29f383104c54a1dc1a" not yet present on node "airbyte-abctl-control-plane", loading... but i'm still getting a 502 error (upstream timeout) when creating a connector for it, what else could be wrong? where can i find more information? the airbyte servers don't show any errors
k
A message from kapa.ai
The knowledge sources don’t explain 502 / upstream timeout causes in detail, but they do point to where to look and what to double‑check. Since you already loaded the image into kind:
Copy code
bash kind load docker-image span/airbyte-source-bitbucket:dev -n airbyte-abctl
the next likely issues are: 1. Image name / tag mismatch in the UI Make sure the connector in Airbyte is actually using `span/airbyte-source-bitbucket:dev`: - Go to
<http://localhost:8000/>
- Workspace settings → Sources - Edit your custom connector (or add a new Docker connector) - Set: - Full image name:
span/airbyte-source-bitbucket
- Tag:
dev
- Save and retry. [[Use custom connector](https://docs.airbyte.com/platform/2.0/operator-guides/using-custom-connectors#4-use-your-custom-docker-connector-in-airbyte)] 2. Connector image not reachable from worker / kind For `abctl`/kind, loading the image is the right step; if you see errors like “image not found” in logs, it usually means the image wasn’t properly loaded. The docs say that if you see errors about the image, it’s likely not loaded correctly and you should re‑run:
Copy code
bash    kind load docker-image span/airbyte-source-bitbucket:dev -n airbyte-abctl
[[Kind loading 2.0](https://docs.airbyte.com/platform/2.0/operator-guides/using-custom-connectors#loading-connector-docker-containers-into-kind)] 3. Check worker / job pod logs (not just server logs) The 502 is often coming from the UI/server proxying to the worker job. To see what’s actually failing: - Export kubeconfig for the
abctl
cluster:
Copy code
bash      kind export kubeconfig -n airbyte-abctl
- List pods:
Copy code
bash      kubectl get pods -n airbyte-abctl
- Look for: -
airbyte-abctl-worker-...
(main worker) - Any short‑lived job pods created when you click “Set up” / “Get spec” - Get logs, for example:
Copy code
bash      kubectl logs -n airbyte-abctl airbyte-abctl-worker-XXXX      kubectl logs -n airbyte-abctl <job-pod-name>
The docs explicitly recommend using
kubectl
against the
airbyte-abctl
namespace to debug pods and logs. [[Developing locally 2.0](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#using-abctl-for-airbyte-development)] 4. Connector spec caching (if you changed the spec) If you’re iterating on the connector and changed its spec, Airbyte may still be using a cached spec. To refresh: - Go to Workspace settings → Sources - Edit the connector - Keep the tag (e.g.
dev
) and click Change to force a spec refresh. [[Connector spec cache](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-specification-caching)] --- If after checking the image name/tag and looking at the worker/job pod logs you still only see a 502 with no useful error, the knowledge sources don’t provide more specific 502 troubleshooting steps. In that case, the next step would be to inspect those pod logs for Python/Java stack traces or connector‑level errors and share them (e.g. in the Airbyte forum) for further analysis.