Jake Duckworth
07/18/2024, 3:26 PMabctl local install
command for the first time. What can I do to correct this issue?
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x2980c36]
goroutine 1 [running]:
<http://github.com/airbytehq/abctl/internal/cmd/local.portAvailable.func1()|github.com/airbytehq/abctl/internal/cmd/local.portAvailable.func1()>
<http://github.com/airbytehq/abctl/internal/cmd/local/check.go:86|github.com/airbytehq/abctl/internal/cmd/local/check.go:86> +0x16
<http://github.com/airbytehq/abctl/internal/cmd/local.portAvailable({0x322c980|github.com/airbytehq/abctl/internal/cmd/local.portAvailable({0x322c980>, 0xc00038a190}, 0x1f40)
<http://github.com/airbytehq/abctl/internal/cmd/local/check.go:90|github.com/airbytehq/abctl/internal/cmd/local/check.go:90> +0x37f
<http://github.com/airbytehq/abctl/internal/cmd/local.NewCmdInstall.func1(0xc00061a908|github.com/airbytehq/abctl/internal/cmd/local.NewCmdInstall.func1(0xc00061a908>, {0x2986ddc?, 0x4?, 0x2986dc0?})
<http://github.com/airbytehq/abctl/internal/cmd/local/local_install.go:66|github.com/airbytehq/abctl/internal/cmd/local/local_install.go:66> +0x36e
<http://github.com/spf13/cobra.(*Command).execute(0xc00061a908|github.com/spf13/cobra.(*Command).execute(0xc00061a908>, {0x4389c60, 0x0, 0x0})
<http://github.com/spf13/cobra@v1.8.0/command.go:968|github.com/spf13/cobra@v1.8.0/command.go:968> +0xa44
<http://github.com/spf13/cobra.(*Command).ExecuteC(0xc00061a008)|github.com/spf13/cobra.(*Command).ExecuteC(0xc00061a008)>
<http://github.com/spf13/cobra@v1.8.0/command.go:1115|github.com/spf13/cobra@v1.8.0/command.go:1115> +0x3ff
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
<http://github.com/spf13/cobra@v1.8.0/command.go:1039|github.com/spf13/cobra@v1.8.0/command.go:1039>
<http://github.com/spf13/cobra.(*Command).ExecuteContext(...)|github.com/spf13/cobra.(*Command).ExecuteContext(...)>
<http://github.com/spf13/cobra@v1.8.0/command.go:1032|github.com/spf13/cobra@v1.8.0/command.go:1032>
<http://github.com/airbytehq/abctl/internal/cmd.Execute({0x322c980|github.com/airbytehq/abctl/internal/cmd.Execute({0x322c980>?, 0xc00038a190?}, 0x77359400?)
<http://github.com/airbytehq/abctl/internal/cmd/cmd.go:41|github.com/airbytehq/abctl/internal/cmd/cmd.go:41> +0x4f
main.main()
<http://github.com/airbytehq/abctl/main.go:45|github.com/airbytehq/abctl/main.go:45> +0x1a7
kapa.ai
07/18/2024, 3:26 PMabctl local install
on Mac systems ([source](https://github.com/airbytehq/airbyte/issues/40745)). In some cases, the issue was introduced in a specific version of abctl
and was not present in earlier versions. Here are a few steps you can take to troubleshoot this issue: 1. Check your abctl
version: Run abctl version
to check the version of abctl
you're using. If you're using a version that's known to have this issue (e.g., v0.5.0 or later), you might consider downgrading to an earlier version. 2. Reinstall abctl
: Uninstall and then reinstall abctl
using the instructions provided in the [Airbyte documentation](https://docs.airbyte.com/using-airbyte/getting-started/oss-quickstart#1-install-abctl). This can help ensure that you have a clean, up-to-date installation of abctl
. 3. Check your system's configuration: Make sure that your system meets the requirements for running abctl
. This includes having a compatible operating system and processor architecture, as well as having Docker installed and properly configured. If none of these steps resolve the issue, I would recommend reaching out to the Airbyte community for further assistance. You can do this by opening a new issue in the [Airbyte GitHub repository](https://github.com/airbytehq/airbyte/issues). Be sure to include all relevant information, including the error message you're seeing and the steps you've taken to troubleshoot the issue.Jake Duckworth
07/18/2024, 3:53 PMkapa.ai
07/18/2024, 3:54 PMJake Duckworth
07/18/2024, 3:58 PMkapa.ai
07/18/2024, 3:58 PMkapa.ai
07/18/2024, 3:58 PMCole Snodgrass (Airbyte)
07/18/2024, 6:32 PMJake Duckworth
07/18/2024, 7:10 PM(⎈|docker-desktop:default)user@machine airbyte % abctl local install
INFO Using Kubernetes provider:
Provider: kind
Kubeconfig: /Users/jakeduckworth/.airbyte/abctl/abctl.kubeconfig
Context: kind-airbyte-abctl
SUCCESS Found Docker installation: version 27.0.3
SUCCESS Port 8000 appears to be available
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x7b03c36]
goroutine 1 [running]:
<http://github.com/airbytehq/abctl/internal/cmd/local.portAvailable.func1()|github.com/airbytehq/abctl/internal/cmd/local.portAvailable.func1()>
<http://github.com/airbytehq/abctl/internal/cmd/local/check.go:86|github.com/airbytehq/abctl/internal/cmd/local/check.go:86> +0x16
<http://github.com/airbytehq/abctl/internal/cmd/local.portAvailable({0x83af980|github.com/airbytehq/abctl/internal/cmd/local.portAvailable({0x83af980>, 0xc000558460}, 0x1f40)
<http://github.com/airbytehq/abctl/internal/cmd/local/check.go:90|github.com/airbytehq/abctl/internal/cmd/local/check.go:90> +0x37f
<http://github.com/airbytehq/abctl/internal/cmd/local.NewCmdInstall.func1(0xc000489508|github.com/airbytehq/abctl/internal/cmd/local.NewCmdInstall.func1(0xc000489508>, {0x7b09ddc?, 0x4?, 0x7b09dc0?})
<http://github.com/airbytehq/abctl/internal/cmd/local/local_install.go:66|github.com/airbytehq/abctl/internal/cmd/local/local_install.go:66> +0x36e
<http://github.com/spf13/cobra.(*Command).execute(0xc000489508|github.com/spf13/cobra.(*Command).execute(0xc000489508>, {0x950cc60, 0x0, 0x0})
<http://github.com/spf13/cobra@v1.8.0/command.go:968|github.com/spf13/cobra@v1.8.0/command.go:968> +0xa44
<http://github.com/spf13/cobra.(*Command).ExecuteC(0xc000488608)|github.com/spf13/cobra.(*Command).ExecuteC(0xc000488608)>
<http://github.com/spf13/cobra@v1.8.0/command.go:1115|github.com/spf13/cobra@v1.8.0/command.go:1115> +0x3ff
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
<http://github.com/spf13/cobra@v1.8.0/command.go:1039|github.com/spf13/cobra@v1.8.0/command.go:1039>
<http://github.com/spf13/cobra.(*Command).ExecuteContext(...)|github.com/spf13/cobra.(*Command).ExecuteContext(...)>
<http://github.com/spf13/cobra@v1.8.0/command.go:1032|github.com/spf13/cobra@v1.8.0/command.go:1032>
<http://github.com/airbytehq/abctl/internal/cmd.Execute({0x83af980|github.com/airbytehq/abctl/internal/cmd.Execute({0x83af980>?, 0xc000558460?}, 0x77359400?)
<http://github.com/airbytehq/abctl/internal/cmd/cmd.go:41|github.com/airbytehq/abctl/internal/cmd/cmd.go:41> +0x4f
main.main()
<http://github.com/airbytehq/abctl/main.go:45|github.com/airbytehq/abctl/main.go:45> +0x1a7
Cole Snodgrass (Airbyte)
07/18/2024, 7:11 PMCole Snodgrass (Airbyte)
07/18/2024, 7:12 PMabctl
(port 8000
) and it already being in use on your host. If you specify a different port (via --port
), I think you’ll be able to get around this panic (until this PR if merged and returns a friendlier error).Jake Duckworth
07/18/2024, 7:54 PMairbyte-abctl
cluster. Here is the full error:
(⎈|docker-desktop:default)user@machine airbyte % abctl local install --port=9000
INFO Using Kubernetes provider:
Provider: kind
Kubeconfig: /Users/jakeduckworth/.airbyte/abctl/abctl.kubeconfig
Context: kind-airbyte-abctl
SUCCESS Found Docker installation: version 27.0.3
SUCCESS Port 9000 appears to be available
INFO No existing cluster found, cluster 'airbyte-abctl' will be created
ERROR Cluster 'airbyte-abctl' could not be created
ERROR unable to create kind cluster: command "docker run --name airbyte-abctl-control-plane --hostname airbyte-abctl-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --label io.x-k8s.kind.cluster=airbyte-abctl --net kind --restart=on-failure:1 --init=false --cgroupns=private --volume=/Users/jakeduckworth/.airbyte/abctl/data:/var/local-path-provisioner --publish=0.0.0.0:9000:80/TCP --publish=127.0.0.1:53968:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8" failed with error: exit status 126
kapa.ai
07/18/2024, 7:54 PMkapa.ai
07/18/2024, 7:54 PMairbyte-abctl
cluster using Docker has failed. This could be due to a variety of reasons such as Docker not running, insufficient permissions, or issues with the Docker command itself. Here are a few things you can try: 1. Ensure Docker is running: You can check this by running docker info
in your terminal. If Docker is not running, you'll need to start it. 2. Check Docker permissions: Ensure that the user running the abctl local install
command has the necessary permissions to execute Docker commands. 3. Check Docker command: The error message shows that the Docker command to create the cluster has failed. You could try running the command separately in your terminal to see if it provides any additional error information. If none of these steps resolve your issue, I would recommend reporting this as a bug on the [Airbyte GitHub issues page](https://github.com/airbytehq/airbyte/issues) with the type of 🐛 [abctl] Report an issue with the abctl tool
. Be sure to include all relevant information including the error message, your operating system, Docker version, and abctl
version. For additional help, you can also check out the [Airbyte community support channels](https://docs.airbyte.com/using-airbyte/getting-started/oss-quickstart#troubleshooting) such as the GitHub Airbyte Forum, Airbyte Knowledge Base, and the Community Slack.Jake Duckworth
07/18/2024, 8:03 PMdocker-compose
version installed before. I did verify that there were no containers, images or volumes left over from that installation in case there was some conflict there.Cole Snodgrass (Airbyte)
07/18/2024, 8:15 PMdocker run --name airbyte-abctl-control-plane --hostname airbyte-abctl-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --label io.x-k8s.kind.cluster=airbyte-abctl --net kind --restart=on-failure:1 --init=false --cgroupns=private --volume=/Users/jakeduckworth/.airbyte/abctl/data:/var/local-path-provisioner --publish=0.0.0.0:9000:80/TCP --publish=127.0.0.1:53968:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8
Jake Duckworth
07/18/2024, 8:19 PMdocker: Error response from daemon: error while creating mount source path '/host_mnt/Users/jakeduckworth/.airbyte/abctl/data': mkdir /host_mnt/Users/jakeduckworth/.airbyte/abctl: permission denied.
Jake Duckworth
07/18/2024, 8:20 PMCole Snodgrass (Airbyte)
07/18/2024, 8:21 PM/host_mnt/Users/jakeduckworth/
the correct user-home for the user running abctl
?Jake Duckworth
07/18/2024, 8:23 PM% echo $HOME
/Users/jakeduckworth
Cole Snodgrass (Airbyte)
07/18/2024, 8:24 PMCole Snodgrass (Airbyte)
07/18/2024, 8:27 PMgo
installed on your machine?Jake Duckworth
07/18/2024, 8:30 PMbrew install abctl
command installed it:
% brew info go
==> go: stable 1.22.5 (bottled), HEAD
Open source programming language to build simple/reliable/efficient software
<https://go.dev/>
Installed
/usr/local/Cellar/go/1.22.5 (12,869 files, 254.2MB) *
Poured from bottle using the formulae.brew.sh API on 2024-07-18 at 14:58:50
From: <https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/g/go.rb>
License: BSD-3-Clause
==> Options
--HEAD
Install HEAD version
==> Analytics
install: 86,189 (30 days), 260,134 (90 days), 1,101,610 (365 days)
install-on-request: 61,329 (30 days), 184,329 (90 days), 808,835 (365 days)
build-error: 147 (30 days)
Cole Snodgrass (Airbyte)
07/18/2024, 8:34 PM/tmp/home.go
(or where-ever your tmp directory is located) and put the following code into that file
package main
import (
"fmt"
"os"
)
func main() {
h, _ := os.UserHomeDir()
fmt.Println(h)
}
Then from the command line run go run /tmp/home.go
and tell me what it returns.Jake Duckworth
07/18/2024, 8:36 PM% go run ./home.go
/Users/jakeduckworth
Cole Snodgrass (Airbyte)
07/18/2024, 8:37 PMCole Snodgrass (Airbyte)
07/18/2024, 8:38 PM/host_mnt
prefix is coming from.Cole Snodgrass (Airbyte)
07/18/2024, 8:43 PMmkdir -p ~/.airbyte/abctl/data
chmod 777 ~/.airbyte/abctl/data
Jake Duckworth
07/18/2024, 8:59 PMINFO Using Kubernetes provider:
Provider: kind
Kubeconfig: /Users/jakeduckworth/.airbyte/abctl/abctl.kubeconfig
Context: kind-airbyte-abctl
SUCCESS Found Docker installation: version 27.0.3
SUCCESS Port 9000 appears to be available
SUCCESS Existing cluster 'airbyte-abctl' found
WARNING Unable to determine which port the existing cluster was configured to use.
Installation will continue but may ultimately fail, in which case it will be necessarily to uninstall first.
SUCCESS Cluster 'airbyte-abctl' validation complete
ERROR Failed to initialize 'local' command
ERROR could not initialize local command: error communicating with kubernetes: could not create rest config: stat /Users/jakeduckworth/.airbyte/abctl/abctl.kubeconfig: no such file or directory
INFO An error occurred while communicating with the Kubernetes cluster.
If this error persists, you may need to run the uninstall command before attempting to run
the install command again.
Cole Snodgrass (Airbyte)
07/18/2024, 8:59 PMabctl local uninstall
and then abctl local install
Cole Snodgrass (Airbyte)
07/18/2024, 8:59 PMJake Duckworth
07/18/2024, 9:09 PMINFO Using Kubernetes provider:
Provider: kind
Kubeconfig: /Users/jakeduckworth/.airbyte/abctl/abctl.kubeconfig
Context: kind-airbyte-abctl
SUCCESS Found Docker installation: version 27.0.3
SUCCESS Existing cluster 'airbyte-abctl' found
WARNING Failed to initialize 'local' command
Uninstallation attempt will continue
SUCCESS Uninstallation of cluster 'airbyte-abctl' completed successfully
SUCCESS Airbyte uninstallation complete
• When I run the install, I'm back to this error:
INFO Using Kubernetes provider:
Provider: kind
Kubeconfig: /Users/jakeduckworth/.airbyte/abctl/abctl.kubeconfig
Context: kind-airbyte-abctl
SUCCESS Found Docker installation: version 27.0.3
SUCCESS Port 9000 appears to be available
INFO No existing cluster found, cluster 'airbyte-abctl' will be created
ERROR Cluster 'airbyte-abctl' could not be created
ERROR unable to create kind cluster: command "docker run --name airbyte-abctl-control-plane --hostname airbyte-abctl-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --label io.x-k8s.kind.cluster=airbyte-abctl --net kind --restart=on-failure:1 --init=false --cgroupns=private --volume=/Users/jakeduckworth/.airbyte/abctl/data:/var/local-path-provisioner --publish=0.0.0.0:9000:80/TCP --publish=127.0.0.1:61643:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8" failed with error: exit status 126
Jake Duckworth
07/18/2024, 9:14 PM% sudo ls ~/.airbyte/abctl/
data
Running chmod 777 ~/.airbyte/abctl/data
then trying again results in the same message as above..Cole Snodgrass (Airbyte)
07/18/2024, 9:15 PMdocker run --name airbyte-abctl-control-plane --hostname airbyte-abctl-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --label io.x-k8s.kind.cluster=airbyte-abctl --net kind --restart=on-failure:1 --init=false --cgroupns=private --volume=/Users/jakeduckworth/.airbyte/abctl/data:/var/local-path-provisioner --publish=0.0.0.0:9000:80/TCP --publish=127.0.0.1:61643:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8
Jake Duckworth
07/18/2024, 9:15 PMdocker: Error response from daemon: error while creating mount source path '/host_mnt/Users/jakeduckworth/.airbyte/abctl/data': mkdir /host_mnt/Users/jakeduckworth/.airbyte/abctl: permission denied.
Cole Snodgrass (Airbyte)
07/18/2024, 9:16 PMv0.8.1
which will include a fix for the first issue you ran into. I’m still looking into what is causing the /host_mnt
problem.Jake Duckworth
07/18/2024, 9:17 PMmkdir
and chmod
commands manually then re-run the docker run
, I get this message:
docker: Error response from daemon: Conflict. The container name "/airbyte-abctl-control-plane" is already in use by container "6fb6e602d3709d5bea6f254b3d6fd26231dcff39da9aaba8c2cc6b4c7d5b315d". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
Jake Duckworth
07/18/2024, 9:20 PMCole Snodgrass (Airbyte)
07/18/2024, 9:22 PMJake Duckworth
07/18/2024, 9:24 PMJake Duckworth
07/18/2024, 9:24 PMJake Duckworth
07/18/2024, 9:24 PMCole Snodgrass (Airbyte)
07/18/2024, 9:27 PMJake Duckworth
07/18/2024, 9:59 PMJake Duckworth
07/18/2024, 10:00 PMJake Duckworth
07/19/2024, 2:27 PMabctl
this AM and that initial error is gone as expected 😄
Now it makes it to the error while creating mount source path
error we were looking into yesterday. I added the folder manually and grant it the appropriate permissions. When I then try to run the docker command again, I get this error _(it is likely a red herring though)_:
docker: Error response from daemon: Conflict. The container name "/airbyte-abctl-control-plane" is already in use by container "a6d1abb5b58c8fd93d2bc52c02850db1e810ca2b618e268c28be402961547049". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
Jake Duckworth
07/19/2024, 4:13 PMdocker-compose
setup does run successfully.Aaron Johnson
07/19/2024, 6:14 PMabctl local install
i get this error. I also previously ran it with docker-compose and the run-ab-platform script in the past.
unable to create kind cluster: failed to init node with kubeadm: command "docker exec --privileged airbyte-abctl-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
AJ Davis
07/23/2024, 5:26 PMhost_mnt
permission error when I run the docker run
cmd.Cole Snodgrass (Airbyte)
07/23/2024, 6:09 PM-v
is passed to the install command). However v0.9.0
will probably not fix this permission issue.
I am actively working on finding and fixing this issue, any insights into any weird docker configurations/limitations that your machine may have would be appreciated.AJ Davis
07/23/2024, 7:31 PMmacOS: Montery v12.7.4
Docker Desktop: 4.30.0
Docker: 26.1.1
abctl: v0.8.2
I don’t think I’ve done anything to my docker configs. They should be out of the box. But if you have any suggestions/questions of what to look for, let me know.AJ Davis
07/23/2024, 8:16 PMabctl v0.9.0
, ran abctl local install -v
Output ended on this step
DEBUG kind - INFO: Deleted nodes: ["airbyte-abctl-control-plane"]
with the same error message.
ERROR unable to create kind cluster: command "docker run...
Cole Snodgrass (Airbyte)
07/23/2024, 8:17 PMAJ Davis
07/23/2024, 8:19 PM-v
flag on the install.Jake Duckworth
07/24/2024, 2:30 PM-v
flag and got a new message pointing out that the port I was trying to use was already taken even though the initial status said it was free. I changed the install to use a different port and it installed successfully, May be worth a try:
abctl local install --port=9002
Aaron Johnson
07/24/2024, 4:53 PMabctl local uninstall && abctl local install -v
still fails for me regardless of port. Still seems to be failing on the creation of the kind cluster
DEBUG kind - INFO: ✗ Starting control-plane 🕹️
DEBUG kind - INFO: Deleted nodes: ["airbyte-abctl-control-plane"]
ERROR Cluster 'airbyte-abctl' could not be created
ERROR unable to create kind cluster: failed to init node with kubeadm: command "docker exec --privileged airbyte-abctl-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
...
... *tried running docker command explicitly and got this, but not sure if that is relevant
❯ docker exec --privileged airbyte-abctl-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6
Error response from daemon: No such container: airbyte-abctl-control-plane
AJ Davis
07/24/2024, 5:52 PMkindest/node
image and airbyte-abctl
container from docker, removed the .airbyte directory, and ran abctl local install
. I am now up and running.