Ben Pankow
10/03/2022, 9:17 PM0.40.13
Helm chart into my EKS cluster:
helm upgrade --install airbyte airbyte/airbyte --debug --namespace airbyte --version 0.40.13
It looks like the minio
pod is waiting on some PVCs, but the PVC is waiting on the minio
pod:
❯ k describe persistentvolumeclaim -n airbyte airbyte-minio-pv-claim-airbyte-minio-0 ✘ 130|
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForPodScheduled 4m51s (x41 over 14m) persistentvolume-controller waiting for pod airbyte-minio-0 to be scheduled
❯ k describe pod -n airbyte airbyte-minio-0
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 14s (x24 over 15m) default-scheduler 0/2 nodes are available: 2 persistentvolumeclaim "airbyte-minio-pv-claim" not found.
Kyryl Skobylko
10/03/2022, 9:21 PMBen Pankow
10/03/2022, 9:22 PMKyryl Skobylko
10/03/2022, 9:24 PM0.40.14
helm repo update
and try to upgrade againBen Pankow
10/03/2022, 9:27 PM❯ helm upgrade --install airbyte airbyte/airbyte --debug --namespace airbyte --version 0.40.14
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/ben/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/ben/.kube/config
history.go:56: [debug] getting history for release airbyte
Release "airbyte" does not exist. Installing it now.
install.go:178: [debug] Original chart version: "0.40.14"
install.go:199: [debug] CHART PATH: /Users/ben/Library/Caches/helm/repository/airbyte-0.40.14.tgz
Error: template: airbyte/templates/airbyte-db.yaml:41:38: executing "airbyte/templates/airbyte-db.yaml" at <.Values.global.image.tag>: nil pointer evaluating interface {}.tag
helm.go:88: [debug] template: airbyte/templates/airbyte-db.yaml:41:38: executing "airbyte/templates/airbyte-db.yaml" at <.Values.global.image.tag>: nil pointer evaluating interface {}.tag
Kyryl Skobylko
10/03/2022, 9:33 PMBen Pankow
10/03/2022, 9:56 PMhelp repo update
& also deleting /Users/ben/Library/Caches/helm/repository/
where the chart is coming from
install.go:199: [debug] CHART PATH: /Users/ben/Library/Caches/helm/repository/airbyte-0.40.14.tgz
Kyryl Skobylko
10/03/2022, 10:00 PM0.40.15
Ben Pankow
10/03/2022, 10:15 PMOct 03, 2022 10:13:13 PM io.grpc.internal.ManagedChannelImpl$NameResolverListener handleErrorInSyncContext
WARNING: [Channel<1>: (airbyte-temporal-svc:7233)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host airbyte-temporal-svc, cause=java.lang.RuntimeException: java.net.UnknownHostException: airbyte-temporal-svc: Name or service not known
at io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:223)
at io.grpc.internal.DnsNameResolver.doResolve(DnsNameResolver.java:282)
at io.grpc.grpclb.GrpclbNameResolver.doResolve(GrpclbNameResolver.java:63)
at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:318)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.net.UnknownHostException: airbyte-temporal-svc: Name or service not known
at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:933)
at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1529)
at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1519)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1377)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1305)
at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:631)
at io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:219)
... 6 more
}
However the temporal svc seems to exist w/ endpoiint:
❯ k get pods -n airbyte
NAME READY STATUS RESTARTS
airbyte-temporal-5f47b89b68-sc7c6 1/1 Running 0 5m17s
❯ k describe svc -n airbyte airbyte-temporal
Name: airbyte-temporal
Namespace: airbyte
Labels: <http://app.kubernetes.io/instance=airbyte|app.kubernetes.io/instance=airbyte>
<http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
<http://app.kubernetes.io/name=temporal|app.kubernetes.io/name=temporal>
<http://app.kubernetes.io/version=0.40.10|app.kubernetes.io/version=0.40.10>
<http://helm.sh/chart=temporal-0.40.13|helm.sh/chart=temporal-0.40.13>
Annotations: <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: airbyte
<http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: airbyte
Selector: <http://app.kubernetes.io/instance=airbyte,app.kubernetes.io/name=temporal|app.kubernetes.io/instance=airbyte,app.kubernetes.io/name=temporal>
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 172.20.242.200
IPs: 172.20.242.200
Port: <unset> 7233/TCP
TargetPort: 7233/TCP
Endpoints: 10.0.1.131:7233
Session Affinity: None
Events: <none>
Kyryl Skobylko
10/03/2022, 10:32 PMBen Pankow
10/03/2022, 10:41 PM2022-10-03 22:44:12 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):221 - Waiting for temporal server...
2022-10-03 22:44:12 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):232 - Waiting for namespace default to be initialized in temporal...
Oct 03, 2022 10:44:15 PM io.grpc.internal.ManagedChannelImpl$NameResolverListener handleErrorInSyncContext
WARNING: [Channel<1>: (airbyte-temporal-svc:7233)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host airbyte-temporal-svc, cause=java.lang.RuntimeException: java.net.UnknownHostException: airbyte-temporal-svc: Name or service not known
at io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:223)
at io.grpc.internal.DnsNameResolver.doResolve(DnsNameResolver.java:282)
at io.grpc.grpclb.GrpclbNameResolver.doResolve(GrpclbNameResolver.java:63)
at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:318)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.net.UnknownHostException: airbyte-temporal-svc: Name or service not known
at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:933)
at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1529)
at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1519)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1377)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1305)
at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:631)
at io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:219)
... 6 more
}
Kyryl Skobylko
10/03/2022, 10:52 PMkubectl get svc -n airbyte
Ben Pankow
10/03/2022, 10:54 PM❯ kubectl get svc -n airbyte
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
airbyte-airbyte-server-svc ClusterIP 172.20.243.12 <none> 8001/TCP 18m
airbyte-airbyte-webapp-svc ClusterIP 172.20.19.52 <none> 80/TCP 18m
airbyte-db-svc ClusterIP 172.20.126.29 <none> 5432/TCP 19m
airbyte-minio-svc ClusterIP 172.20.73.67 <none> 9000/TCP 19m
airbyte-temporal ClusterIP 172.20.158.218 <none> 7233/TCP 18m
dig
inside the cluster
bash-4.2# dig airbyte-temporal.airbyte.svc.cluster.local
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.5.2 <<>> airbyte-temporal.airbyte.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1820
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;airbyte-temporal.airbyte.svc.cluster.local. IN A
;; ANSWER SECTION:
airbyte-temporal.airbyte.svc.cluster.local. 5 IN A 172.20.158.218
;; Query time: 0 msec
;; SERVER: 172.20.0.10#53(172.20.0.10)
;; WHEN: Mon Oct 03 22:55:08 UTC 2022
;; MSG SIZE rcvd: 129
-svc
suffix?Kyryl Skobylko
10/03/2022, 10:57 PM0.40.6
Ben Pankow
10/03/2022, 11:13 PMminio
now, and am hitting an error when setting up a Source in the Airbyte UI. I see the minio SVC but it has no endpoints (& no pods)
❯ k describe svc -n airbyte airbyte-minio-svc
Name: airbyte-minio-svc
Namespace: airbyte
Labels: <http://app.kubernetes.io/instance=airbyte|app.kubernetes.io/instance=airbyte>
<http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
<http://app.kubernetes.io/name=airbyte-minio|app.kubernetes.io/name=airbyte-minio>
<http://app.kubernetes.io/version=0.40.10|app.kubernetes.io/version=0.40.10>
<http://helm.sh/chart=airbyte-0.40.16|helm.sh/chart=airbyte-0.40.16>
Annotations: <http://helm.sh/hook|helm.sh/hook>: pre-install,pre-upgrade
<http://helm.sh/hook-weight|helm.sh/hook-weight>: -1
Selector: <http://app.kubernetes.io/instance=airbyte,app.kubernetes.io/name=airbyte-minio|app.kubernetes.io/instance=airbyte,app.kubernetes.io/name=airbyte-minio>
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 172.20.14.162
IPs: 172.20.14.162
Port: <unset> 9000/TCP
TargetPort: 9000/TCP
Endpoints: <none>
Session Affinity: None
Events: <none>
❯ k get pods -n airbyte
NAME READY STATUS RESTARTS AGE
airbyte-airbyte-bootloader 0/1 Completed 0 3m31s
airbyte-db-0 1/1 Running 1 3m35s
airbyte-pod-sweeper-pod-sweeper-6685c669b-ljpr2 1/1 Running 0 3m7s
airbyte-server-55f548d999-szsjp 1/1 Running 0 3m7s
airbyte-temporal-7d75d4b96d-kpxxr 1/1 Running 0 3m7s
airbyte-webapp-5b99866bf4-k5zc7 1/1 Running 0 3m7s
airbyte-worker-c75f5474c-bcr69 1/1 Running 0 3m7s
ingress-nginx-controller-7d98fb5bd-rhh6n 1/1 Running 0 3m29s
oauth2-proxy-8c656b4dd-t49nf 1/1 Running 0 3m44s
urce-pokeapi-check-46ad1f50-c4a9-4b0a-a39c-c48c7d82a268-0-wmnbu 0/4 Completed 0 53s
Dusty Shapiro
10/03/2022, 11:38 PMsvc
but no podsJean-Pierre Poutcheu
10/04/2022, 3:03 PMminio
pod running ?Kyryl Skobylko
10/05/2022, 8:06 AMkubectl get statefulset
Or kubectl describe statefulset airbyte-minio
Jean-Pierre Poutcheu
10/05/2022, 12:15 PMkubectl describe statefulset airbyte-minio
Name: airbyte-minio
Namespace: bi
CreationTimestamp: Wed, 05 Oct 2022 14:09:49 +0200
Selector: <http://app.kubernetes.io/instance=airbyte,app.kubernetes.io/name=airbyte-minio|app.kubernetes.io/instance=airbyte,app.kubernetes.io/name=airbyte-minio>
Labels: <http://app.kubernetes.io/instance=airbyte|app.kubernetes.io/instance=airbyte>
<http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
<http://app.kubernetes.io/name=airbyte-minio|app.kubernetes.io/name=airbyte-minio>
<http://app.kubernetes.io/version=0.40.10|app.kubernetes.io/version=0.40.10>
<http://helm.sh/chart=airbyte-0.40.16|helm.sh/chart=airbyte-0.40.16>
Annotations: <http://helm.sh/hook|helm.sh/hook>: pre-install,pre-upgrade
<http://helm.sh/hook-weight|helm.sh/hook-weight>: -1
Replicas: 1 desired | 0 total
Update Strategy: RollingUpdate
Partition: 0
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: <http://app.kubernetes.io/instance=airbyte|app.kubernetes.io/instance=airbyte>
<http://app.kubernetes.io/name=airbyte-minio|app.kubernetes.io/name=airbyte-minio>
Containers:
airbyte-minio:
Image: minio/minio:latest
Port: 9000/TCP
Host Port: 0/TCP
Args:
server
/storage
Environment:
MINIO_ROOT_USER: minio
MINIO_ROOT_PASSWORD: minio123
Mounts:
/data from storage (rw)
Volumes: <none>
Volume Claims:
Name: airbyte-minio-pv-claim
StorageClass:
Labels: <none>
Annotations: <none>
Capacity: 500Mi
Access Modes: [ReadWriteOnce]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 91s (x16 over 4m15s) statefulset-controller create Pod airbyte-minio-0 in StatefulSet airbyte-minio failed error: Pod "airbyte-minio-0" is invalid: spec.containers[0].volumeMounts[0].name: Not found: "storage"
Kyryl Skobylko
10/05/2022, 12:21 PMJean-Pierre Poutcheu
10/05/2022, 12:23 PM0.40.16
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
airbyte-minio-pv-claim-airbyte-minio-0 Pending hcloud-volumes 23h
airbyte-volume-db-airbyte-db-0 Bound pvc-0606070f-8a6c-45b3-8cbb-d0e0b444851c 10Gi RWO hcloud-volumes 10m
kubectl describe pvc airbyte-minio-pv-claim-airbyte-minio-0
Name: airbyte-minio-pv-claim-airbyte-minio-0
Namespace: bi
StorageClass: hcloud-volumes
Status: Pending
Volume:
Labels: <http://app.kubernetes.io/instance=airbyte|app.kubernetes.io/instance=airbyte>
<http://app.kubernetes.io/name=airbyte-minio|app.kubernetes.io/name=airbyte-minio>
Annotations: <none>
Finalizers: [<http://kubernetes.io/pvc-protection|kubernetes.io/pvc-protection>]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 107s (x5564 over 23h) persistentvolume-controller waiting for first consumer to be created before binding
Kyryl Skobylko
10/05/2022, 12:28 PM0.40.17
in a few mins(published new ver)Jean-Pierre Poutcheu
10/05/2022, 12:40 PMkubectl get pods
NAME READY STATUS RESTARTS AGE
airbyte-airbyte-bootloader 0/1 Completed 0 53s
airbyte-db-0 1/1 Running 0 57s
airbyte-minio-0 1/1 Running 0 56s
airbyte-pod-sweeper-pod-sweeper-69f567689d-m5599 1/1 Running 0 30m
airbyte-server-856fb9744f-zwkfl 1/1 Running 1 (2m11s ago) 30m
airbyte-temporal-747c8d4c76-tp97m 1/1 Running 0 30m
airbyte-webapp-6b498f4c64-sdkn9 1/1 Running 0 30m
airbyte-worker-79d8dff667-t8rb7 1/1 Running 0 30m
airbyte-minio
pod is now present, was did you changed please ? I see no commits in https://github.com/airbytehq/airbyte/tree/master/charts ThanksDusty Shapiro
10/05/2022, 12:47 PMKyryl Skobylko
10/05/2022, 1:01 PMDusty Shapiro
10/05/2022, 1:17 PMairbyte-airbyte-webapp-svc
the name of the webapp svc in the newer charts?Ben Pankow
10/05/2022, 4:25 PM0.40.17
and get a connector running successfullyclaudio viera
10/14/2022, 1:26 PMKyryl Skobylko
10/14/2022, 1:36 PMkskobylko@Kyryls-MacBook-Air xpuska-charts % helm search repo airbyte
NAME CHART VERSION APP VERSION DESCRIPTION
airbyte/airbyte 0.40.18 0.40.14 Helm chart to deploy airbyte
airbyte/airbyte-bootloader 0.40.18 0.40.14 Helm chart to deploy airbyte-bootloader
airbyte/metrics 0.40.18 0.40.3 Helm chart to deploy airbyte-metrics
airbyte/pod-sweeper 0.40.18 0.39.37-alpha Helm chart to deploy airbyte-pod-sweeper
airbyte/server 0.40.18 0.40.14 Helm chart to deploy airbyte-server
airbyte/temporal 0.40.18 0.40.14 Helm chart to deploy airbyte-temporal
airbyte/webapp 0.40.18 0.40.14 Helm chart to deploy airbyte-webapp
airbyte/worker 0.40.18 0.40.14 Helm chart to deploy airbyte-worker
0.40.18
as for nowclaudio viera
10/14/2022, 1:43 PM