https://linen.dev logo
Join SlackCommunities
Powered by
Hi all, I’m having some trouble deploying the new ...
# ask-community-for-troubleshooting
b
Hi all, I’m having some trouble deploying the new 
0.40.13
Helm chart into my EKS cluster:
Copy code
helm upgrade --install airbyte airbyte/airbyte --debug --namespace airbyte --version 0.40.13
It looks like the 
minio
pod is waiting on some PVCs, but the PVC is waiting on the 
minio
pod:
Copy code
❯ k describe persistentvolumeclaim -n airbyte airbyte-minio-pv-claim-airbyte-minio-0                                                                                                                                                                 ✘ 130|
...
Events:
  Type    Reason                Age                   From                         Message
  ----    ------                ----                  ----                         -------
  Normal  WaitForPodScheduled   4m51s (x41 over 14m)  persistentvolume-controller  waiting for pod airbyte-minio-0 to be scheduled
Copy code
❯ k describe pod -n airbyte airbyte-minio-0
...
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  14s (x24 over 15m)  default-scheduler  0/2 nodes are available: 2 persistentvolumeclaim "airbyte-minio-pv-claim" not found.
k
Hi, @Ben Pankow , let me release a new version in a few mins
This one was already addressed
b
great, thank you!
k
New version published as 
0.40.14
helm repo update
and try to upgrade again
b
Hmm, running into a different error this time:
Copy code
❯ helm upgrade --install airbyte airbyte/airbyte --debug --namespace airbyte --version 0.40.14
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/ben/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/ben/.kube/config
history.go:56: [debug] getting history for release airbyte
Release "airbyte" does not exist. Installing it now.
install.go:178: [debug] Original chart version: "0.40.14"
install.go:199: [debug] CHART PATH: /Users/ben/Library/Caches/helm/repository/airbyte-0.40.14.tgz

Error: template: airbyte/templates/airbyte-db.yaml:41:38: executing "airbyte/templates/airbyte-db.yaml" at <.Values.global.image.tag>: nil pointer evaluating interface {}.tag
helm.go:88: [debug] template: airbyte/templates/airbyte-db.yaml:41:38: executing "airbyte/templates/airbyte-db.yaml" at <.Values.global.image.tag>: nil pointer evaluating interface {}.tag
👀 1
k
See the error, fixed it
try to update the repo and try again should work now
@Ben Pankow are you able to deploy the chart?
b
I’m still running into the same error - I’m not sure if I am clearing my cache properly
I’m running 
help repo update
& also deleting 
/Users/ben/Library/Caches/helm/repository/
where the chart is coming from
Copy code
install.go:199: [debug] CHART PATH: /Users/ben/Library/Caches/helm/repository/airbyte-0.40.14.tgz
k
released as 
0.40.15
b
Great, able to deploy now but hitting some new errors related to services finding one another. The webapp is failing since it can’t reach the server - the server endpoint isn’t populating since the server is erroring with:
Copy code
Oct 03, 2022 10:13:13 PM io.grpc.internal.ManagedChannelImpl$NameResolverListener handleErrorInSyncContext
WARNING: [Channel<1>: (airbyte-temporal-svc:7233)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host airbyte-temporal-svc, cause=java.lang.RuntimeException: java.net.UnknownHostException: airbyte-temporal-svc: Name or service not known
	at io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:223)
	at io.grpc.internal.DnsNameResolver.doResolve(DnsNameResolver.java:282)
	at io.grpc.grpclb.GrpclbNameResolver.doResolve(GrpclbNameResolver.java:63)
	at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:318)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.net.UnknownHostException: airbyte-temporal-svc: Name or service not known
	at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:933)
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1529)
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1519)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1377)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1305)
	at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:631)
	at io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:219)
	... 6 more
}
However the temporal svc seems to exist w/ endpoiint:
Copy code
❯ k get pods -n airbyte
NAME                                              READY   STATUS             RESTARTS   
airbyte-temporal-5f47b89b68-sc7c6                 1/1     Running            0          5m17s
Copy code
❯ k describe svc -n airbyte airbyte-temporal
Name:              airbyte-temporal
Namespace:         airbyte
Labels:            <http://app.kubernetes.io/instance=airbyte|app.kubernetes.io/instance=airbyte>
                   <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
                   <http://app.kubernetes.io/name=temporal|app.kubernetes.io/name=temporal>
                   <http://app.kubernetes.io/version=0.40.10|app.kubernetes.io/version=0.40.10>
                   <http://helm.sh/chart=temporal-0.40.13|helm.sh/chart=temporal-0.40.13>
Annotations:       <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: airbyte
                   <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: airbyte
Selector:          <http://app.kubernetes.io/instance=airbyte,app.kubernetes.io/name=temporal|app.kubernetes.io/instance=airbyte,app.kubernetes.io/name=temporal>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                172.20.242.200
IPs:               172.20.242.200
Port:              <unset>  7233/TCP
TargetPort:        7233/TCP
Endpoints:         10.0.1.131:7233
Session Affinity:  None
Events:            <none>
I’ve tried restarting the server pod a few times and it still can’t seem to connect to temporal
Tried redeploying in the root/default namespace as well & see the same issue in the server pod
k
Hmm, strange
It should have created the service, but it takes sometime a more time than expected for temporal to do it’s stuff
b
I just redeployed - yeah, it looks like temporal may still be setting up. I will wait a bit to see if the issue resolves itself
It’s been quite a while, and not seeing new logs from Temporal. Here’s the point in which the server starts erroring:
Copy code
2022-10-03 22:44:12 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):221 - Waiting for temporal server...
2022-10-03 22:44:12 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):232 - Waiting for namespace default to be initialized in temporal...
Oct 03, 2022 10:44:15 PM io.grpc.internal.ManagedChannelImpl$NameResolverListener handleErrorInSyncContext
WARNING: [Channel<1>: (airbyte-temporal-svc:7233)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host airbyte-temporal-svc, cause=java.lang.RuntimeException: java.net.UnknownHostException: airbyte-temporal-svc: Name or service not known
	at io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:223)
	at io.grpc.internal.DnsNameResolver.doResolve(DnsNameResolver.java:282)
	at io.grpc.grpclb.GrpclbNameResolver.doResolve(GrpclbNameResolver.java:63)
	at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:318)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.net.UnknownHostException: airbyte-temporal-svc: Name or service not known
	at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:933)
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1529)
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1519)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1377)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1305)
	at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:631)
	at io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:219)
	... 6 more
}
k
kubectl get svc -n airbyte
b
Copy code
❯ kubectl get svc -n airbyte
NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
airbyte-airbyte-server-svc   ClusterIP   172.20.243.12    <none>        8001/TCP   18m
airbyte-airbyte-webapp-svc   ClusterIP   172.20.19.52     <none>        80/TCP     18m
airbyte-db-svc               ClusterIP   172.20.126.29    <none>        5432/TCP   19m
airbyte-minio-svc            ClusterIP   172.20.73.67     <none>        9000/TCP   19m
airbyte-temporal             ClusterIP   172.20.158.218   <none>        7233/TCP   18m
I can see the service with 
dig
inside the cluster
Copy code
bash-4.2# dig airbyte-temporal.airbyte.svc.cluster.local

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.5.2 <<>> airbyte-temporal.airbyte.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1820
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;airbyte-temporal.airbyte.svc.cluster.local. IN A

;; ANSWER SECTION:
airbyte-temporal.airbyte.svc.cluster.local. 5 IN A 172.20.158.218

;; Query time: 0 msec
;; SERVER: 172.20.0.10#53(172.20.0.10)
;; WHEN: Mon Oct 03 22:55:08 UTC 2022
;; MSG SIZE  rcvd: 129
could it have to do with the lack of 
-svc
suffix?
k
+
Strange that yours lacks the suffix
let me pull the chart from remote and see what’s inside
my bad, let me revert the changes
check 
0.40.6
b
Thanks, giving it a try
Great, looks like everything deploys. One more issue - I don’t see a pod for 
minio
now, and am hitting an error when setting up a Source in the Airbyte UI. I see the minio SVC but it has no endpoints (& no pods)
Copy code
❯ k describe svc -n airbyte airbyte-minio-svc
Name:              airbyte-minio-svc
Namespace:         airbyte
Labels:            <http://app.kubernetes.io/instance=airbyte|app.kubernetes.io/instance=airbyte>
                   <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
                   <http://app.kubernetes.io/name=airbyte-minio|app.kubernetes.io/name=airbyte-minio>
                   <http://app.kubernetes.io/version=0.40.10|app.kubernetes.io/version=0.40.10>
                   <http://helm.sh/chart=airbyte-0.40.16|helm.sh/chart=airbyte-0.40.16>
Annotations:       <http://helm.sh/hook|helm.sh/hook>: pre-install,pre-upgrade
                   <http://helm.sh/hook-weight|helm.sh/hook-weight>: -1
Selector:          <http://app.kubernetes.io/instance=airbyte,app.kubernetes.io/name=airbyte-minio|app.kubernetes.io/instance=airbyte,app.kubernetes.io/name=airbyte-minio>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                172.20.14.162
IPs:               172.20.14.162
Port:              <unset>  9000/TCP
TargetPort:        9000/TCP
Endpoints:         <none>
Session Affinity:  None
Events:            <none>
Copy code
❯ k get pods -n airbyte
NAME                                                              READY   STATUS      RESTARTS   AGE
airbyte-airbyte-bootloader                                        0/1     Completed   0          3m31s
airbyte-db-0                                                      1/1     Running     1          3m35s
airbyte-pod-sweeper-pod-sweeper-6685c669b-ljpr2                   1/1     Running     0          3m7s
airbyte-server-55f548d999-szsjp                                   1/1     Running     0          3m7s
airbyte-temporal-7d75d4b96d-kpxxr                                 1/1     Running     0          3m7s
airbyte-webapp-5b99866bf4-k5zc7                                   1/1     Running     0          3m7s
airbyte-worker-c75f5474c-bcr69                                    1/1     Running     0          3m7s
ingress-nginx-controller-7d98fb5bd-rhh6n                          1/1     Running     0          3m29s
oauth2-proxy-8c656b4dd-t49nf                                      1/1     Running     0          3m44s
urce-pokeapi-check-46ad1f50-c4a9-4b0a-a39c-c48c7d82a268-0-wmnbu   0/4     Completed   0          53s
d
^ This is the same issue I run into for the past few Helm charts
I also see the Minio 
svc
but no pods
j
Experiencing the same issue, any idea on the root cause ?
@Ben Pankow @Dusty Shapiro were you able to have 
minio
pod running ?
k
Can you try to run 
kubectl get statefulset
Or 
kubectl describe statefulset airbyte-minio
And send output here
j
Copy code
kubectl describe statefulset airbyte-minio
Copy code
Name:               airbyte-minio
Namespace:          bi
CreationTimestamp:  Wed, 05 Oct 2022 14:09:49 +0200
Selector:           <http://app.kubernetes.io/instance=airbyte,app.kubernetes.io/name=airbyte-minio|app.kubernetes.io/instance=airbyte,app.kubernetes.io/name=airbyte-minio>
Labels:             <http://app.kubernetes.io/instance=airbyte|app.kubernetes.io/instance=airbyte>
                    <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
                    <http://app.kubernetes.io/name=airbyte-minio|app.kubernetes.io/name=airbyte-minio>
                    <http://app.kubernetes.io/version=0.40.10|app.kubernetes.io/version=0.40.10>
                    <http://helm.sh/chart=airbyte-0.40.16|helm.sh/chart=airbyte-0.40.16>
Annotations:        <http://helm.sh/hook|helm.sh/hook>: pre-install,pre-upgrade
                    <http://helm.sh/hook-weight|helm.sh/hook-weight>: -1
Replicas:           1 desired | 0 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  <http://app.kubernetes.io/instance=airbyte|app.kubernetes.io/instance=airbyte>
           <http://app.kubernetes.io/name=airbyte-minio|app.kubernetes.io/name=airbyte-minio>
  Containers:
   airbyte-minio:
    Image:      minio/minio:latest
    Port:       9000/TCP
    Host Port:  0/TCP
    Args:
      server
      /storage
    Environment:
      MINIO_ROOT_USER:      minio
      MINIO_ROOT_PASSWORD:  minio123
    Mounts:
      /data from storage (rw)
  Volumes:  <none>
Volume Claims:
  Name:          airbyte-minio-pv-claim
  StorageClass:  
  Labels:        <none>
  Annotations:   <none>
  Capacity:      500Mi
  Access Modes:  [ReadWriteOnce]
Events:
  Type     Reason        Age                   From                    Message
  ----     ------        ----                  ----                    -------
  Warning  FailedCreate  91s (x16 over 4m15s)  statefulset-controller  create Pod airbyte-minio-0 in StatefulSet airbyte-minio failed error: Pod "airbyte-minio-0" is invalid: spec.containers[0].volumeMounts[0].name: Not found: "storage"
k
Which version was deployed?
j
0.40.16
Copy code
kubectl get pvc                           
NAME                                     STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
airbyte-minio-pv-claim-airbyte-minio-0   Pending                                                                        hcloud-volumes   23h
airbyte-volume-db-airbyte-db-0           Bound     pvc-0606070f-8a6c-45b3-8cbb-d0e0b444851c   10Gi       RWO            hcloud-volumes   10m

kubectl describe pvc airbyte-minio-pv-claim-airbyte-minio-0
Name:          airbyte-minio-pv-claim-airbyte-minio-0
Namespace:     bi
StorageClass:  hcloud-volumes
Status:        Pending
Volume:        
Labels:        <http://app.kubernetes.io/instance=airbyte|app.kubernetes.io/instance=airbyte>
               <http://app.kubernetes.io/name=airbyte-minio|app.kubernetes.io/name=airbyte-minio>
Annotations:   <none>
Finalizers:    [<http://kubernetes.io/pvc-protection|kubernetes.io/pvc-protection>]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age                    From                         Message
  ----    ------                ----                   ----                         -------
  Normal  WaitForFirstConsumer  107s (x5564 over 23h)  persistentvolume-controller  waiting for first consumer to be created before binding
k
Try 
0.40.17
in a few mins(published new ver)
You can upgrade btw without any issues as for now(before that there was a few issues with managing deployments and pvc from helm side)
j
Copy code
kubectl get pods
NAME                                               READY   STATUS             RESTARTS           AGE
airbyte-airbyte-bootloader                         0/1     Completed          0                  53s
airbyte-db-0                                       1/1     Running            0                  57s
airbyte-minio-0                                    1/1     Running            0                  56s
airbyte-pod-sweeper-pod-sweeper-69f567689d-m5599   1/1     Running            0                  30m
airbyte-server-856fb9744f-zwkfl                    1/1     Running            1 (2m11s ago)      30m
airbyte-temporal-747c8d4c76-tp97m                  1/1     Running            0                  30m
airbyte-webapp-6b498f4c64-sdkn9                    1/1     Running            0                  30m
airbyte-worker-79d8dff667-t8rb7                    1/1     Running            0                  30m
airbyte-minio
pod is now present, was did you changed please ? I see no commits in https://github.com/airbytehq/airbyte/tree/master/charts Thanks
d
It’s gotten further than it did before. Let me setup a full connection, replicate, and see how it goes 🤞
Looks like it’s back up and running! Thanks
k
@Jean-Pierre Poutcheu forgot to specify correct pvc name in statefulset
I’ll commit and push soon
🙏 1
👍 1
d
Is 
airbyte-airbyte-webapp-svc
the name of the webapp svc in the newer charts?
➕ 1
b
I was also able to set up 
0.40.17
and get a connector running successfully
c
good morning guys
i am install version 0.40.17
and have same error
k
can you try to use lates available in the helm repo?
Copy code
kskobylko@Kyryls-MacBook-Air xpuska-charts % helm search repo airbyte
NAME                            CHART VERSION   APP VERSION     DESCRIPTION                             
airbyte/airbyte                 0.40.18         0.40.14         Helm chart to deploy airbyte            
airbyte/airbyte-bootloader      0.40.18         0.40.14         Helm chart to deploy airbyte-bootloader 
airbyte/metrics                 0.40.18         0.40.3          Helm chart to deploy airbyte-metrics    
airbyte/pod-sweeper             0.40.18         0.39.37-alpha   Helm chart to deploy airbyte-pod-sweeper
airbyte/server                  0.40.18         0.40.14         Helm chart to deploy airbyte-server     
airbyte/temporal                0.40.18         0.40.14         Helm chart to deploy airbyte-temporal   
airbyte/webapp                  0.40.18         0.40.14         Helm chart to deploy airbyte-webapp     
airbyte/worker                  0.40.18         0.40.14         Helm chart to deploy airbyte-worker
it’s 
0.40.18
as for now
c
ok i will try
is good now
thanks
156 Views
Open in Slack
PreviousNext
Powered by