Hi all I m having some trouble deploying the new `0 40 13` H

Question

Hi all I m having some trouble deploying the new `0 40 13` Helm chart into my EKS cluster ```helm upgrade install airbyte airbyte airbyte debug namespace airbyte version 0 40 13``` It looks like the `minio` pod is waiting on some PVCs but the PVC is waiting on the `minio` pod ``` k describe persistentvolumeclaim n airbyte airbyte minio pv claim airbyte minio 0 130| Events Type Reason Age From Message Normal WaitForPodScheduled 4m51s x41 over 14m persistentvolume controller waiting for pod airbyte minio 0 to be scheduled``` ``` k describe pod n airbyte airbyte minio 0 Events Type Reason Age From Message Warning FailedScheduling 14s x24 over 15m default scheduler 0 2 nodes are available 2 persistentvolumeclaim airbyte minio pv claim not found ```

Kyryl Skobylko · Answer

Hi < Ben Pankow> let me release a new version in a few mins

Kyryl Skobylko · Answer

This one was already addressed

Ben Pankow · Answer

great thank you

Kyryl Skobylko · Answer

New version published as `0 40 14`

Kyryl Skobylko · Answer

`helm repo update` and try to upgrade again

Ben Pankow · Answer

Hmm running into a different error this time ``` helm upgrade install airbyte airbyte airbyte debug namespace airbyte version 0 40 14 WARNING Kubernetes configuration file is group readable This is insecure Location Users ben kube config WARNING Kubernetes configuration file is world readable This is insecure Location Users ben kube config history go 56 debug getting history for release airbyte Release airbyte does not exist Installing it now install go 178 debug Original chart version 0 40 14 install go 199 debug CHART PATH Users ben Library Caches helm repository airbyte 0 40 14 tgz Error template airbyte templates airbyte db yaml 41 38 executing airbyte templates airbyte db yaml at lt Values global image tag gt nil pointer evaluating interface tag helm go 88 debug template airbyte templates airbyte db yaml 41 38 executing airbyte templates airbyte db yaml at lt Values global image tag gt nil pointer evaluating interface tag```

Kyryl Skobylko · Answer

See the error fixed it

Kyryl Skobylko · Answer

try to update the repo and try again should work now

Kyryl Skobylko · Answer

< Ben Pankow> are you able to deploy the chart

Ben Pankow · Answer

I m still running into the same error I m not sure if I am clearing my cache properly

Ben Pankow · Answer

I m running `help repo update` amp also deleting ` Users ben Library Caches helm repository ` where the chart is coming from ```install go 199 debug CHART PATH Users ben Library Caches helm repository airbyte 0 40 14 tgz```

Kyryl Skobylko · Answer

released as `0 40 15`

Ben Pankow · Answer

Great able to deploy now but hitting some new errors related to services finding one another The webapp is failing since it can t reach the server the server endpoint isn t populating since the server is erroring with ```Oct 03 2022 10 13 13 PM io grpc internal ManagedChannelImpl$NameResolverListener handleErrorInSyncContext WARNING Channel lt 1 gt airbyte temporal svc 7233 Failed to resolve name status=Status code=UNAVAILABLE description=Unable to resolve host airbyte temporal svc cause=java lang RuntimeException java net UnknownHostException airbyte temporal svc Name or service not known at io grpc internal DnsNameResolver resolveAddresses DnsNameResolver java 223 at io grpc internal DnsNameResolver doResolve DnsNameResolver java 282 at io grpc grpclb GrpclbNameResolver doResolve GrpclbNameResolver java 63 at io grpc internal DnsNameResolver$Resolve run DnsNameResolver java 318 at java base java util concurrent ThreadPoolExecutor runWorker ThreadPoolExecutor java 1136 at java base java util concurrent ThreadPoolExecutor$Worker run ThreadPoolExecutor java 635 at java base java lang Thread run Thread java 833 Caused by java net UnknownHostException airbyte temporal svc Name or service not known at java base java net Inet4AddressImpl lookupAllHostAddr Native Method at java base java net InetAddress$PlatformNameService lookupAllHostAddr InetAddress java 933 at java base java net InetAddress getAddressesFromNameService InetAddress java 1529 at java base java net InetAddress$NameServiceAddresses get InetAddress java 852 at java base java net InetAddress getAllByName0 InetAddress java 1519 at java base java net InetAddress getAllByName InetAddress java 1377 at java base java net InetAddress getAllByName InetAddress java 1305 at io grpc internal DnsNameResolver$JdkAddressResolver resolveAddress DnsNameResolver java 631 at io grpc internal DnsNameResolver resolveAddresses DnsNameResolver java 219 6 more ``` However the temporal svc seems to exist w endpoiint ``` k get pods n airbyte NAME READY STATUS RESTARTS airbyte temporal 5f47b89b68 sc7c6 1 1 Running 0 5m17s``` ``` k describe svc n airbyte airbyte temporal Name airbyte temporal Namespace airbyte Labels Annotations airbyte airbyte Selector Type ClusterIP IP Family Policy SingleStack IP Families IPv4 IP 172 20 242 200 IPs 172 20 242 200 Port lt unset gt 7233 TCP TargetPort 7233 TCP Endpoints 10 0 1 131 7233 Session Affinity None Events lt none gt ```

Ben Pankow · Answer

I ve tried restarting the server pod a few times and it still can t seem to connect to temporal

Ben Pankow · Answer

Tried redeploying in the root default namespace as well amp see the same issue in the server pod

Kyryl Skobylko · Answer

Hmm strange

Kyryl Skobylko · Answer

It should have created the service but it takes sometime a more time than expected for temporal to do it s stuff

Ben Pankow · Answer

I just redeployed yeah it looks like temporal may still be setting up I will wait a bit to see if the issue resolves itself

Ben Pankow · Answer

It s been quite a while and not seeing new logs from Temporal Here s the point in which the server starts erroring ```2022 10 03 22 44 12 INFO i a c t TemporalUtils getTemporalClientWhenConnected 221 Waiting for temporal server 2022 10 03 22 44 12 WARN i a c t TemporalUtils getTemporalClientWhenConnected 232 Waiting for namespace default to be initialized in temporal Oct 03 2022 10 44 15 PM io grpc internal ManagedChannelImpl$NameResolverListener handleErrorInSyncContext WARNING Channel lt 1 gt airbyte temporal svc 7233 Failed to resolve name status=Status code=UNAVAILABLE description=Unable to resolve host airbyte temporal svc cause=java lang RuntimeException java net UnknownHostException airbyte temporal svc Name or service not known at io grpc internal DnsNameResolver resolveAddresses DnsNameResolver java 223 at io grpc internal DnsNameResolver doResolve DnsNameResolver java 282 at io grpc grpclb GrpclbNameResolver doResolve GrpclbNameResolver java 63 at io grpc internal DnsNameResolver$Resolve run DnsNameResolver java 318 at java base java util concurrent ThreadPoolExecutor runWorker ThreadPoolExecutor java 1136 at java base java util concurrent ThreadPoolExecutor$Worker run ThreadPoolExecutor java 635 at java base java lang Thread run Thread java 833 Caused by java net UnknownHostException airbyte temporal svc Name or service not known at java base java net Inet4AddressImpl lookupAllHostAddr Native Method at java base java net InetAddress$PlatformNameService lookupAllHostAddr InetAddress java 933 at java base java net InetAddress getAddressesFromNameService InetAddress java 1529 at java base java net InetAddress$NameServiceAddresses get InetAddress java 852 at java base java net InetAddress getAllByName0 InetAddress java 1519 at java base java net InetAddress getAllByName InetAddress java 1377 at java base java net InetAddress getAllByName InetAddress java 1305 at io grpc internal DnsNameResolver$JdkAddressResolver resolveAddress DnsNameResolver java 631 at io grpc internal DnsNameResolver resolveAddresses DnsNameResolver java 219 6 more ```

Kyryl Skobylko · Answer

`kubectl get svc n airbyte`

Ben Pankow · Answer

``` kubectl get svc n airbyte NAME TYPE CLUSTER IP EXTERNAL IP PORT S AGE airbyte airbyte server svc ClusterIP 172 20 243 12 lt none gt 8001 TCP 18m airbyte airbyte webapp svc ClusterIP 172 20 19 52 lt none gt 80 TCP 18m airbyte db svc ClusterIP 172 20 126 29 lt none gt 5432 TCP 19m airbyte minio svc ClusterIP 172 20 73 67 lt none gt 9000 TCP 19m airbyte temporal ClusterIP 172 20 158 218 lt none gt 7233 TCP 18m```

Ben Pankow · Answer

I can see the service with `dig` inside the cluster ```bash 4 2 dig airbyte temporal airbyte svc cluster local lt lt gt gt DiG 9 11 4 P2 RedHat 9 11 4 26 P2 amzn2 5 2 lt lt gt gt airbyte temporal airbyte svc cluster local global options +cmd Got answer WARNING local is reserved for Multicast DNS You are currently testing what happens when an mDNS query is leaked to DNS gt gt HEADER lt lt opcode QUERY status NOERROR id 1820 flags qr aa rd QUERY 1 ANSWER 1 AUTHORITY 0 ADDITIONAL 1 WARNING recursion requested but not available OPT PSEUDOSECTION EDNS version 0 flags udp 4096 QUESTION SECTION airbyte temporal airbyte svc cluster local IN A ANSWER SECTION airbyte temporal airbyte svc cluster local 5 IN A 172 20 158 218 Query time 0 msec SERVER 172 20 0 10 53 172 20 0 10 WHEN Mon Oct 03 22 55 08 UTC 2022 MSG SIZE rcvd 129```

Ben Pankow · Answer

could it have to do with the lack of ` svc` suffix

Kyryl Skobylko · Answer

+

Kyryl Skobylko · Answer

Strange that yours lacks the suffix

Kyryl Skobylko · Answer

let me pull the chart from remote and see what s inside

Kyryl Skobylko · Answer

my bad let me revert the changes

Kyryl Skobylko · Answer

check `0 40 6`

Ben Pankow · Answer

Thanks giving it a try

Ben Pankow · Answer

Great looks like everything deploys One more issue I don t see a pod for `minio` now and am hitting an error when setting up a Source in the Airbyte UI I see the minio SVC but it has no endpoints amp no pods ``` k describe svc n airbyte airbyte minio svc Name airbyte minio svc Namespace airbyte Labels Annotations pre install pre upgrade 1 Selector Type ClusterIP IP Family Policy SingleStack IP Families IPv4 IP 172 20 14 162 IPs 172 20 14 162 Port lt unset gt 9000 TCP TargetPort 9000 TCP Endpoints lt none gt Session Affinity None Events lt none gt ``` ``` k get pods n airbyte NAME READY STATUS RESTARTS AGE airbyte airbyte bootloader 0 1 Completed 0 3m31s airbyte db 0 1 1 Running 1 3m35s airbyte pod sweeper pod sweeper 6685c669b ljpr2 1 1 Running 0 3m7s airbyte server 55f548d999 szsjp 1 1 Running 0 3m7s airbyte temporal 7d75d4b96d kpxxr 1 1 Running 0 3m7s airbyte webapp 5b99866bf4 k5zc7 1 1 Running 0 3m7s airbyte worker c75f5474c bcr69 1 1 Running 0 3m7s ingress nginx controller 7d98fb5bd rhh6n 1 1 Running 0 3m29s oauth2 proxy 8c656b4dd t49nf 1 1 Running 0 3m44s urce pokeapi check 46ad1f50 c4a9 4b0a a39c c48c7d82a268 0 wmnbu 0 4 Completed 0 53s```

Dusty Shapiro · Answer

^ This is the same issue I run into for the past few Helm charts

Dusty Shapiro · Answer

I also see the Minio `svc` but no pods

Jean-Pierre Poutcheu · Answer

Experiencing the same issue any idea on the root cause

Jean-Pierre Poutcheu · Answer

< Ben Pankow> < Dusty Shapiro> were you able to have `minio` pod running

Kyryl Skobylko · Answer

Can you try to run `kubectl get statefulset` Or `kubectl describe statefulset airbyte minio`

Kyryl Skobylko · Answer

And send output here

Jean-Pierre Poutcheu · Answer

```kubectl describe statefulset airbyte minio``` ```Name airbyte minio Namespace bi CreationTimestamp Wed 05 Oct 2022 14 09 49 +0200 Selector Labels Annotations pre install pre upgrade 1 Replicas 1 desired | 0 total Update Strategy RollingUpdate Partition 0 Pods Status 0 Running 0 Waiting 0 Succeeded 0 Failed Pod Template Labels Containers airbyte minio Image minio minio latest Port 9000 TCP Host Port 0 TCP Args server storage Environment MINIO ROOT USER minio MINIO ROOT PASSWORD minio123 Mounts data from storage rw Volumes lt none gt Volume Claims Name airbyte minio pv claim StorageClass Labels lt none gt Annotations lt none gt Capacity 500Mi Access Modes ReadWriteOnce Events Type Reason Age From Message Warning FailedCreate 91s x16 over 4m15s statefulset controller create Pod airbyte minio 0 in StatefulSet airbyte minio failed error Pod airbyte minio 0 is invalid spec containers 0 volumeMounts 0 name Not found storage ```

Kyryl Skobylko · Answer

Which version was deployed

Jean-Pierre Poutcheu · Answer

`0 40 16`

Jean-Pierre Poutcheu · Answer

```kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE airbyte minio pv claim airbyte minio 0 Pending hcloud volumes 23h airbyte volume db airbyte db 0 Bound pvc 0606070f 8a6c 45b3 8cbb d0e0b444851c 10Gi RWO hcloud volumes 10m kubectl describe pvc airbyte minio pv claim airbyte minio 0 Name airbyte minio pv claim airbyte minio 0 Namespace bi StorageClass hcloud volumes Status Pending Volume Labels Annotations lt none gt Finalizers Capacity Access Modes VolumeMode Filesystem Used By lt none gt Events Type Reason Age From Message Normal WaitForFirstConsumer 107s x5564 over 23h persistentvolume controller waiting for first consumer to be created before binding```

Kyryl Skobylko · Answer

Try `0 40 17` in a few mins published new ver

Kyryl Skobylko · Answer

You can upgrade btw without any issues as for now before that there was a few issues with managing deployments and pvc from helm side

Jean-Pierre Poutcheu · Answer

```kubectl get pods NAME READY STATUS RESTARTS AGE airbyte airbyte bootloader 0 1 Completed 0 53s airbyte db 0 1 1 Running 0 57s airbyte minio 0 1 1 Running 0 56s airbyte pod sweeper pod sweeper 69f567689d m5599 1 1 Running 0 30m airbyte server 856fb9744f zwkfl 1 1 Running 1 2m11s ago 30m airbyte temporal 747c8d4c76 tp97m 1 1 Running 0 30m airbyte webapp 6b498f4c64 sdkn9 1 1 Running 0 30m airbyte worker 79d8dff667 t8rb7 1 1 Running 0 30m```

Jean-Pierre Poutcheu · Answer

`airbyte minio` pod is now present was did you changed please I see no commits in <https github com airbytehq airbyte tree master charts> Thanks

Dusty Shapiro · Answer

It s gotten further than it did before Let me setup a full connection replicate and see how it goes crossed fingers

Dusty Shapiro · Answer

Looks like it s back up and running Thanks

Kyryl Skobylko · Answer

< Jean Pierre Poutcheu> forgot to specify correct pvc name in statefulset

Kyryl Skobylko · Answer

I ll commit and push soon

Dusty Shapiro · Answer

Is `airbyte airbyte webapp svc` the name of the webapp svc in the newer charts

Ben Pankow · Answer

I was also able to set up `0 40 17` and get a connector running successfully

claudio viera · Answer

good morning guys

claudio viera · Answer

i am install version 0 40 17

claudio viera · Answer

and have same error

Kyryl Skobylko · Answer

can you try to use lates available in the helm repo

Kyryl Skobylko · Answer

```kskobylko Kyryls MacBook Air xpuska charts helm search repo airbyte NAME CHART VERSION APP VERSION DESCRIPTION airbyte airbyte 0 40 18 0 40 14 Helm chart to deploy airbyte airbyte airbyte bootloader 0 40 18 0 40 14 Helm chart to deploy airbyte bootloader airbyte metrics 0 40 18 0 40 3 Helm chart to deploy airbyte metrics airbyte pod sweeper 0 40 18 0 39 37 alpha Helm chart to deploy airbyte pod sweeper airbyte server 0 40 18 0 40 14 Helm chart to deploy airbyte server airbyte temporal 0 40 18 0 40 14 Helm chart to deploy airbyte temporal airbyte webapp 0 40 18 0 40 14 Helm chart to deploy airbyte webapp airbyte worker 0 40 18 0 40 14 Helm chart to deploy airbyte worker ```

Kyryl Skobylko · Answer

it s `0 40 18` as for now

claudio viera · Answer

ok i will try

claudio viera · Answer

is good now

claudio viera · Answer

thanks