https://datahubproject.io logo
Join Slack
Powered by
Hi team :sunny: I’m unable to connect my DB(`pgSQL...
# troubleshoot
p
Hi team ☀️ I’m unable to connect my DB(
pgSQL
) from DH deployment I’m able to connect from my local & from a connector(EC2) in the same VPC. I’ve also double-checked the PWD from k8s secret I’m using Following the error from the pod:
Copy code
Failed to pull image "acryldata/datahub-postgres-setup:v0.9.6.1": rpc error: code = Unknown desc = Error response from daemon: manifest for acryldata/datahub-postgres-setup:v0.9.6.1 not found: manifest unknown: manifest unknown
I’ve tried to set the version to 
v0.9.6.1rc4
as described here. No change Any idea?
✅ 1
b
The image specified is wrong - it is still looking for tag v0.9.6.1, not v0.9.6.1rc4. how did you specify it in helm
p
@better-orange-49102 in 
values.yaml
Copy code
datahub:
    version: "v0.9.6.1rc4"
b
if you run kubectl describe job 
datahub-postgres-setup
, what image is it attempting to pull?
p
@better-orange-49102
Copy code
Normal   Scheduled  18s                default-scheduler  Successfully assigned datahub-prod/datahub-postgresql-setup-job-vsz4f to ip-10-96-173-226.ec2.internal
  Normal   BackOff    15s (x2 over 16s)  kubelet            Back-off pulling image "acryldata/datahub-postgres-setup:v0.9.6.1"
  Warning  Failed     15s (x2 over 16s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    4s (x2 over 17s)   kubelet            Pulling image "acryldata/datahub-postgres-setup:v0.9.6.1"
  Warning  Failed     4s (x2 over 17s)   kubelet            Failed to pull image "acryldata/datahub-postgres-setup:v0.9.6.1": rpc error: code = Unknown desc = Error response from daemon: manifest for acryldata/datahub-postgres-setup:v0.9.6.1 not found: manifest unknown: manifest unknown
  Warning  Failed     4s (x2 over 17s)   kubelet            Error: ErrImagePull
I’ve downgraded to 
0.9.5
& it’s working
The error above is when:
Copy code
datahub:
  version: "v0.9.6.1"
Not 
v0.9.6.1rc4
@better-orange-49102 I’ve changed the chart to work with 
v0.9.6.1rc4
Same error for 
v0.9.6.1
Copy code
Normal   Scheduled  37s                default-scheduler  Successfully assigned datahub-prod/datahub-postgresql-setup-job-2mklv to ip-10-96-173-226.ec2.internal
  Normal   Pulling    20s (x2 over 36s)  kubelet            Pulling image "acryldata/datahub-postgres-setup:v0.9.6.1"
  Warning  Failed     20s (x2 over 36s)  kubelet            Failed to pull image "acryldata/datahub-postgres-setup:v0.9.6.1": rpc error: code = Unknown desc = Error response from daemon: manifest for acryldata/datahub-postgres-setup:v0.9.6.1 not found: manifest unknown: manifest unknown
  Warning  Failed     20s (x2 over 36s)  kubelet            Error: ErrImagePull
  Normal   BackOff    7s (x2 over 36s)   kubelet            Back-off pulling image "acryldata/datahub-postgres-setup:v0.9.6.1"
  Warning  Failed     7s (x2 over 36s)   kubelet            Error: ImagePullBackOff
It also crush the 
datahub-elasticsearch-setup-job-5mftp
pod now, but with 
v0.9.6.1rc4
(after 
helm upgrade
)
Copy code
Normal   Scheduled  4m11s                 default-scheduler  Successfully assigned datahub-prod/datahub-elasticsearch-setup-job-5mftp to ip-10-96-173-226.ec2.internal
  Warning  Failed     2m49s (x6 over 4m8s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    2m36s (x4 over 4m9s)  kubelet            Pulling image "linkedin/datahub-elasticsearch-setup:v0.9.6.1rc4"
  Warning  Failed     2m36s (x4 over 4m9s)  kubelet            Failed to pull image "linkedin/datahub-elasticsearch-setup:v0.9.6.1rc4": rpc error: code = Unknown desc = Error response from daemon: manifest for linkedin/datahub-elasticsearch-setup:v0.9.6.1rc4 not found: manifest unknown: manifest unknown
  Warning  Failed     2m36s (x4 over 4m9s)  kubelet            Error: ErrImagePull
  Normal   BackOff    2m25s (x7 over 4m8s)  kubelet            Back-off pulling image "linkedin/datahub-elasticsearch-setup:v0.9.6.1rc4"
@better-orange-49102 @bulky-electrician-72362 @astonishing-answer-96712 can you pls. assist with this?
a
Use global.datahub.version v0.9.6.1 but specify v0.9.6.2 for the postgresql setup job (image tag). That's what I did!
p
@adventurous-nightfall-90271 where exactly? I'm setting dh version only in 1 place at the chart
a
postgresqlSetupJob.image.tag: "v0.9.6.2"
There are container-level overrides for every image
b
I used container level overrides with rcXX appended in the tag myself and it works for me as well
p
@adventurous-nightfall-90271 
postgresqlSetupJob.image.tag: "v0.9.6.2"
Where should that take place in 
values.yaml
?
a
@powerful-cat-68806 at the base level i.e. no indentation.
p
All good it’s working now @adventurous-nightfall-90271 🙂 Now error with the GMS pod
Copy code
ERROR: Cannot connect to GMSat host datahub-datahub-gms port 8080. Make sure GMS is on the latest version and is running at that host before starting the migration.
BTW - I don’t see on DH release the version 
0.9.6.2
Can I use 
tag: "latest"
for GMS & upgrade pods?
i
You can but I would highly advised agaisn’t it as the 
latest
tag is a mutable tag and we make no guarantees that it won’t brake something on your instance if deployed incorrectly. We make best efforts for that to not be case but it is an effort.
Also, true 
latest
would also force you to have a pull policy of Always.
p
@incalculable-ocean-74010 didn’t understand your last comment What version do I need to use for: • datahub-gms • datahub-upgrade ?
i
Please set them to 0.9.6.1
What helm chart version and values.yaml are you using?
p
3.7.2
What do you ask about 
values.yaml
?
i
The latest helm chart version is 0.2.150. Where did you pull the helm chart from?
p
It’s running on my IDE
It’s breaking with 0.9.6.1
Copy code
repository: linkedin/datahub-gms
tag: "v0.9.6.1" # defaults to .global.datahub.version

repository: acryldata/datahub-upgrade
tag: "v0.9.6.1" # defaults to .global.datahub.version

repository: acryldata/datahub-actions
    tag: "v0.0.7"
Regarding pulling the helm chart - 
helm install/upgrade
suppose to do it, right? Unless my 
chart.yaml
is not updated
i
To get the latest helm chart you must run 
helm repo update
to get the latest remote versions pulled into your system
As our values.yaml suggest please make use of 
.global.datahub.version
, see this file for reference: https://github.com/acryldata/datahub-helm/blob/3bb596d57edb7c72a7ad6c4b08278e371522e6e5/charts/datahub/values.yaml#L317
I would highly suggest removing your custom version from the values.yaml and use helm chart v0.2.148. This is the last chart with datahub version v0.9.6.1, after that it is a major bump to v0.10
p
@incalculable-ocean-74010 DH chart didn’t updated & status:failed
i
Which pods failed?
p
I didn’t run upgrade yet…..just 
helm repo update
i
helm ls
is the current state of a helm deployment. Yours is 
failed
, to understand why please show the output of 
kubectl -n datahub-prod get po
And for all failed pods, please share the logs and the output of 
kubectl describe -n datahub-prod <pod name>
4 Views
Open in Slack
PreviousNext
Powered by