https://datahubproject.io logo
Join Slack
Powered by
Hi team I have deployed data hub on kubernetes My...
# all-things-deployment
c
Hi team I have deployed data hub on kubernetes My gms and front end pods are running But Mae-consumer and Mce-consumer are not getting deployed
e
Hi. By default, we run gms and the two consumers in the same pod to reduce maintenance overhead. You can set this value to true https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L102 to have standalone consumer pods!
c
And also ingestion cron job is also not getting deployed
e
Have you explicitly enabled it? That one is disabled by default
c
Yes I enabled from values.ya file
But still not running in cron job
e
Do you see the cronjob when you run
Copy code
kubectl get cronjobs -n <<namespace>>
c
I can see only these two cron jobs
Not for ingestion
Can you please let me know on this
e
Can you share the full values.yaml that you are using? You can obfuscate any sensitive information
c
Values.yaml of ingestion job or main values.yaml that we are using for modification
e
You have two separate values.yaml file?
c
I am modifying all chart values from data hub values.yaml file With the images I downloaded from docker
e
Oh is the values.yaml of ingestion job referring to the recipe file? If so, then I am referring to the helm values.yaml file
c
This values file for ingestion
e
This chart creates one cronjob per cron. I see that you are setting crons field to empty. This is why you see none. What are you trying to achieve with ingestion cron?
c
I want my recipe.yaml file to run by this cron job as per schedule
I have set the attributes now and cron job is deployed it is not ingesting data at that time
e
Could you post the recipe file?
also your revised helm values.yaml file as well
c
e
I don’t think you need the command part
c
My recipe.yaml
Then how it will run ingest ?
If I don't pass command to ingest
e
The helm chart automatically generates the command that it should run
Seems like your config server address is not valid. Can you double check? assuming it is in the same namespace, you can just run http//&lt;&lt;datahub gms service name&gt;&gt;8080 if it’s in another namespace, it should be http://<<datahub-gms-service-name>>_.<<namespace>>.svc.cluster.local_
c
So it will automatically run that ingest job and I can see metadata in front end Right?
My service name is edp-datahub-datahub-gms
In data hub namespace
So server name will be edp-data hub-databub-gms.datahub.gms:8080
e
please refer to the msg above
the above is not a valid kubernetes dns name
c
Then what will be the valid one ?
According to my service
Even after removing the command step from recipe file it did not run ingest job
e
c
e
yup
c
I tried from front end to ingest but it is getting failed every time
e
where is this error coming from?
the actions pod?
c
From UI when I trying to execute ingestion
e
which version of datahub are you running right now?
image tag of datahub-gms and datahub-frontend-react that is running on kbuernetes
c
0.8.31
Gms and front end images are also having 0.8.31 version
e
do you restrict egress from your kubernetes cluster? i.e. can the actions pod talk to pypi to fetch packages?
c
How can I check that ?
Actually the datahub actions image I pulled from docker hub having 0.8.31 version But in datahub chart you are pulling from public ecr So that can be the reason ??
e
That should be fine. We have started posting to both docker repositories
c
One thing we are using datahub actions from acryl datahub but other images are from LinkedIn
Any reason for that
But this error is still there when I am running ingest from UI
Please help me on this
e
@big-carpet-38439 any idea on this error on ui ingestion?
c
@big-carpet-38439 can you help me in this
b
@creamy-van-28626 To use UI ingestion, we depend on the ability to talk to PIP over the public internet. If you cannot reach the public internet from your docker deployment, it simply will not work. It appears that this is the case, as we are unable to reach pip to resolve the "acryl-datahub" package!
c
But the acryl datahub package is installed
Then what I need to do
Or how can I make my acryl datahub actions image talk to public
??
b
Are you working in an airgapped environment?
c
Yup working in VDI
b
Got it - Do you know if your company offers a proxy to public pip repository?
If it does, we can set up a proxy to talk to it. If not, we can try to pre-install the acryl-datahub dependencies by mounting them into the container before you deploy (a bit harder)
c
I need to check that with my manager on this I recently joined
One thing @big-carpet-38439 what's the purpose of datahub actions image here Why we are using this ? As per my understanding datahub ingestion cron job is basically using ingestion image to perform ingestion What's the purpose of acryl datahub action image here ?
Any update ?
b
Actions image is mainly used to power the UI-based on-demand ingestion experience
This simply removes some of the hassle with manually ingesting via the DataHub CLI- However, you can of course continue to use that directly 🙂
c
So if we don't use datahub actions image it will going to effect anyway ?
b
Nope
Its fine to not use that image for now
Its purely a nice to have
c
It's basically for running ingestion in UI Right? But if we are ingesting recipe file from cron job it will not needed.
b
Yes exactly
c
Okay thanks
And one more thing why we are using datahub action from acryldata and other images are from LinkedIn
b
I think this question was answered previously. Basically its a historical fact. Prior to Acryl taking over, we pushed images to a linkedin-owned image repository. Now that Acryl is driving development of the project (team from LinkedIn), we are pushing to the new Acryl repository
c
So later on all other images will pulled from acryldata ?
b
New images yes, until we get a chance to do any migration of the core images
c
Okay thanks for this update
3 Views
Open in Slack
PreviousNext
Powered by