hihi penguin Hello, DataHub Community! We have been getting questions on building DataHub images and deploying them in an air-gapped environment, and we would love to gather knowledge from the Community teamwork If you have either built images in a completely air-gapped environment or you deployed inside one, please share your experiences/best practices! • Building in an air-gapped env: ◦ What are the steps you took to make sure all DataHub dependencies are downloaded from a custom artifactory? ◦ Were there any issues you ran into that other people would likely also face? • Deploying in an air-gapped env: ◦ Which deployment methods did you use to deploy in the air-gapped env? (Docker-compose, Kubernetes, etc.) ◦ What steps did you take to make sure the containers are available in the environment (if you did not build it yourself) ◦ Any gotchas that you faced while deploying?

We have infra that mirrors dockerhub and maven, so not too bad. For things that we can't mirror, we build a base image outside and publish it to docker hub, for instance the dockerfiles using the jwilderize binary.

We have similar setup at @better-orange-49102 as it's air-gapped and hence cannot reach to the internet to fetch images from dockergub. what we do is this: • we pull images from docker hub locally build and push to AWS ECR and then used

Pretty much same setup here also. I have 2 environments where I deploy datahub and one of those has internet access. I use helm charts to deploy, so I first deploy in the internet accessible environment without modifying the image references in the helm value files. Once deployed, I introspect the various pods and extract the referenced images. I have a script that pulls those images locally, which I then push to my air-gapped environment. I then modify the helm values (image refs) for the air-gapped deployment.

@better-orange-49102 In order to leverage your mirror infrastructure, did you need to update the gradle files to point to it (i.e. adding it to repositories.gradle) and were there any other places it needed to be specified? I'm trying to do something similar and have the gradle distribution and dependencies correctly using our mirror, but getting errors from java related to

org.apache.commons.cli.ParseException

not being located. Not sure if this is something others have faced.

@handsome-football-66174 Have you been able to get through the recent troubles while building? Feel free to share the experience here! We may be able to find someone who also ran into the issue

@bland-balloon-48379 there were a couple of places if I recall that required updating. Repos.gradle was one, but for some reason I had to insert a couple into build.gradle (the root folder one) and another one somewhere(which I can't remember). I could check, but I can only access the files on Monday ( good Friday today)

We've been making progress on this and should get a few changes out to the project, when back in the office, I will check the current state. A big helper was init.gradle, this is per user and overrides the build.gradle, it allows redirection without editing the build.gradle. There are a few places where the build files reference internet locations, for these we would like to change to project properties, hence can be overridden in the init.gradle I've made a change to some unit tests to use environment variables to override defaults, to point at internal locations (some improvements still to finish). Another to do, was bash or not bash, as a lot of build scripts assume sh is bash and some scripts explicitly use bash syntax, still unsure is standardising on bash is a good or bad thing. The UI is still WIP, a few changes there, especially ingestion as it references the editor from the internet, though this isn't a high priority at the moment.

@better-orange-49102 If you are able to check on the locations of those updates, that would be a big help. So far i've updated the distributionUrl in

gradle-wrapper.properties

, added the repo to

repositories.gradle

, and added the same repo to a pluginManagement object in

settings.gradle

since plugins didn't seem to be affected by the update to repositories.gradle. Haven't changed anything in build.gradle though so may look at that. The

init.gradle

file seems promising for making the code more adaptable!

sorry for the delay, I inserted repo locations in the following additional locations: 1. /entity-registry/custom-test-model/build.gradle 2. /metadata-models-custom/build.gradle I don't understand gradle much 🤷‍♂️

Using init.gradle does avoid the need to edit files. One example of a change which uses an environment variable is in datahub-web-react/build.gradle around line 21, where it checks for a project property, this property is set in the init.gradle file. Just rebuilt and it seems with the init.gradle alllows the two projects to build without editing.

Hi everyone, I'm currently trying to deploy DataHub in an air-gapped environment using the Helm repository locally, but so far I have not succeeded to complete the deployment. I was able to deploy the re-requisites and I'm not stuck on deploying DataHub itself. Are there any more detailed instructions on how to do this? Or perhaps a different (and hopefully easier) way of doing this in an air-gapped environment? I appreciate any pointers.

"so far I have not succeeded to complete the deployment. I was able to deploy the re-requisites and I'm not stuck on deploying DataHub itself." Consider posting some details in the troubleshooting channel..