Hi all! We’ve been using DataHub for a while now and are really enjoying it! We’re now trying to deploy a forked version of the repo, and working on getting our CI/CD sorted. We’ve been able to modify our Helm chart to pull from ECS successfully, the only part we’re stuck on is how to build and deploy the docker images up into ECS. I’ve tried using this:

- create JAVA env
- create python env
- login to ecr...

- name: Gradle Build
  continue-on-error: true
  run: |
    ./gradlew clean
    ./gradlew build

- name: Deploy to ECS
  run: |
     ./gradlew quickstart --info
     {tag and upload images to ECS}

The build is successful (once I exclude tests), but it always fails at the

./gradlew quickstart

step with varying errors. We haven’t made any changes to the codebase yet. I’m curious if anyone else has solved this problem already? Otherwise I’ll update here with what I did to get it working.

@billions-tailor-78534 might be able to speak to this!

We managed to get this working via the following workflow:

frontend_build:
    name: Build and Push DataHub Frontend Docker Image
    runs-on: [ self-hosted, <our_runner> ] # this could be ubuntu for others
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Login to ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v1

      - uses: actions/setup-python@v4
        with:
          python-version: "3.10"
          cache: pip

      - name: Set up JDK 11
        uses: actions/setup-java@v3
        with:
          distribution: "zulu"
          java-version: 11
          cache: 'gradle'

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v2
        with:
          registry: <http://ghcr.io|ghcr.io>
          username: ${{ github.actor }}
          password: ${{ secrets.GIT_TOKEN }}

      - name: Gradle Build
        continue-on-error: true
        run: |
          ./gradlew :datahub-frontend:dist -x test -x yarnTest -x yarnLint --parallel
          mv ./datahub-frontend/build/distributions/datahub-frontend-*.zip datahub-frontend.zip

      - name: Deploy to ECR
        env:
          IMAGE_REGISTRY: ${{ steps.login-ecr.outputs.registry }}/cko-dp
        run: |
          docker build -f ./docker/datahub-frontend/Dockerfile -t $IMAGE_REGISTRY/$FRONTEND_IMAGE_NAME:$IMAGE_TAG . 
          docker images
          docker push $IMAGE_REGISTRY/$FRONTEND_IMAGE_NAME:$IMAGE_TAG

And for the gms build it was similar but looks like this:

- name: Gradle Build
        continue-on-error: true
        run: |
          ./gradlew :metadata-service:war:build -x test --parallel
          mv ./metadata-service/war/build/libs/war.war .

      - name: Deploy to ECR
        env:
          IMAGE_REGISTRY: ${{ steps.login-ecr.outputs.registry }}/cko-dp
        run: |
          docker build -f ./docker/datahub-gms/Dockerfile -t $IMAGE_REGISTRY/$GMS_IMAGE_NAME:$IMAGE_TAG .
          docker images
          docker push $IMAGE_REGISTRY/$GMS_IMAGE_NAME:$IMAGE_TAG