👋 Hello, team! i am new to this world i need to ingest datasets on to the datahub aws hosting please guide me how to achieve this please recommend me some best practices as well..

Hello Mridul, please have a look at DataHub’s ingestion docs: https://datahubproject.io/docs/metadata-ingestion

{datahub.cli.ingest_cli:86} - DataHub CLI version: 0.8.31.6 [2022-03-31 150321,961] WARNING {urllib3.connectionpool:744} - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7a039228e0>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/gms/config

Hello Mridul, How was datahub deployed?

When doing ingestion through DataHub’s UI you need to point to databases that are accessible to the datahub deployed pods. In this case k8s. In this case you are trying to get a k8s pod to connect to your local machine. Pointing to local host from the pod’s perspective is itself so that won’t work as there is no Postgres running in the frontend datahub pod.

deployed with k8s helm component

When you specify the ingestion in the UI, the ingestion logic is running in same kubernetes pod as DataHub’s frontend, i.e:

datahub-frontend-react

pod. Since it is running inside a k8s pod, this config:

config:
      host_port: 'localhost:5432'
      database: COVID
      username: postgres
      password: mynewpassword

Must be in specified in the context of the k8s pod.

localhost:5432

in the frontend container would mean that the frontend container can access it’s localhost:5432 port and connect to a postgres database. That is not the case right? You have no postgres database accessible in port 5432 of the frontend pod do you?

yes exactly thanks for the right guidance

What do you mean with

github trigger

?

since we are downloading covid dataset in our local machine and then import that data into the our postgres and then trying to ingest into the datahub.

Where is postgres hosted?

in local

You must deploy Postgres in k8s and make a service so that datahub can connect to it.

okay