DataHub #getting-started

wooden-barista-16690

07/20/2022, 9:48 AM

Hello everyone, Since I have the MySQL DB app installed, I tried to deploy DataHub instance locally with the datahub docker quickstart --mysql-port POSITIVEINT (tried a few open port connections) command to override the default 3306 port but getting the attached errors. Do help me resolve this issue.

elegant-salesmen-99143

07/20/2022, 11:52 AM

Hey. Can anyone please help me clarify the difference betwen Create Domains vs Manage Domains Platforms privileges? According to descriptions one allows the actor to create new Domains, and the other allows actor to create and remove any Domains. So the difference is the ability to remove Domains? Or something else?

colossal-sandwich-50049

07/20/2022, 4:25 PM

Hello, very general question: is there parity between the various datahub programmatic APIs/clients (OpenAPI + CLI + GraphQL + emitters) and the datahub UI? I.e. can one expect that everything that can be done via the UI can also be done programmatically? Context: my company might want to build an internal UI over datahub, so we want to ensure that the ability to do everything programmatically is there cc: @great-toddler-2251

elegant-salesmen-99143

07/20/2022, 5:25 PM

Is there any way to trace changes made within DataHub, for example wich user added a Glossary Term, or edited Documentation, etc?

many-salesclerk-29098

07/21/2022, 3:23 AM

Hi, we are exploring datahub as a solution for our data-lineage usecase. Speaking about connectors for Postgres, I have a query regarding the lineage graph. Does it create a lineage graph for stored procedures and views also?

many-salesclerk-29098

07/21/2022, 3:26 AM

Also, is there a way, it would update the lineage graph if we make an insert to a table from another table's data. For example, if we run a query like this:

Copy code

INSERT INTO new_table (ID, Date, Name) 
SELECT id, date, name 
FROM old_table

many-waiter-16918

07/21/2022, 2:01 PM

Hey to you all, although I have configured the

bigquery-usage

plug-in following the guide, I still cannot see statistics. Is there someone who faces this issue also?

able-evening-90828

07/21/2022, 4:40 PM

Hi DataHub team, we want to try out DataHub and would like to conduct a preliminary security review of the platform first. Do you have any documentation on the best practices for hardening the deployment (both the datahub components and dependencies) from a security perspective? We use GKE and are interested in learning how we can restrict traffic to only certain static IP addresses or internal network. I also want to point out that the default

datahub/datahub

user doesn't seem to be a good practice. Chrome immediately flags it. It would be good if you can document in the deployment instructions how to configure a different username/password so that it encourages better security practices when deploying to the cloud.

👍 2

brief-church-81973

07/21/2022, 11:33 PM

Hello datahub enthousiasts, I'm looking for a way to "Add link" to the datajob object using java api. I couldn't find any example. Is that done by gms? Any help would be much appreciated. in other words, I want to do what the add link button does in the screenshot below, but using API.

breezy-shoe-41523

07/22/2022, 7:45 AM

Hello i’m deploying datahub with helm3 and i’m curious how can i set sso authentification to frontend server how can i set sso to my datahub? please help thank you

rapid-king-93225

07/22/2022, 9:23 AM

Where can I simply start a docker-compose file (quickstart)? https://raw.githubusercontent.com/datahub-project/datahub/master/docker/quickstart/docker-compose-without-neo4j-m1.quickstart.yml seems to contain variables which are not resolved. Where do I get a suitable .env file for these variables? I am aware of https://datahubproject.io/docs/quickstart/ the python CLI - but would prefer to not require that wrapper (rather get a normal docker-compose to work).

famous-florist-7218

07/22/2022, 9:25 AM

👋 Hi everyone! I’m a newcomer. I just go through this article

Running Airflow locally with DataHub

but I face to a discomfort problem. When I want to quick open an airflow task on datahub, it brings me to the wrong place. My airflow web URL is

localhost:58080

but datahub points to

localhost:8080

. Note: I’ve deployed with datahub quickstart and local airflow.

✅ 1

flaky-soccer-57765

07/25/2022, 10:22 AM

Hi All, newbie here. I am trying to install datahub on docker in a no internet machine. Docker data hub quickstart won't work because of network restrictions, I do have access to docker hub through portainer to pull images. How can I install the data hub stack offline in the machine please?

lemon-engine-23512

07/25/2022, 3:19 PM

Hi all, am trying follow Quickstart doc. setting up on my Mac local system. but never successful. I even updated the docker-compose with

platform: linux/amd64

for all services, still the mysql-setup and elastic search-setup keeps exiting and setup never completes. anyone face similar issues?

great-motherboard-71467

07/26/2022, 10:28 AM

Hi Team, Is there any other solution to build the datahub but not in docker, but as a part of a system ? Based on the Github project, most probably with a gradle building from sources ? Is there any other documentation which might be worth in that scenario ? Thanks

bright-diamond-60933

07/26/2022, 3:14 PM

I was able to build datahub easily on Mac

sparse-barista-40860

07/26/2022, 3:50 PM

hi dears, how can change pwd for datahub user?

jolly-balloon-85466

07/26/2022, 5:14 PM

👋 Hi everyone!

✅ 1

jolly-balloon-85466

07/26/2022, 5:14 PM

I'm trying to install datahub on kubenetes

✅ 1

jolly-balloon-85466

07/26/2022, 5:17 PM

Problem with dial: dial tcp <ip>3306 connect: connection timed out. Sleeping 1s

✅ 1

jolly-balloon-85466

07/26/2022, 5:17 PM

I'm getting above error

✅ 1

sparse-barista-40860

07/26/2022, 5:19 PM

I have a issue with updating locally datahub for latest version

sparse-barista-40860

07/26/2022, 5:19 PM

Copy code

Creating network "datahub_network" with the default driver

sparse-barista-40860

07/26/2022, 5:19 PM

and i cant ssh or scp

sparse-barista-40860

07/26/2022, 5:19 PM

how can solve that

sparse-barista-40860

07/26/2022, 5:20 PM

Copy code

datahub docker quickstart
No Datahub Neo4j volume found, starting with elasticsearch as graph service.
To use neo4j as a graph backend, run
`datahub docker quickstart --quickstart-compose-file ./docker/quickstart/docker-compose.quickstart.yml`
from the root of the datahub repo

Fetching docker-compose file <https://raw.githubusercontent.com/datahub-project/datahub/master/docker/quickstart/docker-compose-without-neo4j.quickstart.yml> from GitHub
Pulling elasticsearch          ... done
Pulling elasticsearch-setup    ... done
Pulling mysql                  ... done
Pulling datahub-gms            ... done
Pulling datahub-frontend-react ... done
Pulling datahub-actions        ... done
Pulling mysql-setup            ... done
Pulling zookeeper              ... done
Pulling broker                 ... done
Pulling schema-registry        ... done
Pulling kafka-setup            ... done

Creating network "datahub_network" with the default driver

sparse-barista-40860

07/26/2022, 5:38 PM

and this

sparse-barista-40860

07/26/2022, 5:38 PM

https://stackoverflow.com/questions/41736187/docker-compose-network-creation-kicks-me-out-of-ssh

sparse-barista-40860

07/26/2022, 5:38 PM

is not useful

great-motherboard-71467

07/27/2022, 2:07 PM

Dears, i`m analyzing the dockerfile of elasticsearch setup and there is a script which is responsible of indieces creation.

Copy code

<https://github.com/datahub-project/datahub/blob/master/docker/elasticsearch-setup/create-indices.sh>

There is a function which is checking if there is a defined INDEX_PREFIX of elasticsearch indieces. How to define this prefix ? I was trying to define this with dockerfile env as: INDEX_PREFIX=mytestprefix, but nothing is working or even was trying using ELASTIC_INDEX_PREFIX which is defined under application.yml for datahub-frontend-react which might be correlated with that settings. But still my indices are following

Copy code

yellow open datajobindex_v2                                          ldJo3uE4QJewBhNzr9XdXA 1 1 0 0   208b   208b
yellow open datahubsecretindex_v2                                    haJzqxCHSYu58IMnaSduSQ 1 1 0 0   208b   208b
yellow open dataset_datasetprofileaspect_v1                          Y2qj3-drTWqutplPcLvvhQ 1 1 0 0   208b   208b
yellow open datahubexecutionrequestindex_v2                          mrgBox2jRtKQeMPtHLaT5w 1 1 0 0   208b   208b
yellow open dataflowindex_v2                                         0_D7Q1iGRW-z2bFgHC0fZg 1 1 0 0   208b   208b
yellow open mlmodelgroupindex_v2                                     Vnah4hvCQhK61JdkwlqnPw 1 1 0 0   208b   208b
yellow open mlmodelindex_v2                                          PZKoGxrsTvmGk25VNNjHLQ 1 1 0 0   208b   208b
yellow open datahubpolicyindex_v2                                    brZRjdiNTryRDNRBUSsKyQ 1 1 5 0 10.3kb 10.3kb
yellow open assertionindex_v2                                        QozAdKuxS364gVmIek_pqw 1 1 0 0   208b   208b
yellow open corpuserindex_v2                                         XPPJPMjyR3OJIV4GUSy7sg 1 1 0 0   208b   208b
yellow open dataprocessindex_v2                                      CPSGwCMvSmiJYsnvUb8s3w 1 1 0 0   208b   208b
yellow open chartindex_v2                                            ra5VG-JfSMKwcN2-VpEQmQ 1 1 0 0   208b   208b
yellow open tagindex_v2                                              AXOqaMg8SJWXXnyhD2p_KQ 1 1 0 0   208b   208b
yellow open mlmodeldeploymentindex_v2                                sJt2p1YvSQWW1ewRKoKuEA 1 1 0 0   208b   208b
yellow open datajob_datahubingestioncheckpointaspect_v1              LTmBTaAJTbmd3tktC66wiQ 1 1 0 0   208b   208b
yellow open dataplatforminstanceindex_v2                             SfvFJfC-SZOnDYBL44x4Eg 1 1 0 0   208b   208b
yellow open assertion_assertionruneventaspect_v1                     js_KYeqPSEix2g5EsRHaMg 1 1 0 0   208b   208b
yellow open dashboardindex_v2                                        J4sC64YlRVmSmDyJ6G4T2A 1 1 0 0   208b   208b
yellow open telemetryindex_v2                                        EFpZSHf_S6Whj-8bVvOswQ 1 1 0 0   208b   208b
yellow open datasetindex_v2                                          FT8o_fmdQvCcqhTxJlYVAg 1 1 0 0   208b   208b
yellow open mlfeatureindex_v2                                        vCLwtkvvQ9ugeec6pVjR4Q 1 1 0 0   208b   208b
yellow open dashboard_dashboardusagestatisticsaspect_v1              sygR_ZRDSJG5nHZvAwC5_g 1 1 0 0   208b   208b
yellow open dataplatformindex_v2                                     QhyOxlslRdKzMLpP5evHTg 1 1 0 0   208b   208b
yellow open datajob_datahubingestionrunsummaryaspect_v1              4ky9JVTvT5WleFMBtbZRhw 1 1 0 0   208b   208b
yellow open dataprocessinstanceindex_v2                              enwXRh_zQaW8tiYUwNCB3Q 1 1 0 0   208b   208b
yellow open glossarynodeindex_v2                                     HmHkL_o-RMGQsALQxz6vgQ 1 1 0 0   208b   208b
yellow open datahubingestionsourceindex_v2                           5WShz3TgTtyM5E5kslJ_tA 1 1 0 0   208b   208b
yellow open invitetokenindex_v2                                      ZJ75LBP8QY-UpHDpZXTZ_g 1 1 0 0   208b   208b
yellow open datahubretentionindex_v2                                 nu642knUTYqQ1L6pM9lYIQ 1 1 0 0   208b   208b
yellow open graph_service_v1                                         fVLYqPbOTOmLSJ5-zDkmqg 1 1 0 0   208b   208b
yellow open dataprocessinstance_dataprocessinstanceruneventaspect_v1 loO3jJUDTB6Bw_wxU1DI6w 1 1 0 0   208b   208b
yellow open dataset_operationaspect_v1                               9nwFViX-SI6WWd8SKLjPuQ 1 1 0 0   208b   208b
yellow open system_metadata_service_v1                               21XSV1HfSxagi0Irbew1uA 1 1 0 0   208b   208b
yellow open datahubaccesstokenindex_v2                               JLquhnmlQgea7q0hUsjkpA 1 1 0 0   208b   208b
yellow open containerindex_v2                                        Zzeucw97Q8Cl9VnVMdcoXQ 1 1 0 0   208b   208b
yellow open schemafieldindex_v2                                      TLPPap0ORYS1ej-dOiVxjA 1 1 0 0   208b   208b
yellow open domainindex_v2                                           EXmwmFX5Qg6VMssb-oTiXQ 1 1 0 0   208b   208b
yellow open testindex_v2                                             gkkE8iP_T2GTHmYtS7Qgeg 1 1 0 0   208b   208b
yellow open mlfeaturetableindex_v2                                   -WW5qco8R560Z7rADnYwLA 1 1 0 0   208b   208b
yellow open notebookindex_v2                                         AHsVbrz7RhuQ51nVV9Qtfw 1 1 0 0   208b   208b
yellow open datahubupgradeindex_v2                                   6eXzTZLCR_KkBHfveZbJgw 1 1 0 0   208b   208b
yellow open glossarytermindex_v2                                     nGMaf6RQS5KXkhtQ-ghABQ 1 1 0 0   208b   208b
yellow open mlprimarykeyindex_v2                                     g4_WJniiR2SOPuVPRWFyEQ 1 1 0 0   208b   208b
yellow open corpgroupindex_v2                                        vfHg9BD0QyWiUl_2HIbq6A 1 1 0 0   208b   208b
yellow open dataset_datasetusagestatisticsaspect_v1                  jBuQor2tRRe2CGsWC2GCFg 1 1 0 0   208b   208b

As i would like to use external elasticsearch and would like to have possibility to much more control on the name of the indieces, just in case i would need to do the cleanup. It would be good to understood where to define this prefix. Of course if it is possible at all.