Team, Im looking for someone who can give me some light to my investigation which i did with ElasticSearch. if any one of you can confirm or extend what are requirements need to be passed to use own instance of Elastic ? This is what i have already observed. • abilities to create index_template - it can be created manually (elasticsearch-setup docker) • abilities to create index_policy (ilm policy) - it can be any of policies i guess just assigne to index_template propperly (elasticsearch-setup docker) • abilities to create indexes (datahub-gms) • ElasticSearch with version at least 7.8 (from that version ILM Policy is supported) • ElasticSearch is using Java High Level REST Client for creating and updating indexes - based on my onw investigation of a classes which are imported to the code. (This one will be deprecated in version 7.15.0 of ELK with ability to use with future version with compatibility mode enabled) • How much amount of data i can expect on a daily basis - i know it is really hard to estimate, but as you already deployed you have more insights of the estimation. Do you know if there are any other requirements which need to be fulfilled to host own ELK or use ELK in the different environments, especially what type of permission users is requiring to work with Datahub ? read/write permission to indexes, what else ? Especially in context of using Elasticsearch as well as Graph Service Implementation ? https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html Thank you in advance Jaroslaw Gronowski

Team, please advise. *kube:*Error: parse error in "datahub/templates/datahub-encryption-secrets***redacted*** template: datahub/templates/datahub-encryption-secrets***redacted*** function "lookup" not defined

Hi everyone, I have questions that I would like to ask regarding GraphiQL. Firstly, what is the syntax of the container to put in field and value? Secondly, how can I get glossary term dataset from only one container? I am very new to this, so thank you very much.

Hi Team I am using v0.8.23 and i was trying to enable TLS. So i used following ingress:

apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  annotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: azure/application-gateway
 name: datahub
spec:
  rules:
  - host: <http://datahub.d.foo-bar.net|datahub.d.foo-bar.net>
    http:
      paths:
      - backend:
          service:
            name: datahub-frontend
            port:
              name: http
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - <http://datahub.d.foo-bar.net|datahub.d.foo-bar.net>
    secretName: agic-tls

but the ui seems to be redirecting to https://datahub.d.foo-bar.net/authenticate?redirect_uri=%2F and in logs i get :

! @7ojk5i2mk - Internal server error, for (GET) [/authenticate?redirect_uri=%2F] ->
play.api.UnexpectedException: Unexpected exception[TechnicalException: java.net.ConnectException: Connection refused (Connection refused)]
        at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:247)
        at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:176)
        at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:363)
        at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:361)
        at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:346)
        at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:345)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:92)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:92)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:92)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.pac4j.core.exception.TechnicalException: java.net.ConnectException: Connection refused (Connection refused)
        at org.pac4j.oidc.config.OidcConfiguration.internalInit(OidcConfiguration.java:136)
        at org.pac4j.core.util.InitializableObject.init(InitializableObject.java:20)
        at auth.sso.oidc.custom.CustomOidcClient.clientInit(CustomOidcClient.java:22)
        at org.pac4j.core.client.IndirectClient.internalInit(IndirectClient.java:58)
        at org.pac4j.core.util.InitializableObject.init(InitializableObject.java:20)
        at org.pac4j.core.client.IndirectClient.getRedirectAction(IndirectClient.java:93)
        at org.pac4j.core.client.IndirectClient.redirect(IndirectClient.java:79)
        at controllers.AuthenticationController.redirectToIdentityProvider(AuthenticationController.java:151)
        at controllers.AuthenticationController.authenticate(AuthenticationController.java:85)
        at router.Routes$$anonfun$routes$1$$anonfun$applyOrElse$4$$anonfun$apply$4.apply(Routes.scala:374)
        at router.Routes$$anonfun$routes$1$$anonfun$applyOrElse$4$$anonfun$apply$4.apply(Routes.scala:374)
        at play.core.routing.HandlerInvokerFactory$$anon$3.resultCall(HandlerInvoker.scala:134)
        at play.core.routing.HandlerInvokerFactory$$anon$3.resultCall(HandlerInvoker.scala:133)
        at play.core.routing.HandlerInvokerFactory$JavaActionInvokerFactory$$anon$8$$anon$2$$anon$1.invocation(HandlerInvoker.scala:108)
        at play.core.j.JavaAction$$anon$1.call(JavaAction.scala:88)
        at play.http.DefaultActionCreator$1.call(DefaultActionCreator.java:31)
        at play.core.j.JavaAction$$anonfun$9.apply(JavaAction.scala:138)
        at play.core.j.JavaAction$$anonfun$9.apply(JavaAction.scala:138)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
        at play.core.j.HttpExecutionContext$$anon$2.run(HttpExecutionContext.scala:56)
        at play.api.libs.streams.Execution$trampoline$.execute(Execution.scala:70)
        at play.core.j.HttpExecutionContext.execute(HttpExecutionContext.scala:48)
        at scala.concurrent.impl.Future$.apply(Future.scala:31)
        at scala.concurrent.Future$.apply(Future.scala:494)
        at play.core.j.JavaAction.apply(JavaAction.scala:138)
        at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:96)
        at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:89)
        at play.api.libs.streams.StrictAccumulator$$anonfun$mapFuture$2$$anonfun$1.apply(Accumulator.scala:174)
        at play.api.libs.streams.StrictAccumulator$$anonfun$mapFuture$2$$anonfun$1.apply(Accumulator.scala:174)
        at scala.util.Try$.apply(Try.scala:192)
        at play.api.libs.streams.StrictAccumulator$$anonfun$mapFuture$2.apply(Accumulator.scala:174)
        at play.api.libs.streams.StrictAccumulator$$anonfun$mapFuture$2.apply(Accumulator.scala:170)
        at scala.Function1$$anonfun$andThen$1.apply(Function1.scala:52)
        at play.api.libs.streams.StrictAccumulator.run(Accumulator.scala:207)
        at play.core.server.AkkaHttpServer$$anonfun$14.apply(AkkaHttpServer.scala:357)
        at play.core.server.AkkaHttpServer$$anonfun$14.apply(AkkaHttpServer.scala:355)
        at akka.http.scaladsl.util.FastFuture$.akka$http$scaladsl$util$FastFuture$$strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$$anonfun$transformWith$extension1$1.apply(FastFuture.scala:51)
        at akka.http.scaladsl.util.FastFuture$$anonfun$transformWith$extension1$1.apply(FastFuture.scala:50)
        ... 13 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:288)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:203)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:189)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
        at com.nimbusds.jose.util.DefaultResourceRetriever.getInputStream(DefaultResourceRetriever.java:249)
        at com.nimbusds.jose.util.DefaultResourceRetriever.retrieveResource(DefaultResourceRetriever.java:201)
        at org.pac4j.oidc.config.OidcConfiguration.internalInit(OidcConfiguration.java:133)
        ... 52 common frames omitted

by looking at it i dont fully understand the issue. Can you please point me to right direction? What do i miss? thx

Why does datahub not automatically log you out if you go to http://localhost:9002/logOut? It works if I log out using the logOut nav item. Is there another way I can logout/terminate the session from another web app programmatically?

hi team, my deployment keeps failing at

datahub-datahub-upgrade-job

( I ran

helm upgrade --install --namespace datahub -f values.yaml datahub datahub/datahub --debug

). I appreciate any help here

I am getting build errors when I am trying to build

datahub-frontend

image using the tag Here is the error 🧵 Any help on this would be great. Thanks!

spark submit cluster mode errors 🧵

Hello team,

we are setting up datahub on eks cluster and created ingestion with snowflake. however post completion of ingestion i am unable see linage

This message was deleted.

Using curl to create a group, i can create a group, but the users in the group cannot be created. sos

Unable to build latest spark lineage jar from master branch gradle build job, anyone tried to do this and successfully did it? https://github.com/datahub-project/datahub/tree/master/metadata-integration/java/spark-lineage

Hello, I am having trouble getting the UI ingestion working on datahub on docker-compose. Everything seems properly set up, but when I click on "RUN" it does not show me the output logs only the execution count increases with no status or outputs (image below). any suggestions? I used the the same recipe on datahub CLI 0.8.43.1 locally and the metadata was properly ingested. So it seems the problem is either in the UI or GMS. I'm using Datahub 0.8.43 I'll post the gms logs on the thread Edit: I'm using Google OIDC with

METADATA_SERVICE_AUTH_ENABLED=true

Hello. I am encountering an interesting issue where I am not able to view the contents in my local instance of datahub (localhost:9002) after I ingested new glossary terms. The icon for glossary appears on the homepage but the screen goes blank after I click on any glossary term. Has anybody encountered this issue or know how to resolve so that I can see the glossary contents?

Hi all. I use airflow plugin and I can't mass delete data with platdorm "airflow" using CLI. Command

datahub delete --env TEST --entity_type=datajob --platform=airflow --hard

return "0 entities with 0 rows" but I can delete specific urn or other platform such "kafka" and "postgres". How can I make mass delete for airflow?

Hi all - I am running Datahub with quickstart image in local containers on Docker. I am able to ingest some Postegres tables from my database and I am adding some description, tags and terms to make a presentation. I tried to make a backup (to not lose all my work) with CLI command:

datahub docker quickstart --backup

But what I get is this:

Error: No such option: --backu

I'm running in Windows environment.

Anyone know how I can fix this error I get during

./gradlew build

on an M1 mac? 🧵

Hi Team, Facing this issue when connecting to Redshift Tables , ( followed the prerequisites for shared for redshift from here https://datahubproject.io/docs/generated/ingestion/sources/redshift/#prerequisites-1 to the user which we are using to connect to redshift ) -

datahub.configuration.common.PipelineExecutionError: ('Source reported errors', RedshiftReport(workunits_produced=0, workunit_ids=[], warnings={}, failures={'version': ["Error: invalid literal for int() with base 10: 'redshift:'"]}, cli_version='0.8.41', cli_entry_location='/root/.venvs/airflow/lib/python3.7/site-packages/datahub/__init__.py', py_version='3.7.10 (default, Jun  3 2021, 00:02:01) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-13)]', py_exec_path='/root/.venvs/airflow/bin/python', os_details='Linux-4.14.287-215.504.amzn2.x86_64-x86_64-with-glibc2.2.5', tables_scanned=0, views_scanned=0, entities_profiled=0, filtered=[], soft_deleted_stale_entities=[], query_combiner=None, saas_version='', upstream_lineage={}))

Anyone having issues accessing the sample yaml-formatted file for Business Glossary terms? I am following the link here.. https://datahubproject.io/docs/generated/ingestion/sources/business-glossary/#quickstart-recipe

Hello, can anyone help me?https://datahubspace.slack.com/archives/CUMUWQU66/p1660839719285489

Hey Guys, I was using datahub v0.8.42 but was experiecing the issue reported here: https://github.com/datahub-project/datahub/issues/4854 It's supposed fixed in this version correct? But I am still having the "Sign Out" button redirect to the main page using Google OIDC. Anyone having the same problem?

Hello everyone, my team encountered an issue while working with the

redshift-usage

source. We've set up a pipeline with the following configuration:

...
table_pattern:
      deny:
        - 'analytics.*.*requests*'
        - 'analytics.public.requests_raw_stg'
...

we triggered ingestion and in the gms logs we observed entries like the following:

16:07:58.319 [qtp1830908236-16] INFO  c.l.m.r.entity.AspectResource:126 - INGEST PROPOSAL proposal: {aspectName=datasetUsageStatistics, systemMetadata={lastObserved=1660832494657, runId=redshift-usage-2022_08_18-14_10_28}, entityUrn=urn:li:dataset:(urn:li:dataPlatform:redshift,analytics.public.requests_current_year_old,PROD), entityType=dataset, aspect={contentType=application/json, value=ByteString(length=1336,bytes=7b227469...205b5d7d)}, changeType=UPSERT}
...
16:08:35.622 [qtp1830908236-1878] INFO  c.l.m.r.entity.AspectResource:126 - INGEST PROPOSAL proposal: {aspectName=datasetUsageStatistics, systemMetadata={lastObserved=1660832496738, runId=redshift-usage-2022_08_18-14_10_28}, entityUrn=urn:li:dataset:(urn:li:dataPlatform:redshift,analytics.analytics_sources.potential_signup_requests,PROD), entityType=dataset, aspect={contentType=application/json, value=ByteString(length=1526,bytes=7b227469...205b5d7d)}, changeType=UPSERT}
...

there are no records for

analytics.public.requests_raw_stg

. Could it be that only the last regex is being considered?

Hi team, please explain about data process in model picture below, thanks.

image.png

Hi everyone! I have some problem with launching quickstart. Usually, I have no problem (yesterday was ok). I have this error when pulling datahub-actions:

ERROR: for datahub-actions  failed to register layer: error creating overlay mount to /var/lib/docker/overlay2/8e4a978b78ea7e210f5feb6cc5d864ca03ed1cc652bde6a0a6e0772fb1ab71b2/merged: too many levels of symbolic links
ERROR: failed to register layer: error creating overlay mount to /var/lib/docker/overlay2/8e4a978b78ea7e210f5feb6cc5d864ca03ed1cc652bde6a0a6e0772fb1ab71b2/merged: too many levels of symbolic links
Error while pulling images. Going to attempt to move on to docker-compose up assuming the images have been built locally

I've tried to delete all my container/images/volumes. But I still have this error. Thank you for your help!

Hey if we are upgrading images to 0.8.43 then mae and mce are failing. Can someone pls explain me how mae and mce-consumer has impact on frontend?

And due to mae and mce i am not able to ingest anything on ui

👋 Hello, team! i use Datahub with docker on a Linux VM. On this VM is also use the dbt. but i can`t ingest this dbt Path. Do i have to bind a volume in the docker compose yml? I tried this, under 'datahub-gms' i added:

volumes:

- /home/adm_avs/dwh_prod:/home/dbt

and in the Datahub UI under Ingest --> Source, i added this code:

source:

type: dbt

config:

manifest_path: /home/dbt/target/manifest_file.json'

test_results_path: /home/dbt/target/run_results.json'

load_schemas: /home/dbt/target/sources_file.json'

target_platform: my_target_platform_id

catalog_path: /home/dbt/target/catalog_file.json'

When i run

docker colume ls

i don't see the volume. Does anybody hagve an idea?

Team, after upgrading to

0.8.43

I am facing issues with Okta Ingestion. Looks like an additional parameter

report-to

has been added in this version

--report-to TEXT                Provide an output file to produce a\n'
           'This version of datahub supports report-to functionality\n'
           'datahub ingest run -c /tmp/datahub/ingest/6ff6e569-5133-4711-accb-2a321ade586a/recipe.yml --report-to

Any thoughts on the resolution?