https://datahubproject.io logo
Join Slack
Powered by
# getting-started
  • f
    Copy code
    source:
    type: postgres
    config:
        host_port: '10.187.3.209:5432'
        database: registry
        username: postgres
        password: postgres
        include_tables: true
        include_views: true
        profiling:
            enabled: false
sink:
    type: datahub-rest
    config:
        server: '<http://localhost:9002/api/gms>'
    b
    • 2
    • 1
  • a
    anyone got an example of 
    Postgresql
    instead of 
    mysql
    ?
    b
    i
    • 3
    • 11
  • w
    Hi all, I'm trying to get lineage from snowflake. I'm able to get a decent lineage, but when I focus on specific tables I'm missing upstream tables, however I do have the right columns in the schema. I believe it's due to the use of a with statement in the query. Is it a known bug?
    l
    s
    +3
    • 6
    • 20
  • f
    For Postgres ingestion, getting error like OperationalError: (psycopg2.OperationalError) SCRAM authentication requires libpq version 10 or above\n'
    b
    i
    s
    • 4
    • 3
  • b
    This message was deleted.
    i
    • 2
    • 1
  • i
    Hi，Is someone can help me?
    i
    • 2
    • 6
  • s
    hi, Is 
    Schema Version History
    enabled in latest datahub? I saw this from

    https://www.youtube.com/watch?v=IVazVgcNRdw&amp;t=2306s▾

    s
    e
    • 3
    • 4
  • m
    Hi, I want to load metadata (through datahub ingestion UI) from certain schemas and for tables with specific pattern in their names (Postgresql source). My yaml looks similar to this: schema_pattern: allow: - abc - pqr - test table_pattern: allow: - test_base_tbl - check_validations - user_info I assumed it to follow logical AND of these schemas and tables but I see that all objects from the given schemas (irrespective of the object name pattern) are being ingested. Also, is there a way to restrict the views that we ingest. (I am using include_views as True which is pulling all the views) Any sample on how to provide schema and table regex pattern that can be used in conjunction will help. Thanks
    d
    i
    • 3
    • 2
  • a
    Hey all, we are trying to export the metadata from datahub and therefore we need to query the dataset fields and one part would be the tags. When querying the tags on dataset field level it returns null, is there anything I miss? Example on the Online Demo Datahub Dataset: https://demo.datahubproject.io/dataset/urn:li:dataset:(urn:li:dataPlatform:postgres,jaff[…]p.dbt_jaffle.customers,PROD)/Schema?is_lineage_mode=false
    Copy code
    {dataset(urn:"urn:li:dataset:(urn:li:dataPlatform:postgres,jaffle_shop.dbt_jaffle.customers,PROD)")
  {schemaMetadata{
    fields{
      fieldPath,
      tags{tags{tag{urn}}},
      description
    }
  },tags{tags{tag{urn}}}
  }
}
    Result:
    Copy code
    {
  "data": {
    "dataset": {
      "schemaMetadata": {
        "fields": [
          {
            "fieldPath": "customer_id",
            "tags": null,
            "description": "This is a unique identifier for a customer"
          },
          {
            "fieldPath": "first_name",
            "tags": null,
            "description": "Customer's first name. PII."
          },
          {
            "fieldPath": "last_name",
            "tags": null,
            "description": "Customer's last name. PII."
          },
          {
            "fieldPath": "first_order",
            "tags": null,
            "description": "Date (UTC) of a customer's first order"
          },
          {
            "fieldPath": "most_recent_order",
            "tags": null,
            "description": "Date (UTC) of a customer's most recent order"
          },
          {
            "fieldPath": "number_of_orders",
            "tags": null,
            "description": "Count of the number of orders a customer has placed"
          },
          {
            "fieldPath": "customer_lifetime_value",
            "tags": null,
            "description": null
          }
        ]
      },
      "tags": {
        "tags": [
          {
            "tag": {
              "urn": "urn:li:tag:test"
            }
          },
          {
            "tag": {
              "urn": "urn:li:tag:jaffle"
            }
          },
          {
            "tag": {
              "urn": "urn:li:tag:shop"
            }
          }
        ]
      }
    }
  }
}
    b
    • 2
    • 2
  • h
    Hello everyone, Just a quick question: is there a way to add a Glossary Business term from the UI? or is using transformers the only way?
    b
    • 2
    • 2
  • a
    Hey does anybody know if Athena is able to perform stateful ingestion? Getting the following Warning inside the logs:
    Copy code
    WARNING  {datahub.ingestion.source.state_provider.datahub_ingestion_checkpointing_provider:135} - No state '
           'available to commit for DatahubIngestionCheckpointingProvider
    Copy code
    source:
    type: athena
    config:
        work_group: xxx
        aws_region: xxx
        s3_staging_dir: '<s3://xxx>'
        stateful_ingestion:
            enabled: true
            remove_stale_metadata: true
            state_provider:
                type: datahub
                config:
                    datahub_api: {server: 'xxx:8080'}
pipeline_name: Athena_Pipeline
sink:
    type: datahub-rest
    config:
        server: 'xxx:8080'
    d
    h
    • 3
    • 4
  • e
    Hi Team I was trying to integrate keycloak OIDC as described here: https://datahubproject.io/docs/how/auth/sso/configure-oidc-react/ do i have defined:
    Copy code
    extraEnvs:
  - name: AUTH_OIDC_ENABLED
    value: "true"
  - name: AUTH_OIDC_CLIENT_ID
    value: sixsense-dev-datahub-frontend
  - name: AUTH_OIDC_CLIENT_SECRET
    value: f70238e0-d17d-469d-8b9a-edcebfa3ba94
  - name: AUTH_OIDC_DISCOVERY_URI
    value: <https://xxx/auth/realms/NonProd/.well-known/openid-configuration>
  - name: AUTH_OIDC_BASE_URL
    value: <http://datahub-datahub-frontend:9002>
    but when i try to reach UI i am getting following error:
    Copy code
    ! @7n8fb50i3 - Internal server error, for (GET) [/callback/oidc] ->
play.api.UnexpectedException: Unexpected exception[CompletionException: org.pac4j.core.exception.TechnicalException: Missing state parameter]
            at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:247)
            at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:176)
            at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:363)
            at play.core.server.AkkaHttpServer$$anonfun$2.applyOrElse(AkkaHttpServer.scala:361)
            at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:346)
            at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:345)
            at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
            at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
            at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
            at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
            at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
            at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
            at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
            at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
            at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
            at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
            at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
            at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
            at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.util.concurrent.CompletionException: org.pac4j.core.exception.TechnicalException: Missing state parameter
            at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
            at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
            at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592)
            at play.core.j.HttpExecutionContext$$anon$2.run(HttpExecutionContext.scala:56)
            ... 6 common frames omitted
Caused by: org.pac4j.core.exception.TechnicalException: Missing state parameter
            at org.pac4j.oidc.credentials.extractor.OidcExtractor.extract(OidcExtractor.java:71)
            at org.pac4j.oidc.credentials.extractor.OidcExtractor.extract(OidcExtractor.java:32)
            at org.pac4j.core.client.BaseClient.retrieveCredentials(BaseClient.java:65)
            at org.pac4j.core.client.IndirectClient.getCredentials(IndirectClient.java:140)
            at org.pac4j.core.engine.DefaultCallbackLogic.perform(DefaultCallbackLogic.java:89)
            at react.auth.AuthModule$2.perform(AuthModule.java:84)
            at react.auth.AuthModule$2.perform(AuthModule.java:79)
            at org.pac4j.play.CallbackController.lambda$callback$0(CallbackController.java:56)
            at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
            ... 7 common frames omitted
    I noticed that datahub uses pac4j as openid client on version 3.6.0. Now the error says that response doesnt contain 
    state
    however does somebody know what should set that state? Or can be omitted by some config? thx
    l
    i
    • 3
    • 2
  • c
    I want to ask a question. I want to know that I can extend a data model to maintain a new model in this way https://datahubproject.io/docs/metadata-modeling/extending-the-metadata-model . For example, extend a term to associate with entities. Is this method feasible?
    d
    • 2
    • 2
  • a
    Anyone has been able to successfully use DataHub with Druid? Any help/pointer is greatly appreciated.
    g
    d
    • 3
    • 36
  • m
    Used docker restart for individual container components, wondering if there is single command that can bring up/down all
    d
    • 2
    • 1
  • w
    Hi all, I deleted a dataset in a Snowflake database, however it's still available in the UI and I can't seem to find a way to tag this dataset as deleted or to remove it from the UI. Do you know how to do this?
    d
    • 2
    • 2
  • m
    Hi, I am trying to deploy a first version of Datahub in Docker using Portainer. I have decided not use Neo4j as I don't have much experience on it and Datahub, as far as I know, is able to work without it. My problem is that it keeps waiting for Neo4j although I haven't specify anything for it.
    b
    e
    • 3
    • 3
  • a
    Hi All, I am looking for some help understanding this file "\Python310\lib\site-packages\datahub\entrypoints.py". I've successfully setup a sandbox environment on a personal machine, however when I try to setup datahub installation using docker desktop inside of my company network I get an error from this file. Perhaps someone (a DEV) could provide some insight into this file and what it requires in order to pull the docker images when the "py -m datahub docker quickstart" command is run. Thanks in advance.
    o
    • 2
    • 3
  • c
    hi，`build/dist/metadata-models-custom-0.0.0-dev.zip`I want to know where this extended metadata model needs to be placed under docker deployment?
    • 1
    • 3
  • m
    c.l.m.g.e.ElasticSearchGraphService:232 - Setting up elastic graph index
    datahub_dh-gms.1.j1zmcfmymrqt      | java.net.ConnectException: Connection refused
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:849)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:259)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.elasticsearch.client.IndicesClient.exists(IndicesClient.java:974)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at com.linkedin.metadata.search.elasticsearch.indexbuilder.ESIndexBuilder.buildIndex(ESIndexBuilder.java:49)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at com.linkedin.metadata.graph.elastic.ElasticSearchGraphService.configure(ElasticSearchGraphService.java:234)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at com.linkedin.metadata.kafka.hook.UpdateIndicesHook.<init>(UpdateIndicesHook.java:80)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:311)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:296)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1389)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1309)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:953)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:921)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:554)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:888)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:357)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1443)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1407)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:821)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:276)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.server.Server.start(Server.java:407)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.server.Server.doStart(Server.java:371)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.runner.Runner.run(Runner.java:520)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.eclipse.jetty.runner.Runner.main(Runner.java:565)
    datahub_dh-gms.1.j1zmcfmymrqt      | Caused by: java.net.ConnectException: Connection refused
    datahub_dh-gms.1.j1zmcfmymrqt      |    at <http://sun.nio.ch|sun.nio.ch>.SocketChannelImpl.checkConnect(Native Method)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at <http://sun.nio.ch|sun.nio.ch>.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
    datahub_dh-gms.1.j1zmcfmymrqt      |    at java.lang.Thread.run(Thread.java:748)
    d
    i
    • 3
    • 7
  • p
    Hello. We have deployed our GMS server behind a reverse proxy that implements SSL/TLS. I was wondering why the frontend wasn't logging in, but have just discovered this: https://github.com/datahub-project/datahub/blob/master/docker/datahub-frontend/env/docker.env#L11
    Copy code
    # Uncomment and set these to support SSL connection to GMS
# NOTE: Currently GMS itself does not offer SSL support, these settings are intended for when there is a proxy in front
#       of GMS that handles SSL, such as an EC2 Load Balancer.
#DATAHUB_GMS_USE_SSL=true
#DATAHUB_GMS_SSL_PROTOCOL=
    I'll add these values to our helm chart for the frontend now, but I wondered if anyone had any relevant experience and time-saving tips for using this kind of setup? Is it only the frontend that needs this configuration, or should I make the environment variable available to all of the containers? I see that the same variable is mentioned in the metdata-service source, but not in the consumers' source code. Am I going to run into trouble with these jobs? Many thanks for any help.
    • 1
    • 1
  • a
    Should urns for custom entities normally start with 
    urn:li
    or is the 
    li
    for LinkedIn entities? Specifically I’m using the Java RestEmitter to send proposals for a custom Dataset Entity, and wondering when I make a urn for my entity if I should use 
    urn:customnamespace
    or 
    urn:li
    .
    o
    • 2
    • 2
  • c
    Hi, I would like to know if there is such a function in Datahub. “When the description in the glue is changed, not overwite, but save the original description and migrate to the new description” Thank you for making such a good program.
    b
    • 2
    • 2
  • n
    One quick help - I am trying to run datahub in aws EKS - Couple of things i need some inputs - How do we point datahub pod when trying to ingest point to our own organization pypy repo proxy ? The other thing is when you want to enable plugins like Glue, Athena, etc . per say , how do we do this ? Now the only thing i was doing is to ssh to pods, do pip install “Datahub-Plugins”, this might be a tidious task going further. Appreciate your help ! Thanks in Advance !
    e
    s
    • 3
    • 12
  • c
    Hi, how can I find ‘docker/datahub-frontend/env/docker.env’ in mac OS?
    b
    s
    • 3
    • 11
  • j
    Hi, everyone! , I want to know how to enable admins to block the ability to view sample data values given by profiling runs in Stats view. Please instruct me on how to do so.
    e
    b
    • 3
    • 8
  • c
    Hi everyone, can someone let me know the difference between the below topics ? MetadataChangeProposal_v1 vs MetadataChangeEvent_v4 MetadataChangeLog_Versioned_v1 vs MetadataAuditEvent_v4 in my deployment only the topics MetadataChangeProposal_v1 & MetadataChangeLog_Versioned_v1 seems to be in use. I could not find any messages in the other topics (MetadataChangeEvent_v4 & MetadataAuditEvent_v4)
    e
    • 2
    • 2
  • f
    as well as would like to know if we have ingestion source as Iceberg. If no, how we could write custom source for Iceberg
    d
    m
    • 3
    • 6
  • c
  • f
    Hi. There is a critical vulnerability CVE-2022-22965 in datahub-gms, datahub-upgrade, datahub-mae-consumer, datahub-mce-consumer docker images, and there is a fix available for this vulnerability, it is stopping us to release datahub in to production. When can expect for new images in which this vulnerability is fixed.
    b
    • 2
    • 1
1...242526...80Latest