if we use Postgres as our

sql

datasource, is there an officially documented way to use a specific schema within that Postgres database? like just

datahub

instead of the defalut

public

schema? A use case is using the same postgres database for metadata across multiple systems, one of which would be DataHub (others could be Airflow, Superset, etc.)

Hi With the VPN connected, I'm not able to connect to https://helm.datahubproject.io/ So what are the other options to run helm repo add datahub https://helm.datahub project.io/ Any alternative to do it locally without connected to Internet?

Hi Vertica ingestion was working and suddenly start to fail, error:

datahub.configuration.common.ConfigurationError: vertica is disabled; try running: pip install 'acryl-datahub[vertica]'

but from venv it works

source /python_ve/bin/activate
(python_ve) [root@dp-test006 ~]# python
Python 3.6.8 (default, Nov 16 2020, 16:55:22)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from sqlalchemy_vertica.base import VerticaDialect

This message was deleted.

To ingest metadata from hive, I can see from UI ingestion or from yml file. If we want to write some crawler, Is there any guide or documents regarding that ?

Hi, The airflow integration was working fine. I am getting the below error suddenly. What might have caused this?

[2022-09-06, 14:55:50 UTC] {base.py:68} INFO - Using connection ID 'datahub_rest_default' for task execution.
[2022-09-06, 14:55:50 UTC] {logging_mixin.py:115} INFO - Exception: Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/datahub_airflow_plugin/datahub_plugin.py", line 318, in custom_on_failure_callback
    datahub_on_failure_callback(context)
  File "/home/airflow/.local/lib/python3.9/site-packages/datahub_airflow_plugin/datahub_plugin.py", line 125, in datahub_on_failure_callback
    DatahubGenericHook(context["_datahub_config"].datahub_conn_id)
  File "/home/airflow/.local/lib/python3.9/site-packages/datahub_provider/hooks/datahub.py", line 77, in make_emitter
    return datahub.emitter.rest_emitter.DatahubRestEmitter(*self._get_config())
AttributeError: module 'datahub' has no attribute 'emitter

I’ve seen some comments in this thread about Elasticsearch on AWS, does this mean that people are using Elasticsearch proper? or AWS’s Open Search? I’m trying the latter and am having difficulties getting the startup job to work.

Hi I'm trying to compile the datahub code, but it gives error saying it couldn't get Openjdk11u-jdk_x64_mac_hotspot_11.0.16.1_1.tar.go , but I'm not getting why it's trying to download this jdk 11 when I'm trying to use java 8. Can anyone help me on this ? I'm stuck at building the code

hi,team。I implemented user-defined login verification, but I found that although I had logged in to the page, 401 would still be returned. I request the [getme] interface to pass my own authentication header, but when I got the request from the page, I found that there was a cookie play_session is found to be unable to pass the verification. Please customize the implementation of play after login_ How to set the play_session?

I'm facing the unable to find valid “certification path to requested target “ issue when compiling the code and in the datahub-protobuf module , has anyone faced this problem before ? How to solve this ? And it's trying to download some open jdk 11 tar, eventhough I'm using java 8. Help help would be great here . Thanks

New to the DataHub ecosystem, What will be the best way to run sql query on Mysql database from Datahub ?

@bumpy-needle-3184, as we can see db_name and db_user are is different in base repo itself, let us know why it could be same for our case? https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#:~:text=sql%3A,mysql%2Droot%2Dpassword INSERT INTO metadata_aspect_v2 SELECT * FROM temp_metadata_aspect_v2 WHERE NOT EXISTS (SELECT * from metadata_aspect_v2); DROP TABLE temp_metadata_aspect_v2; psql: error: connection to server at "10.240.154.202", port 5432 failed: FATAL: database "datahub" does not exist 2022/09/08 054322 Command exited with error: exit status 2

Hey hey folks 🙂 Deploying on Azure, trying to set some specific values for datahub helm chart, i'm having this "new" error to me :

coalesce.go220 warning: cannot overwrite table with non table for datahub.datahub-frontend.env (map[JMXPORT:1099])

coalesce.go220 warning: cannot overwrite table with non table for datahub.datahub-gms.env (map[JMXPORT:1099])

hi,team。Today, I suddenly found that the graphqL response speed of the interface was very slow, After debugging, it is found that the searchacrossentitiesresolver has finished processing in about 1 s, but it returns to the front end until 6-7 S. do you know why？ 170814.762 [ForkJoinPool.commonPool-worker-6] INFO c.l.m.search.cache.CacheableSearcher:75 - CacheableSearcher->getSearchResults 239ms 2022-09-08T170814.768+0800 [QUIET] [system.out] 170814.770 [ForkJoinPool.commonPool-worker-7] INFO c.l.m.s.a.AllEntitiesSearchAggregator:138 - AllEntitiesSearchAggregator.search 1112ms 2022-09-08T170814.768+0800 [QUIET] [system.out] 170814.770 [ForkJoinPool.commonPool-worker-7] INFO c.l.m.search.cache.CacheableSearcher:75 - CacheableSearcher->getSearchResults 1112ms 2022-09-08T170814.768+0800 [QUIET] [system.out] 170814.770 [ForkJoinPool.commonPool-worker-7] INFO c.l.entity.client.JavaEntityClient:313 - JavaEntityClient->searchAcrossEntities.search 1116ms 2022-09-08T170814.771+0800 [QUIET] [system.out] 170814.772 [ForkJoinPool.commonPool-worker-7] INFO c.l.d.g.r.s.SearchAcrossEntitiesResolver:62 - SearchAcrossEntitiesResolver->get 1118ms 2022-09-08T170817.273+0800 [QUIET] [system.out] 170817.271 [ForkJoinPool.commonPool-worker-3] INFO c.l.datahub.graphql.GraphQLEngine:100 - _graphQL->execute 3667ms

I find my query is slow. I want to try to find optimization points

what should the value of

KAFKA_SCHEMAREGISTRY_URL

be in the helm chart if we are using AWS_GLUE?

where do the database migrations for DataHub happen? I’m getting this error:

Query threw SQLException:ERROR: relation "metadata_aspect_v2" does not exist

I retype the image and report an error version 0.8.41 I didn’t change anything @green-football-43791 @loud-island-88694 @dazzling-judge-80093

01:59:36.622 [main] WARN  o.s.w.c.s.XmlWebApplicationContext:591 - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanDefinitionStoreException: Failed to parse configuration class [com.linkedin.gms.factory.common.GitVersionFactory]; nested exception is java.io.FileNotFoundException: class path resource [git.properties] cannot be opened because it does not exist
01:59:36.651 [main] ERROR o.s.web.context.ContextLoader:313 - Context initialization failed
org.springframework.beans.factory.BeanDefinitionStoreException: Failed to parse configuration class [com.linkedin.gms.factory.common.GitVersionFactory]; nested exception is java.io.FileNotFoundException: class path resource [git.properties] cannot be opened because it does not exist
        at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:189)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:331)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:247)

@bumpy-needle-3184 and Team, Please advise. ERROR: Cannot connect to GMSat host sg-rcube-datahub-datahub-gms port 8080. Make sure GMS is on the latest version and is running at that host before starting the migration. Failed Step 2/6: GMSQualificationStep. Failed after 2 retries. Exiting upgrade NoCodeDataMigration with failure. Upgrade NoCodeDataMigration completed with result FAILED. Exiting...

Please confirm latest version

@bumpy-needle-3184 Please advise.[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 6.1.4-ccs [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: c9124241a6ff43bc [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1662720480602 WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. Error while executing config command with args '--command-config /tmp/connection.properties --bootstrap-server kafka-886515205-1-1200869063.scus.kafka-sams-edf-stg.ms-df-messaging.stg-az-southcentralus-8.prod.us.walmart.net:9092,kafka-886515205-2-1200869066.scus.kafka-sams-edf-stg.ms-df-messaging.stg-az-southcentralus-8.prod.us.walmart.net:9092,kafka-886515205-3-1200869069.scus.kafka-sams-edf-stg.ms-df-messaging.stg-az-southcentralus-8.prod.us.walmart.net:9092,kafka-886515205-4-1200869072.scus.kafka-sams-edf-stg.ms-df-messaging.stg-az-southcentralus-8.prod.us.walmart.net:9092,kafka-886515205-5-1200869075.scus.kafka-sams-edf-stg.ms-df-messaging.stg-az-southcentralus-8.prod.us.walmart.net:9092,kafka-886515205-6-1200869078.scus.kafka-sams-edf-stg.ms-df-messaging.stg-az-southcentralus-8.prod.us.walmart.net:9092 --entity-type topics --entity-name _schemas --alter --add-config cleanup.policy=compact' java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272) at kafka.admin.ConfigCommand$.getResourceConfig(ConfigCommand.scala:552) at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:322) at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:302) at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97) at kafka.admin.ConfigCommand.main(ConfigCommand.scala) Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException:

hello, I am trying to deploy datahub v0.8.44 and I am getting this error in mae consumer, any ideas?:

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributorRegistry]: Factory method 'healthContributorRegistry' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'solrHealthContributor' defined in class path resource [org/springframework/boot/actuate/autoconfigure/solr/SolrHealthContributorAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributor]: Factory method 'solrHealthContributor' threw exception; nested exception is java.lang.NoClassDefFoundError: org/apache/solr/client/solrj/impl/BaseHttpSolrClient$RemoteSolrException
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
	... 26 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'solrHealthContributor' defined in class path resource [org/springframework/boot/actuate/autoconfigure/solr/SolrHealthContributorAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributor]: Factory method 'solrHealthContributor' threw exception; nested exception is java.lang.NoClassDefFoundError: org/apache/solr/client/solrj/impl/BaseHttpSolrClient$RemoteSolrException
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:671)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:659)
	at org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:1300)
	at org.springframework.boot.actuate.autoconfigure.health.HealthEndpointConfiguration.healthContributorRegistry(HealthEndpointConfiguration.java:82)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	... 27 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributor]: Factory method 'solrHealthContributor' threw exception; nested exception is java.lang.NoClassDefFoundError: org/apache/solr/client/solrj/impl/BaseHttpSolrClient$RemoteSolrException
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
	at

Reposting here as well.

Hi I'm trying to ingest hive metadata from UI. I have started hive server on 10000 port, and in UI I have given host as localhost:10000 , but it's failing to ingest with error as could not connect to any of ‘127.0.0.1’, 10000 Am I missing anything here ?

@bumpy-needle-3184 and team, Could advise us, how did we setup schema registry got kafka setup.

Kafka setup job failing. [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1662954739489 WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. Error while executing config command with args '--command-config /tmp/connection.properties --bootstrap-server kafka-886515205-1-1200869063.scus.kafka-sams-edf-stg.ms-df-messaging.stg-az-southcentralus-8.prod.us.walmart.net:9092,kafka-886515205-2-1200869066.scus.kafka-sams-edf-stg.ms-df-messaging.stg-az-southcentralus-8.prod.us.walmart.net:9092,kafka-886515205-3-1200869069.scus.kafka-sams-edf-stg.ms-df-messaging.stg-az-southcentralus-8.prod.us.walmart.net:9092,kafka-886515205-4-1200869072.scus.kafka-sams-edf-stg.ms-df-messaging.stg-az-southcentralus-8.prod.us.walmart.net:9092,kafka-886515205-5-1200869075.scus.kafka-sams-edf-stg.ms-df-messaging.stg-az-southcentralus-8.prod.us.walmart.net:9092,kafka-886515205-6-1200869078.scus.kafka-sams-edf-stg.ms-df-messaging.stg-az-southcentralus-8.prod.us.walmart.net:9092 --entity-type topics --entity-name _schemas --alter --add-config cleanup.policy=compact'

As were not using AWS/GCP glue for kafka service, couldn't configure the schema registry.

Hello everyone, I am trying to setup datahub on our private cluster (using kubeadm), but everytime I try to install the helm chart I get:

Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"

After searching, I found that kubernetes latest versions don't support some type of api's such as

PodDisruptionBudget

. I wanna see if there will be a newer version that works with the latest versions of kubernetes or is there another solution.