https://datahubproject.io logo
Join SlackCommunities
Powered by
# troubleshoot
  • w
    something's wrong with our datahub helm deploy and I'm not sure what to do with it except delete everything and deploy again (which leads me to my previous two threads in #getting-started & #all-things-deployment which seem to have died, asking for useful ways to have datahub not be the master of the data, but rather a view on the data) what do I do when helm times out and e.g. datahub-mysql-setup-job is still running after 22m ?
    ✅ 1
    a
    • 2
    • 2
  • r
    Could someone help me with MSSQL 
    ModuleNotFoundError: No module named 'pyodbc'
    ? That is quickstart docker compose.
    ✅ 1
    a
    • 2
    • 1
  • k
    Hi everyone – I'm having trouble depicting the table lineage in datahub that I've generated in Snowflake. I used the ACCOUNT_HISTORY feature but the lineage doesn't show up in my datahub instance. Would anyone have an idea how to handle this? Thanks a lot in advance
    a
    g
    • 3
    • 3
  • b
    Hey! I have a question about Trino Views in Datahub. Is there native support for adding lineage from a view to the underlying tables? Or do we, as end users, need to find that information and submit it through the API?
    ✅ 1
    a
    • 2
    • 2
  • w
    I am getting follwoing error from ES server:`GET _ilm/policy/datahub_usage_event_policy response code is 000` for elastic search setup job. I have given following in the config:
    Copy code
    port: "9200"
insecure: "true"
useSSL: "true"
skipcheck: "true"
  • w
    Hi team, i have been trying a lot but unable to overcome 
    Caused by: <http://javax.net|javax.net>.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    this exception for Elastic Search. I have also included all the keystore/trusttore values in helm config: Can someone please help me on this....I have been struggling on this for a week now!
    Copy code
    - name: ELASTICSEARCH_HOST
  value: "{{ .Values.global.elasticsearch.host }}"
- name: ELASTICSEARCH_PORT
  value: "{{ .Values.global.elasticsearch.port }}"
- name: SKIP_ELASTICSEARCH_CHECK
  value: "{{ .Values.global.elasticsearch.skipcheck }}"
{{- with .Values.global.elasticsearch.useSSL }}
- name: ELASTICSEARCH_USE_SSL
  value: {{ . | quote }}
{{- end }}
{{- with .Values.global.elasticsearch.auth }}
- name: ELASTICSEARCH_USERNAME
  value: {{ .username }}
- name: ELASTICSEARCH_PASSWORD
  {{- if .password.value }}
  value: {{ .password.value | quote }}
  {{- else }}
  valueFrom:
    secretKeyRef:
      name: "{{ .password.secretRef }}"
      key: "{{ .password.secretKey }}"
  {{- end }}
{{- end }}
- name: ELASTICSEARCH_SSL_PROTOCOL
  value: "{{ .Values.elastic.protocol }}"
- name: ELASTICSEARCH_SSL_TRUSTSTORE_FILE
  value: "{{ .Values.elastic.truststore }}"
- name: ELASTICSEARCH_SSL_TRUSTSTORE_TYPE
  value: "{{ .Values.elastic.trustType }}"
- name: ELASTICSEARCH_SSL_TRUSTSTORE_PASSWORD
  valueFrom:
    secretKeyRef:
      name: {{ .Values.elastic.secretEnv.secretRef }}
      key: {{ .Values.elastic.secretEnv.secretKey }}
- name: ELASTICSEARCH_SSL_KEYSTORE_FILE
  value: "{{ .Values.elastic.keystore }}"
- name: ELASTICSEARCH_SSL_KEYSTORE_TYPE
  value: "{{ .Values.elastic.trustType }}"
- name: ELASTICSEARCH_SSL_KEYSTORE_PASSWORD
  valueFrom:
    secretKeyRef:
      name: {{ .Values.elastic.secretEnv.secretRef }}
      key: {{ .Values.elastic.secretEnv.secretKey }}
    d
    a
    +2
    • 5
    • 9
  • k
    Hi everyone, I've dropped a table in snowflake and committed the changes. Though the lineage in DataHub still shows the dropped table. Can anyone guess why that is and how i fix it? Thanks a lot in advance
    ✅ 2
    g
    • 2
    • 3
  • s
    Hey, can anyone help? Three containers just start and quit: - Container datahub-upgrade - Container elasticsearch-setup - Container kafka-setup
    Copy code
    >datahub docker quickstart --quickstart-compose-file C:\Users\<username>\.datahub\quickstart\docker-compose-without-neo4j.quickstart.yml
Saved quickstart config to C:\Users\<username>/.datahub/quickstart/quickstart_version_mapping.yaml.

[+] Running 12/12
 - Container mysql                      Running                                                                    0.0s
 - Container zookeeper                  Running                                                                    0.0s
 - Container datahub-upgrade            Started                                                                    1.0s
 - Container mysql-setup                Running                                                                    0.0s
 - Container broker                     Running                                                                    0.0s
 - Container elasticsearch              Running                                                                    0.0s
 - Container datahub-gms                Running                                                                    0.0s
 - Container schema-registry            Running                                                                    0.0s
 - Container kafka-setup                Started                                                                    1.1s
 - Container datahub-datahub-actions-1  Running                                                                    0.0s
 - Container elasticsearch-setup        Started                                                                    1.0s
 - Container datahub-frontend-react     Running                                                                    0.0s
    Could this involve MySQL? I have an other process that is using port 3306, so I changed all the 3306 ports in the docker-compose-without-neo4j.quickstart.yml file to 3307.
    Copy code
    PS C:\Users\jdunson> Get-NetTCPConnection | where Localport -eq 3306  | select Localport,OwningProcess

Localport OwningProcess
--------- -------------
     3306         18956
     3306         11160
    g
    a
    • 3
    • 6
  • s
    image.png
  • h
    Team, Using 0.9.3 version of Datahub. we have OIDC authentication enabled. Unable to connect to datahub using admin account nor via Oauth. GMS logs
    Copy code
    2023-03-10 21:19:38.443:WARN:oejsh.ErrorHandler:qtp1125736023-88: Error page too large: 500 javax.servlet.ServletException: org.springframework.web.util.NestedServletException: Async processing failed; nested exception is java.lang.StackOverflowError Request(POST //<hostname>/api/graphql)@e82557e
2023-03-10 21:19:38.444:INFO:oejsh.ErrorHandler:qtp1125736023-88: Disabling showsStacks for ErrorPageErrorHandler@295c6a0c{STARTED}
Exception in thread "gmsEbeanServiceConfig.heartBeat" java.lang.RuntimeException: invalid key or spec in GCM mode
    ✅ 1
    g
    a
    • 3
    • 3
  • w
    I am trying to install datahub in docker using "datahub docker quickstart". I keep getting this error even the status show datahub-gm is already running: [+] Running 12/12 ⠿ Container elasticsearch Running 0.0s ⠿ Container zookeeper Running 0.0s ⠿ Container mysql Running 0.0s ⠿ Container elasticsearch-setup Started 0.5s ⠿ Container mysql-setup Started 0.5s ⠿ Container datahub-upgrade Running 0.0s ⠿ Container broker Running 0.0s ⠿ Container datahub-gms Running 0.0s ⠿ Container schema-registry Running 0.0s ⠿ Container kafka-setup Started 0.5s ⠿ Container datahub-datahub-actions-1 Running 0.0s ⠿ Container datahub-frontend-react Running 0.0s .............. Unable to run quickstart - the following issues were detected: - datahub-gms is still starting
    a
    b
    a
    • 4
    • 51
  • w
    I have 16GB memory for desktop docker running on my macbook pro (M1)
  • w
    Any hint what might be wrong. Thanks for any help
  • w
    I noticed in the docker dashboard, all container has a warning: AMD64 warning sign and when hover over, it says: "image may have poor performance or fail, if run via emulation."
    ✅ 1
    a
    • 2
    • 1
  • w
    Tried with: datahub docker quickstart --arch M1. No differences
  • a
    Hello, currently I’m installing datahub in gcp and using helm chart but in my workloads I got these errors, did anyone know how to fix this?
    👀 1
    ✅ 1
    a
    i
    • 3
    • 32
  • e
    Hi, everyone. When I try to run latest quickstart with latest python package I get
    Copy code
    Unable to run quickstart - the following issues were detected:
- datahub-gms is running by not yet healthy
    Any help would be greatly appreciated.
    tmpoo_xr3ny.log
    plus1 2
    r
    a
    • 3
    • 5
  • r
    Hi every one i have got a issue in datahub-action docker-datahub-actions-1 | 2023/03/13 132848 Problem with request: Get “http///health” http: no Host in request URL. Sleeping 1s Can anyone help me?
    ✅ 1
    a
    • 2
    • 1
  • b
    hi, I have a custom connector that supports stateful ingestion. and it soft deletes stale metadata as it should. but if I add back the soft-deleted entities (such that they get picked up by the recipes again) they do no get reinstated. is this by design or am I missing something? edit: fixed this by using 
    make_dataset_urn_with_platform_instance
    rather than 
    make_dataset_urn
    when generating urns for dataset entities.     edit2: the above just produced duplicates with slight variation in the urn and I mistakenly thought these were soft-deleted entries being reinstated. the real fix, for anyone reading this in the future, is to add the aspect 
    Status(removed=False)
    to your dataset snapshot object, for example
    Copy code
    snapshot = DatasetSnapshot(
  urn=dataset_urn,
  aspects=[Status(removed=False)],
)
    thanks bear 1
    ✅ 1
  • m
    Hi, i'm having trouble using SQL Server, i'm getting the message / PipelineInitError: Failed to configure the source (mssql): No module named 'pyodbc' / I read other answers with the same problem, but i got nothing.
    a
    • 2
    • 4
  • d
    in kubernetes i have some issues with datahub-gms ERROR c.l.m.s.e.query.ESSearchDAO:61 - Search query failed org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahubpolicyindex_v2]] at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911)
    plus1 1
    h
    • 2
    • 1
  • a
    hello, datahub-gms is giving these errors in v0.10.0 :
    Copy code
    2023-03-14 12:14:56,367 [pool-11-thread-1] ERROR c.l.m.s.e.query.ESSearchDAO:61 - Search query failed
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index [datahubpolicyindex_v2]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1911)
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1888)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1645)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1602)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1572)
	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1088)
	at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.executeAndExtract(ESSearchDAO.java:57)
	at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.search(ESSearchDAO.java:90)
	at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.fullTextSearch(ElasticSearchService.java:111)
	at com.linkedin.metadata.client.JavaEntityClient.search(JavaEntityClient.java:312)
	at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:50)
	at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:42)
	at com.datahub.authorization.DataHubAuthorizer$PolicyRefreshRunnable.run(DataHubAuthorizer.java:223)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [<https://vpc-awie-es-dataeng-dh-01-r3us77zzpucodobuxpbyicgmgu.eu-west-1.es.amazonaws.com:443>], URI [/datahubpolicyindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [datahubpolicyindex_v2]","resource.type":"index_or_alias","resource.id":"datahubpolicyindex_v2","index_uuid":"_na_","index":"datahubpolicyindex_v2"}],"type":"index_not_found_exception","reason":"no such index [datahubpolicyindex_v2]","resource.type":"index_or_alias","resource.id":"datahubpolicyindex_v2","index_uuid":"_na_","index":"datahubpolicyindex_v2"},"status":404}
    I ran the elasticsearch-setup-job. Is there any special parameter needed there? why are the indexes missing?
    h
    a
    a
    • 4
    • 9
  • a
    I am using kubernetes and I am using the helm chart from acryl. I ran all the init jobs and the upgrade job. However, in datahub-gms I am getting the following error:
    Copy code
    org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer$ListenerConsumerRebalanceListener failed on invocation of onPartitionsAssigned for partitions [DataHubUpgradeHistory_v1-0]
java.lang.IllegalArgumentException: seek offset must not be a negative number
	at org.apache.kafka.clients.consumer.KafkaConsumer.seek(KafkaConsumer.java:1599)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer$InitialOrIdleSeekCallback.seek(KafkaMessageListenerContainer.java:3075)
	at com.linkedin.metadata.kafka.boot.DataHubUpgradeKafkaListener.lambda$onPartitionsAssigned$1(DataHubUpgradeKafkaListener.java:70)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
	at java.base/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1764)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at com.linkedin.metadata.kafka.boot.DataHubUpgradeKafkaListener.onPartitionsAssigned(DataHubUpgradeKafkaListener.java:69)
	at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.onPartitionsAssigned(MessagingMessageListenerAdapter.java:302)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.seekPartitions(KafkaMessageListenerContainer.java:1127)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.access$3800(KafkaMessageListenerContainer.java:518)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer$ListenerConsumerRebalanceListener.onPartitionsAssigned(KafkaMessageListenerContainer.java:2968)
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.invokePartitionsAssigned(ConsumerCoordinator.java:278)
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:419)
	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:439)
	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:358)
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:490)
	at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1275)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1241)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1414)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1251)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1163)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
    In my understanding, that means that the application is trying to seek for messages in a negative offset. Is this the error? Why is this happening? Any ideas?
    a
    a
    • 3
    • 4
  • t
    Hi everyone, I am currently trying to configure mongo datasource with slave member of a replicaset through cross account VPC(AWS Privatelink) in datahub, our analytics and production are hosted in two different AWS accounts, when i am trying ingestion on mongodb dataset, it is getting failed as it is trying to resolve the actual hostnames in the replica set though we have configured the connection uri using privatelink dns. Any ideas on how to resolve this?
    ✅ 1
    a
    b
    • 3
    • 2
  • w
    Hello
    ✅ 1
  • w
    I have just added 55 domains
  • w
    1. sorting does not work properly
    b
    a
    b
    • 4
    • 5
  • w
    2. there 3 pages but I cannot access page 2
  • w
    or 3
  • w
    this is a showstopper
1...828384...119Latest