i have the same issue

I am facing the same issue. How do you guys resolved it? Any solutions?

Datahub-LocalDevelopmentBuild.txt

bumping this, it keeps creating

../java_pid69259.hprof

type files locally which I think is contributing to the heap space error

And, FWIW, I don’t see quickstart in any of the projects of the gradle wrapper: Not even sure if it should be here.

[jgarza@fedora datahub]$ ./gradlew projects

Root project 'datahub'
+--- Project ':datahub-frontend'
+--- Project ':datahub-graphql-core'
+--- Project ':datahub-ranger-plugin'
+--- Project ':datahub-upgrade'
+--- Project ':datahub-web-react'
+--- Project ':docs-website'
+--- Project ':entity-registry'
|    \--- Project ':entity-registry:custom-test-model'
+--- Project ':ingestion-scheduler'
+--- Project ':li-utils'
+--- Project ':metadata-dao-impl'
|    \--- Project ':metadata-dao-impl:kafka-producer'
+--- Project ':metadata-events'
|    +--- Project ':metadata-events:mxe-avro-1.7'
|    +--- Project ':metadata-events:mxe-registration'
|    +--- Project ':metadata-events:mxe-schemas'
|    \--- Project ':metadata-events:mxe-utils-avro-1.7'
+--- Project ':metadata-ingestion'
+--- Project ':metadata-ingestion-modules'
|    \--- Project ':metadata-ingestion-modules:airflow-plugin'
+--- Project ':metadata-integration'
|    \--- Project ':metadata-integration:java'
|         +--- Project ':metadata-integration:java:datahub-client'
|         +--- Project ':metadata-integration:java:datahub-protobuf'
|         \--- Project ':metadata-integration:java:spark-lineage'
+--- Project ':metadata-io'
+--- Project ':metadata-jobs'
|    +--- Project ':metadata-jobs:mae-consumer'
|    +--- Project ':metadata-jobs:mae-consumer-job'
|    +--- Project ':metadata-jobs:mce-consumer'
|    +--- Project ':metadata-jobs:mce-consumer-job'
|    \--- Project ':metadata-jobs:pe-consumer'
+--- Project ':metadata-models'
+--- Project ':metadata-models-custom'
+--- Project ':metadata-models-validator'
+--- Project ':metadata-perf'
+--- Project ':metadata-service'
|    +--- Project ':metadata-service:auth-api'
|    +--- Project ':metadata-service:auth-filter'
|    +--- Project ':metadata-service:auth-impl'
|    +--- Project ':metadata-service:auth-ranger-impl'
|    +--- Project ':metadata-service:auth-servlet-impl'
|    +--- Project ':metadata-service:factories'
|    +--- Project ':metadata-service:graphql-servlet-impl'
|    +--- Project ':metadata-service:openapi-servlet'
|    +--- Project ':metadata-service:restli-api'
|    +--- Project ':metadata-service:restli-client'
|    +--- Project ':metadata-service:restli-servlet-impl'
|    +--- Project ':metadata-service:servlet'
|    \--- Project ':metadata-service:war'
+--- Project ':metadata-testing'
|    +--- Project ':metadata-testing:metadata-models-test-utils'
|    \--- Project ':metadata-testing:metadata-test-utils'
+--- Project ':metadata-utils'
+--- Project ':smoke-test'
\--- Project ':test-models'

Thanks!

Bumping this: I am trying to get relationship data from one entity inside the mapper file

Anyone? Or maybe #all-things-deployment is the right place for this?

bumping this again since I'm blocked on development, any help is appreciated!

Just wanted to follow up and see if anyone had any other ideas for this issue. Thanks!!!

It seems that I have the same problem. I tried to upgrade datahub, but it kafka-setup, elasticsearch-setup and mysql-setup doesn't run after upgrade. I attached the log as well. Thank you for looking into this.

Hi Team..I am using v0.9.5 experiencing similar kind of issue in search UI. As a test scenario I have 6k+ records with wild card(*) search and I have set page size as 10. While try to view the last page records by clicking the pagination option getting datahub-gms | WARN [DefaultHandlerExceptionResolver] Resolved [org.springframework.web.context.request.async.AsyncRequestTimeoutException] in docker log of GMS container. Please advice what kind of configuration help us to resolve the issue.

Hi @able-evening-90828..your observation is correct. Recently I have also faced the same issue in GraphQL ..I have tried with graphQL search query to find out schema field level tags but it seems the search api is only focused on fieldTags but not on editableFieldTags field. So the api return partial result but those schema level tags that we have added from Datahub UI is not coming from the GraphQL. So I think Datahub team should look into this. @little-megabyte-1074 --Can u pls look into this problem. The sample Graph QL I have used posted below : { search(input: {type: DATASET, query: "editFieldTags:not null", start: 0, count: 50, }) { start count total searchResults { entity { urn ... on Dataset { urn type subTypes { typeNames } name platform { name } properties { name } schemaMetadata { fields { fieldPath jsonPath label tags { tags { tag { urn properties { name description } } } } } } } } } } }

Hi team, Can someone assist pls. pls.?

HI! I'm already tried

datahub docker quickstart  --version v0.9.6.1

with the same errors

another thing we noticed around this is that once an entity is soft-deleted, if it’s seen again in a future ingestion, it won’t be un soft-deleted, is that a bug or intended behavior? I’m not sure I agree with it, wondering if we might be better off churning through all urns every so often and handling soft-deletion/un-soft-deletion in a separate job based on whether there’s a recent update to a lastObserved timestamp on any aspect?

yes I was able to, now for some reason I cant

I haven't had a chance to retry, but I'm not sure what to try next. Pretty certain it isn't a memory/CPU thing as advised above. They've come out with a new update, so I was hoping to try it out again soon.

I try to use the guide under link but i got the other issue: Exception in thread "main" java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required ( Forefront TMG requires authorization to fulfill the request. Access to the Web Proxy filter is denied. )" But we dont have user and password in pur proxies

I am also getting the same error on 10.1. I am ingesting a looker source and for all intents and purposes, DH is logging in, scanning all assets, but hitting that error towards the end.

TypeError: 'str' object is not callable

Here are the last few lines

File "/tmp/datahub/ingest/venv-looker-0.10.0/lib/python3.10/site-packages/datahub/utilities/sql_lineage_parser_impl.py", line 8, in <module>
    from sqllineage.core.holders import Column, SQLLineageHolder
File "/tmp/datahub/ingest/venv-looker-0.10.0/lib/python3.10/site-packages/sqllineage/__init__.py", line 41, in <module>
    _monkey_patch()
File "/tmp/datahub/ingest/venv-looker-0.10.0/lib/python3.10/site-packages/sqllineage/__init__.py", line 35, in _monkey_patch
    _patch_updating_lateral_view_lexeme()
File "/tmp/datahub/ingest/venv-looker-0.10.0/lib/python3.10/site-packages/sqllineage/__init__.py", line 24, in _patch_updating_lateral_view_lexeme
    if regex("LATERAL VIEW EXPLODE(col)"):

#ingestion

When installing the

sqlalchemy version==1.4.20

this warning disappears. However, the errors described above remain. It is possible that data profiling is not fully performed due to these errors. No statistics are collected for numeric fields, except those that are independent of the data type (Null and Distinct). Please write if this behavior is expected, and while the extraction from Oracle DB is inferior compared to Postgres (for example).

On the same topic. How can the variable REST_API_AUTHORIZATION be enabled to be true via the helm charts during deployment? I assume with an the extraenvs like this underneath gms*?* I'm trying to avoid users using Swagger without authentication.

extraEnvs:

- name: REST_API_AUTHORIZATION_ENABLED

value: "true"

Hi team, I'm still unable to solve this. Any inputs ?

@flat-painter-78331, check this section of the datahub-system-update-job.yml (/helm/datahub-0.2.161/datahub/templates/datahub-upgrade/datahub-system-update-job.yml). Somewhere around line 14 there was a pre-install hook "helm.sh/hook": pre-install,pre-upgrade The fix was to change back to a post-install hook, like prior helm chart versions had. "helm.sh/hook": post-install,post-upgrade

I m currently using 0.10.1 version

Just opened a ticket https://github.com/datahub-project/datahub/issues/7998

So the postgres got sorted out but still struggling with MongoDB. To reiterate the issue - running MongoDB on my localhost but cannot access from datahub docker. Tried using private IP instead of localhost as well; ufw opened up the ports to allow any IP

datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure the source (mongodb): 192.168.124.14:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 6459fcfbbdda95b6863dbf5d, topology_type: Unknown, servers: [<ServerDescription ('192.168.124.14', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('192.168.124.14:27017: [Errno 111] Connection refused')>]>

SOLVED version of DataHub : 0.10.2.2 deployment method: via docker and offline OS: Redhat 8.6 with docker-ce 20.10.24 docker compose 2.17.3 1- on a PC connected to internet) • I've previouly downloaded images (via datahub docker quickstart) • I saved images (docker save) • I exported image on the offline server 2- on the offline server • I've imported images (docker load) • I followed your advice: All containers are built but I have an error for the broker (kafka) : "dependency failed to start: container broker exited" I can't run the broker, so I can't have the log inside the container. So, I've stopped all containers and erased data: datahub docker nuke Then I've run again datahub docker quickstart --quickstart-compose-file= .... and this time all is OK !!!!! error on the broker

please help

HI @modern-artist-55754, thanks for the reply . Would you be able to give us more details like what port and connectivity needs to be open. Out of 3 url from repositories.gradle, I can get the details only for 1 but while doing the telnet check with IP and 443 port , its connecting. Also from our side , FW has been enable to accept the secure port (443) can communicate . Any help or lead would be great help

Hi, I still need help on Analytics issue: Our Analytics page does not display info properly, and we discovered that the getHighlghts and getAnalyticsChart queries on it return empty from backend. What can be done to make it right? I weren't able to find an answer in documentation