This message was deleted.

Hello, I'm on an older version from 2022, and Im working on upgrading to the newest version. The datahub is on a single aws ec2 linux instance, installed with datahub docker quickstart, and a custom docker-compose file to enable OIDC. I ran python3 -m pip install --upgrade pip wheel setuptools python3 -m pip install --upgrade acryl-datahub datahub version I also updated the system with: sudo yum update -y After this I get the error bad gateway, I then ran: docker-compose -p datahub -f docker-compose.yml -f docker-compose.yml up -d datahub-frontend-react since I did see that only one docker was running. After this I see that 4 dockers are running, datahub-frontend-react, datahub-gms, mysql:5.7 and datahub-actions. I then get the red error with: Oops, an error occurred This exception has been logged with id 810jjdhed. Any idea what I can do to fix this? The next plan was to upgrade docker compose to v2 and then run datahub docker quickstart.

Hello, I am trying to ingest some metadata to Datahub from Power BI using a recipe, and i would like to make modification of the dataset identifier before ingestion, more precisely to change the table name all into capital letters. For this purpose i can use transformers, but i am not sure if there is already a provided transformer that i can use, or in order to solve this i need to make a custom transformer? Your opinion will mean a lot to me. Thanks 🙂

can anyone tell me why bigquery doesn’t support platform instances? is it just a product decision to block it or is there an actual technical issue behind this decision?

Does anyone know why inserting a new role membership for a user directly into MySQL table

metadata_aspect_v2

doesn't reflect the role on the UI? I've inserted the aspect in the same format as the UI does. I've set column

version

=0, to make it the most recent. Even after deleting the old roleMembership aspect rows from the database for this user, it still shows up on the UI with the role that has already been deleted from the database ("Reader") instead of the new role ("Admin"). I've refreshed the UI, and even signed in from a new incognito window, so it's not browser caching. But it looks like there's some other sort of caching in the backend service. I know we can set roles using the graphQL API, but I'm exploring Datahub architecture/code and I'd like to understand why this happens. Thank you.

Hi Team, Trying to upgrade to 0.9.3( by updating the images and the helm values.yml ) from 0.8.45, Getting errors in UI and unable to access GMS

hello Datahub team, I'm running into an issue with custom metadata models in datahub 0.9.6.1 and 0.10.0; making a plugin (via

metadata-models-custom

) to add a custom aspect to the

container

entity results in this error:

java.lang.UnsupportedOperationException: Failed to find Snapshot associated with Config-based Entity

this seems to be caused by datahub expecting the base entity to have an entity snapshot, which containers do not have; is this intended behavior? I was under the impression that snapshot-based entities have been deprecated for over a year now

Hi Team, was able to finally upgrade to 0.9.3 , but notice that frontend authenticates continuously. Any suggestions for this? Reference Issue: found https://datahubspace.slack.com/archives/CV2UVAPPG/p1675671029931549?thread_ts=1675669380.773319&cid=CV2UVAPPG

Hey, I am running

v0.10.0

and was trying to figure if I can do an advanced filter with an expression like

Domain = null

? Can't figure this out in the UI, maybe I am missing something?

Hi everyone. datahub I'm trying to build the project, however i run into an error when trying to build the whole project with: ./gradlew build I then tried running the partially builds, and found that the problem were with: ./gradlew metadata ingestioninstallDev I have tried to debug it, but I cant seem to find the solution. 🫠 I get the following error:

Hey guys! I am using Datahub with Great Expectations. I connected an a instance of GE with Teradata database. When running the expectations there is no display in the validation tab in the Datahub UI. Only with Teradata i get this problem. Has someone any clues? 😄

getting the following error in the UI trying to view lineage for snowflake lineage

Failed to load results! An unexpected error occurred.

the graph lineage works and lineage works for my looker objects, any ideas? I think im just going to restart the pods and see if that helps but any ideas appreciated

👋 hopefully an easy one… I’ve forked the datahub project and am trying to install it via poetry but couldn’t find docs on how to do this. I’ve tried the following in pyproject.toml but it’s definitely not right…

[tool.poetry.dependencies]
python = "^3.9"
acryl-datahub = { git = "<https://github.com/FundingCircle/datahub.git>", subdirectory = "metadata-ingestion", branch = "master", extras = ["datahub-rest","dbt","glue","kafka","postgres","tableau"] }

Hi Team, lineage got wiped out after upgrading datahub to v0.10.0.

Hello everyone, we’re trying to bring Datahub to production and struggle with the Kubernetes setup. The issue is basically very simple - we can’t make any ingestion work. It does not execute ingestion tasks and Status is shown as “Pending...“. datahub-gms shown only one single log line: 123544.420 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:47 - Successfully fed bulk request. Number of events: 4 Took time ms: -1 Trying the same on a quickstart installation works fine. Given our lack of familiarity with how Datahub work on the inside we’re stuck. Does that look like a common problem? Raising log verbosity did not provide us with additional insights. Any help is much appreciated. Thanks!

hi folks ! I am using data hub , but I want to restore my datahub information . I followed the commands and steps to create backup.sql file. When I downloaded datahub on a new sytema and used the command to restore that sql backup file , I expected to get same information and restored databut I didnt any one has done it or can help me out ?

Hi everyone – I'm currently trying to Ingest Datasets from Azure Synapse using MSSQL and I get the following error response regardless of the driver version I'm using: Command failed: Failed to configure the source (mssql): (pyodbc.Error) ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 13 for SQL Server' : file not found (0) (SQLDriverConnect)") (Background on this error at: https://sqlalche.me/e/14/dbapi) Does anyone have an idea on how to fix that? Thanks a lot 🙂

Hi all! I'm trying to inject a dataset and datajob through a .json file, but I get the following error "com.linkedin.pegasus2avro.usage.UsageAggregation is missing required field: bucket". This field is not mentioned in the documentation for the description of the entity. Where should it be placed, what is it responsible for and how does it work? Thanks!

Hi Team, we have recently upgraded to datahub v0.10.0 using helm, but stateful ingestion was not working there is some patch released for that, but when we try to get those changes and deploy gms using our custom image we see this issue and gms is not starting: 2023-03-17 124339,944 [ThreadPoolTaskExecutor-1] INFO c.l.m.b.k.DataHubUpgradeKafkaListener:84 - Latest system update version: v0.10.0.6-1 2023-03-17 124339,944 [ThreadPoolTaskExecutor-1] INFO c.l.m.b.k.DataHubUpgradeKafkaListener:88 - System version is not up to date: null-1. Waiting for datahub-upgrade to complete...

Hi Team, We have ingested from two sources -- athena and an s3 bucket. the athena table's location is the s3 bucket, however in the lineage graph, there is no connection between the two. Instead, the lineage creates a new s3 entity of the same name as the folder ingested but as a dataset (when ingesting s3, folders are categorized as containers not datasets) so same name different urn. If we want to connect athena with the original s3 urn, is the only way to do this manually? This seems like a bug, but i'm not sure if this was intended or not

Hi everyone, I wonder if people have experienced read-after-write consistency issues with DataHub? I’m using the Python emitter to emit metadata to DataHub, and right after that I’m trying to retrieve this metadata (with GraphQL) as part of a different process. However, there’s a slight delay between the write operation and it being available for consumption. I would appreciate any insights on this 🙂 Thanks

Hey, is there a way to reset the password for the root user? *Edit: As an aside, could I just delete the root user? I dont feel like its needed after creating the first admin account, but Im not sure if there is some special privilege that the root user has that an admin wont

👋 Hello! I am getting

* What went wrong:
Java heap space

when trying to build gms locally using

./gradlew :metadata-service:war:build

. Any idea how I can fix this? It happens somewhat randomly and when building frontend service as well

Hi all, We are using snowflake and dbt in datahub and I wanted to know how can I bring the column level descriptions of dbt to snowflake datasets (tables). I see the table description from dbt but not the column level. Is it configurable? Thanks

hi guys. im using Great Expectations and Datahub. Its says in the documentation that the integration will be limited if: • v2 Datasources such as SqlAlchemyDataset • v3 Datasources using execution engine other than SqlAlchemyExecutionEngine (Spark, Pandas) I was wondering if Teradata is affected by one of those limitations?

Hi Datahub Version: v0.9.5 Deployment mode: docker We are trying to use "Apache Ranger" as the authorization backend but unable to integrate it. Docs followed : https://github.com/mohdsiddique/datahub/blob/474407c6aad452898dde6a6183c35a977a25a25b/datahub-ranger-plugin/README.md, https://datahubproject.io/docs/how/configuring-authorization-with-apache-ranger/ Please help us with below doubts: 1. Do we need to set below parameter in GMS env, What other parameters are needed to be disabled/enabed

export RANGER_AUTHORIZER_ENABLED=true

2. What will be the value of "ranger.plugin.datahub.service.name" in ranger-datahub-security.xml? will it be "ranger_datahub" or "datahub" 3. Will it make any difference if the user in ranger/AD is in small case while the user in datahub is in upper case ? 4. Will this plugin be compatible with future releases of datahub ? Problem: Ranger policies are not taking effect. I have created a policy and assigned all platform privileges but still user is unable to administer datahub. As the policies are not working, is there ay other way to confirm that the plugin was loaded properly.

Hello everyone, I am trying to run a quickstart deploy with a different port for DataHub Web Application (datahub-frontend) because 9002 is already used by another software. Based on the documentation, it seems easy to do for other containers (mysql, zookeeper, kafka broker, schema registry, elasticsearch) by passing flags to the command, and for the GMS service by using an environment variable (DATAHUB_MAPPED_GMS_PORT), but I can't find a way to modify the port for the front end container. I tried this (from a Windows command line) :

set DATAHUB_MAPPED_FRONTEND_PORT=9003 && datahub docker quickstart

but i get the error : "error decoding 'Ports': Invalid hostPort". Is there any way to do it ? Thanks.

Hi everyone! I have been running into an issue. I was able to successfully run a local Front-End Datahub version on my Linux system a few days ago, but now I am back at square one 😞 I was able to successfully build several components, but one of the last steps in the local development guide is not working for me. System: NAME=“Fedora Linux” VERSION=“36 (Server Edition)” ID=fedora VERSION_ID=36 following the following guide: https://datahubproject.io/docs/developers Here is a list of what was built successfully: 1. I removed my entire datahub directory and pulled again: git clone https://github.com/jgdallas/datahub.git 2. ./gradlew --version ------------------------------------------------------------ Gradle 6.9.2 ------------------------------------------------------------ 3. ./gradlew clean BUILD SUCCESSFUL in 31s 4. ./gradlew build Many tests ran, some skipped, I did see this for the datahub-web-react: yarnInstall

> Task :datahub-web-react:yarnInstall
info fsevents@2.3.2: The platform "linux" is incompatible with this module.
info "fsevents@2.3.2" is an optional dependency and failed compatibility check. Excluding it from installation.
info fsevents@1.2.13: The platform "linux" is incompatible with this module.
info "fsevents@1.2.13" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " > @cypress/webpack-preprocessor@5.8.0" has unmet peer dependency "@babel/core@^7.0.1".
....
Removing for brevity here :-)
....

I am also uploading the final results of the build When I attempt the following I keep hitting this error: in the datahub directory…

./gradlew quickstart

FAILURE: Build failed with an exception. * What went wrong: Task ‘quickstart’ not found in root project ‘datahub’.

Hello everyone, I’m pretty new to Datahub and GraphQL I trying to query Dataset for getting specific dataset like “where lastIngested = “2023-03-20"” Now, my problem is I don’t know I can filter with lastIngested and if It can please suggest how to filter with lastIngested in this query. Any help is appreciated.

Hi, folks! I am a new with datahub environment . so I am trying to deploy with docker. keep getting loop of running. also I check the status as well I am using Mac M1 I did

datahub docker quickstart --arch m1
datahub docker quickstart --kafka-setup.  ----> because i got issue with it
docker pull acryldata/datahub-kafka-setup:head
datahub docker quickstart --version=v0.9.3

but still i can’t run I need your help. Thank you! leave your knowledge at the thread!