Hello, I'm testing the editing and glossary creation functionalities in the UI and I got the behavior below when trying to add a documentation. Would this be a bug? Datahub version: v0.8.38 Deployment type: Without neo4j and on a GKE cluster.

If I want to install a new plugin for sources in an existing datahub instance, which docker container should I install into?

Hi All 👋. Does anyone know what it means to get a 404 on

aspects?action=ingestProposal

when trying to delete a dataset using the CLI?

Hello everyone, Since yesterday, for some reason I still don't understand, all ingested data stopped showing up on the frontend http://localhost:9002/ I'm deploying the DataHub locally with quickstart. Yesterday morning all ingestions were working perfectly. And since I'm still developing the DataHub ingestion pipelines, I restart (with docker nuke and docker quickstart) the container frequently to test if my scripts work. But all of a sudden the ingestions stopped showing up on the frontend. In the terminal, there is no error. All ingestions end successfully. I checked the DataHub database and all data is correctly stored. I figured it could be the Elasticsearch indices that for some reason weren't being updated by the page, so I launched

./docker/datahub-upgrade/datahub-upgrade.sh -u RestoreIndices

to reset the indices... But nothing worked. I tried ingesting the data directly into the UI and nothing happens. Not even the card with the name of the ingestion appears. Which makes me imagine that there is some communication problem between the interface and the bank. I asked my colleague to git pull my script and run it on his computer, and it worked. Can anybody help me ? I do not know what else to do. Thanks a lot!

Hi, have a quick question. For the metadata ingestion for snowflake, should the user have any specific role. I have full access on the schema but it is throwing login error, which is working absolutely fine when used with other applications

Hi, I am facing this issue for some of my dags https://github.com/datahub-project/datahub/issues/4546 . Is there any workaround for this?

@big-carpet-38439 need help in understanding datahub actions code what I am able to figure out is that when an ingestion is ran it executes and pipeline in datahub actions that triggers executor_actions which gets the config and run the act method I am unable to get two things 1. how the below code is able to get the task config

# Build default task config
        local_task_config = TaskConfig(
            name="RUN_INGEST",
            type="acryl.executor.execution.sub_process_ingestion_task.SubProcessIngestionTask",
            configs=dict({}),
        )

2. How act method calls the library in metadata-ingestion module ? please let me know in case my understanding is wrong or I am missing something

Hey all! We are trying to determine who exactly our WAU's are. Is this info currently stored anywhere? Where are most of the stats in the default analytics panel stored? Thanks!

Hi Everyone, Trying to use https://datahubproject.io/docs/metadata-ingestion/transformers/#mark-dataset-status But don't see the change reflect (the dataset is still visible in the UI)

{
  "source": {
    "type": "glue",
    "config": {
      "env": "PROD",
      "aws_region": "us-east-1",
      "extract_transforms": "false",
      "table_pattern": {
        "allow": [
          "testdb.*$"
        ],
        "ignoreCase": "false"
      }
    }
  },
  "transformers": [
    {
      "type": "mark_dataset_status",
      "config": {
        "removed": "true"
      }
    }
  ]
}

Hi all, Please help me to get all dataset base on browspath startwith using GraphiQL. Here is my GraphiQL

{
  search(
    input: {type: DATASET, query: "*", start: 0, count: 10000, 
      filters: [{field: "browsePaths", value: "/prod/myDept/sakilaDb"} ]}
  ) {
    start
    count
    total
    searchResults {
      entity {
        urn
        type
        ... on Dataset {
          name
        }
      }
    }
  }
}

Hi, I got this error while ingesting from my database.

" 'warnings': {'cardcore_issuing_2.view_cardcore_card_master': ['Ingestion error: Table or view named '\n"
           "                                                               '`cardcore_issuing_2`.`view_cardcore_card_master` could not be reflected: "
           "'\n"
           '                                                               \'(pymysql.err.OperationalError) (1356, "View \'\n'
           '                                                               "\'cardcore_issuing_2.view_cardcore_card_master\' references invalid '
           'table(s) or "\n'
           "                                                               'column(s) or function(s) or definer/invoker of view lack rights to use "
           'them")\\n\'\n'
           "                                                               '[SQL: DESCRIBE `cardcore_issuing_2`.`view_cardcore_card_master`]\\n'\n"
           "                                                               '(Background on this error at: <http://sqlalche.me/e/13/e3q8)>'],\n"

May I know what is the cause of this error?

Hello everyone, I installed datahub cli and ingested data from big query, i. want to extract the tags available in a dataset. For doing so, I added a tag on dataset from the UI. and used below api to fetch the tags and other aspects of the dataset but it shows

error: [HTTP Status:404]: No root resource defined for path '/prod'

whereas when I try to search the tag , it gets correct result. urn that I am using is this : urnlidataset:(urnlidataPlatform:bigquery,temp,PROD) can anyone help me, point out what am I missing here, seems like i need to set some environment variable to point it to correct directory but i am not sure which directory it should point to.

DOCKER_BUILDKIT=1 docker build -t your_datahub_frontend -f ./docker/datahub-frontend/Dockerfile

#16 160.3 * What went wrong: #16 160.3 A problem occurred configuring project ':datahub-frontend'. #16 160.3 > Could not resolve all artifacts for configuration 'datahub frontendclasspath'. #16 160.3 > Could not resolve io.github.kobylynskyigraphql java codegen gradle plugin4.1.1. #16 160.3 Required by: #16 160.3 project :datahub-frontend > io.github.kobylynskyi.graphql.codegenio.github.kobylynskyi.graphql.codegen.gradle.plugin4.1.1 #16 160.3 > Could not resolve io.github.kobylynskyigraphql java codegen gradle plugin4.1.1. #16 160.3 > Could not get resource 'https://plugins.gradle.org/m2/io/github/kobylynskyi/graphql-java-codegen-gradle-plugin/4.1.1/graphql-java-codegen-gradle-plugin-4.1.1.pom'. #16 160.3 > Could not GET 'https://plugins-artifacts.gradle.org/io.github.kobylynskyi/graphql-java-codegen-gradle-plugin/4.1.1/224dad42cfe8ff38cfdb42d6e4418da2a231b55cdb1b232e97b0e33c697dbad1/graphql-java-codegen-gradle-plugin-4.1.1.pom'. #16 160.3 > plugins-artifacts.gradle.org: Try again #16 160.3 #16 160.3 * Try: #16 160.3 Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. #16 160.3 #16 160.3 * Get more help at https://help.gradle.org #16 160.3 #16 160.3 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. #16 160.3 Use '--warning-mode all' to show the individual deprecation warnings. #16 160.3 See https://docs.gradle.org/6.9.2/userguide/command_line_interface.html#sec:command_line_warnings #16 160.3 #16 160.3 BUILD FAILED in 2m 39s

Hello I’m trying to ingest lookml using sample projects that looker provides For example, I want to ingest

welcome_to_looker

that is provided for demo purposes I’m getting error:

1 validation error for LookMLSourceConfig
base_folder
  file or directory at path "/projects/welcome_to_looker" does not exist (type=value_error.path.not_exists; path=/projects/welcome_to_looker)

My yaml config:

base_folder: /projects/welcome_to_looker

Does that somebody have some tips to specify path correctly?

Hi all I am trying to use the Generate documentation for metadata feature Given the

./gradelw

command is relative to the current directory, from which directory should I be issue this command?

I just nuked my local installation and did a new deploy via datahub docker quickstart (v 0.8.39). I can ingest data

Sink (datahub-rest) report:
{'records_written': 147,
 'warnings': [],
 'failures': [],
 'downstream_start_time': datetime.datetime(2022, 6, 27, 13, 54, 33, 123014),
 'downstream_end_time': datetime.datetime(2022, 6, 27, 13, 54, 44, 555620),
 'downstream_total_latency_in_seconds': 11.432606,
 'gms_version': 'v0.8.39'}

but it's not showing on the UI... the compose logs are showing:

elasticsearch             | {"type": "server", "timestamp": "2022-06-27T15:24:43,298Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "flood stage disk watermark [95%] exceeded on [NT1i9TTQSwW42zsgX84oAw][elasticsearch][/usr/share/elasticsearch/data/nodes/0] free: 2.2gb[2.4%], all indices on this node will be marked read-only", "cluster.uuid": "sQMENsZFR7CcBJCms4Qydg", "node.id": "NT1i9TTQSwW42zsgX84oAw"  }

is that some issue in the quickstart config?

Hello everyone! I am trying to start using analytics page in our locally running Datahub instance (v0.8.39) The first half of the page is pretty empty, but that may be because we haven't been using it that long. However the second half of the analytics page is suspiciously empty, even more than I expected.

hello, I am having this issue on datahub v0.8.36. I can see in the main screen that there are 8 glossary terms, but when I click, I can't see any. And the number of glossary terms is not changing when I am adding a new glossary term via the UI. Any ideas?

Hello How do we authenticate when using locust. I have tried specifying the headers but gms logs still show request is missing authorization headers.

I have an error with datahub-mae-consumer in v0.8.39 using helm chart (datahub-0.2.84).

[main] ERROR o.s.b.d.LoggingFailureAnalysisReporter - 

***************************
APPLICATION FAILED TO START
***************************

Description:

Parameter 1 of constructor in com.linkedin.metadata.kafka.hook.siblings.SiblingAssociationHook required a bean named ‘entityAspectDao’ that could not be found.

The injection point has the following annotations:
    - @javax.anotation.Nonnull(when =ALWAYS)

Action:

Consider defining a bean named ‘entityAspectDao’ in your configuration.

I think the following PR is related. https://github.com/datahub-project/datahub/pull/5190

This message was deleted.

Hi, I am trying to deploy Datahub 0.8.39 with ElasticSearch. Although I have specified

GRAPH_SERVICE_IMPL=elasticsearch

, I think it doesn't work correctly as in the following error log some lines containing Neo4j can be seen. That is not the only problem, but I have no clue what the rest of the errors' cause. Could someone help me narrow it down??

Hi, is it normal that the upstream in lineage doesn't link to the dataset. Or is it an error in my configuration?

Hello everyone! I'm getting the following error at the moment to execute an ingestion from Redsfhit:

acryl.executor.execution.task.TaskError: Failed to resolve secret with name REDSHIFT_HOST_PORT. Aborting recipe execution

. The datasource was created a month ago and it was working good, but since I upgraded my datahub instance to the 0.3.38 version the data ingestion has been broken. I could't find a solution to this exception, I can't read any secret from that ingestion. Is there any consideration that I missed at the moment of the version upgrade? Hope someone can help me! Thanks!

im finding that sometimes, upon ingesting a new version of an aspect, the original version of the aspect will disappear from mySQL. But if i were to follow up and emit a slightly different value again, the newly inserted aspect does get incremented correctly. What are some possible areas to check? These aspects are not using ingestion sources, but generated mcps from the python library. I did not set a retention policy yet, and there are no errror messages in the frontend/gms logs. I also didnt specify a version# in the MCP.

Hello, I have trouble running the docker images on mac with M1 Chip. It seems that you don"t build the docker images for this architecture. I'm using this for local tests and demo purposes internally. Has someone gotten the docker images to run on this architecture?

How do I run the

ClearSearchServiceStep

as part of the R`estoreIndices` upgrade command?

hello. I am trying to use datahub in Korea. I am currently using v0.8.32 installed. When I try to input Korean in urn or description, etc., an error occurs. I tried to install v0.8.36 or v0.8.39, but I couldn't use it with a red error window saying "Oops, an error occurred". I think I need utf8 encoding to input Korean. What should I do?

Hi Team, My build is failing at ./gradlew metadata ingestioninstallDev can someone help on it. My system is macbook Intel Core i7.