@early-lamp-41924 Thanks for that hint. When debugging, I found that when trying to get the next version of the aspect here The result in the database wasn't getting just the max(version) ( which should have been just 1) but instead it was getting all the versions for that aspect.

i.e. 0 and 1

select aspect, max(version) from metadata_aspect_v2 where urn='urn:li:corpuser:<redacted>' and aspect ='groupMembership' group by  urn, aspect, version

So, picking the first returned value 0 it's returning the next version as 1 again causing the primary key error. Postgres Databases are created using this script. I suspect something with our postgres setup....not clear on where the issue could be.. Let me know if you have any thoughts.

@early-lamp-41924 I added more debug logs to GMS here What i see is: • Depending on which version gets last from DB that is overwritten as the

version

in

dbResults

• In my case as the version 1 comes first then version 0 the list has

groupMembership

and version as

0

as the latest • So when incrementing for the next version it gets 1 again causing primary key error.. Here are debug logs from the environment to prove my finding:

16:01:26 [qtp1025799482-17] INFO  c.l.m.entity.ebean.EbeanAspectDao - From DB- Urn: urn:li:corpuser:<redacted>, Aspect: corpUserInfo, currVersion: 0
16:01:26 [qtp1025799482-17] INFO  c.l.m.entity.ebean.EbeanAspectDao - From DB- Urn: urn:li:corpuser:<redacted>, Aspect: groupMembership, currVersion: 1
16:01:26 [qtp1025799482-17] INFO  c.l.m.entity.ebean.EbeanAspectDao - From DB- Urn: urn:li:corpuser:<redacted>, Aspect: groupMembership, currVersion: 0
16:01:26 [qtp1025799482-17] INFO  c.l.m.entity.ebean.EbeanAspectDao - Contains ASpect - Urn: urn:li:corpuser:<redacted>, Aspect: corpUserInfo, nextVersion: 1
16:01:26 [qtp1025799482-17] INFO  c.l.m.entity.ebean.EbeanAspectDao - Added to result - Aspect: corpUserInfo, nextVersion: 1
16:01:26 [qtp1025799482-17] INFO  c.l.m.entity.ebean.EbeanAspectDao - Contains ASpect - Urn: urn:li:corpuser:<redacted>, Aspect: groupMembership, nextVersion: 1
16:01:26 [qtp1025799482-17] INFO  c.l.m.entity.ebean.EbeanAspectDao - Added to result - Aspect: groupMembership, nextVersion: 1
16:01:26 [qtp1025799482-17] INFO  c.l.m.e.ebean.EbeanEntityService - Urn - corpUserInfo, AspectName- 1, nextVersion- {}
16:01:26 [qtp1025799482-17] INFO  c.l.m.e.ebean.EbeanEntityService - Urn - groupMembership, AspectName- 1, nextVersion- {}
16:01:26 [qtp1025799482-17] INFO  c.l.m.filter.RestliLoggingFilter - POST /entities?action=ingest - ingest - 500 - 47ms
16:01:26 [qtp1025799482-17] ERROR c.l.m.filter.RestliLoggingFilter - <http://Rest.li|Rest.li> error:
com.linkedin.restli.server.RestLiServiceException: com.datahub.util.exception.RetryLimitReached: Failed to add after 3 retries

Findings: • It looks like this is a basic functionality in Datahub so i doubt if this has to do with Postgres? Any inputs on this would be helpful

Hello, I am digging into the datahub source code to contribute to the S3 Source. I want to add the option to ingest bucket and object tags as tags in Datahub. I am having some difficulty finding where i would add the tags in the file here. It seems like the objects being yielded are

MetadataChangeEvent

and

MetadataWorkUnit

however i am having difficulty finding the definition of these objects and if these are related to tags. I was wondering if anyone could give me some advice. Much appreciated in advanced! Edit: I think this example https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/dataset_add_tag.py is what i was looking for

Hi, I encountered this problem when using this interface

/aspects?action=ingestProposal

,This string should be OK

is not a valid string representation of bytes

Hi, I encountered this error when I replaced the ES container with a local ES with a username and password

Hi! My Glue (a table based on our S3 data) ingested just completed successfully on DataHub, but I cannot find the metadata on the DataHub console.

Hello again, I have the following issue: after metadata ingestion I found that partitions of partitioned postgres tables are treated as separate objects. Is there any way or setting to avoid this?

I see query being formatted and trimmed but don't see any hashing being done to the query values. So is there a possibility of PII column values being exposed thru the Queries tab?

hi Team... i want to debug ingestion local and i am run

@cold-hydrogen-10513 I have the same error, how did you solve it

{'workunits_produced': 0,
 'workunit_ids': [],
 'warnings': {},
 'failures': {'version': ['Error: (snowflake.connector.errors.DatabaseError) 250001 (08001): Failed to connect to DB: '
                          '<http://mycompay.eu-central-1.snowflakecomputing.com:443|mycompay.eu-central-1.snowflakecomputing.com:443>. Incorrect username or password was specified.\n'
                          '(Background on this error at: <http://sqlalche.me/e/13/4xp6)']>},

How can I ingest Airflow? I’m now using helm chart of Datahub.

Now that i can ingest the superset install, i wanted to check if i can see the "changes" in the lineage if i change the dataset source. I mean are we notified when a dataset is "changed" and can we see the impact in the lineage graph or somewhere else ?

if I have a bucket look like this: bucketName: • table1 ◦ parquet1 ◦ parquet2 ◦ parquet3 ◦ . ◦ . ◦ . • table2 ◦ parquet1 ◦ parquet2 ◦ parquet3 ◦ . ◦ . ◦ . • table3 ◦ parquet1 ◦ parquet2 ◦ parquet3 ◦ . ◦ . ◦ . • . • . • . and i want to ingest all of this tables there us a recipe for it?

Hello, I am trying to ingest a glue catalog from an AWS account that is different to the AWS account that datahub is hosted on. The doco says that this can be done using the catalog_id config https://datahubproject.io/docs/metadata-ingestion/source_docs/glue/ My recipe is of this form

I am trying to put only lineage information in an existing mssql table using file-based-lineage. A new dataset is being created. Is there any way to add only lineage information?

Hello,

Hello All,

"source": { "type": "mysql", "config": { "username": "usrname", "password": "pass", "database": "test_datahub", "host_port": "test.mysql.database.azure.com:3306", }, }, "sink": { "type": "datahub-rest", "config": {"server": "http://localhost:8080"}, }, }

@big-carpet-38439 It looks like from what i can tell from the debug logs

[2022-05-03 16:53:06,994] DEBUG    {datahub.cli.ingest_cli:94} - Using config: {'source': {'type': 'datahub-stream', 'config': {'auto_offset_reset': 'latest', 'connection': {'bootstrap': '<redacted>', 'schema_registry_url': '<redacted>', 'consumer_config': {'security.protocol': 'SASL_SSL'}}, 'actions': [{'type': 'executor', 'config': {'local_executor_enabled': True, 'remote_executor_enabled': 'False', 'remote_executor_type': 'acryl.executor.sqs.producer.sqs_producer.SqsRemoteExecutor', 'remote_executor_config': {'id': 'remote', 'aws_access_key_id': '""', 'aws_secret_access_key': '""', 'aws_session_token': '""', 'aws_command_queue_url': '""', 'aws_region': 'us-east-1'}}}], 'topic_routes': {'mae': 'MetadataAuditEvent_v4', 'mcl': 'MetadataChangeLog_Versioned_v1'}}}, 'sink': {'type': 'console'}, 'datahub_api': {'server': '<redacted>:8080', 'extra_headers': {'Authorization': 'Basic __datahub_system:JohnSnowKnowsNothing'}}}

sasl.mechanism

that is passed is not set in the consumer configs....

Hi everyone, I'm trying to ingest data from an existing Elasticsearch cluster. My recipe:

source:
  type: "elasticsearch"
  config:
    # Coordinates
    host: '<https://internal_ip:9200>'

    # Credentials
    username: the_user
    password: the_pass

sink:
    type: "console"

because I am not passing any SSL config params, I am getting:

elasticsearch.exceptions.SSLError: ConnectionError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131))

I dug into the source code a bit, and I am seeing that the connection to elasticsearch is set by passing only the host and the http_auth (basically username and pass):

Hi, The following error occurred when I ingested the CK data source. The field is missing. Is it a bug or the version of CK

image.png

Hello How we define custom run ids in yml while doing ingestion using cli.

Hi All, I'm having trouble for parsing manifest.json for ingest data dbt.

'JSONDecodeError: Invalid control character at: line 1594 column 4096 (char 65977)\n'

However when I'm running in local I'm having in the same error, but I modified in the local with this solution it works with assigned content of json file into variable https://stackoverflow.com/questions/9156417/valid-json-giving-jsondecodeerror-expecting-delimiter https://stackoverflow.com/questions/63107394/jsonload-jsondecodeerror-invalid-control-character-at But if I tried to execute in datahub UI it always error. Does anyone know which file I can edit to add

strict=False

in json.loads?

Hello, for Redash ingestion, which all Redash versions are supported? https://datahubproject.io/docs/generated/ingestion/sources/redash/

Hey team, a question about partition. As I see from the GraphQL Reference,

partition

sits in

dataset.datasetProfiles.partitionSpec

.Does this mean we can only surface

partition

as part of

datasetProfiles

? IMHO, shouldn’t dataset partition exists by itself, not depending on the dataset profiles?

Hello, I have a Great Expectations script pushing validations to datahub. Everything works as expected except the script refuses to terminate even with a sys.exit() at the end of the script. I have to force it to terminate with CTRL+C. Any idea why this is happening?

Hi All

I am trying to do CLI ingestion from mysql to data hub rest.