hi all, how to use

profiling

in the ingestion action, I want to get the column statistics? I got some error with blow recipe.yml

image.png

Hello all. Could you help to understand what happened with DBT tests? run_results.json contains several records with success tests.

image.png

Sorry to unbury old threads, but wondering @numerous-holiday-52504 if you ever got this solved? I’m having the same issues. No matter how I specify the options for the Snowflake connection, I get a

403

error. Here’s the recipe: ``source:`

type: snowflake

config:

check_role_grants: true

provision_role:

enabled: false

dry_run: true

run_ingestion: false

admin_username: '${SNOWFLAKE_USER}'

admin_password: '${SNOWFLAKE_PASS}'

account_id: ak43980

warehouse: COMPUTE_WH

username: '${SNOWFLAKE_USER}'

password: '${SNOWFLAKE_PASS}'

role: ROLENAME

`ignore_start_time_lineage: true`` And, from the logs:

'[2022-07-02 00:59:43,674] INFO     {datahub.cli.ingest_cli:99} - DataHub CLI version: 0.8.40\n'
           '[2022-07-02 00:59:44,598] INFO     {datahub.ingestion.source_config.sql.snowflake:236} - using authenticator type '
           "'DEFAULT_AUTHENTICATOR'\n"
           '[2022-07-02 00:59:44,742] INFO     {datahub.cli.ingest_cli:115} - Starting metadata ingestion\n'
           '[2022-07-02 00:59:44,743] INFO     {datahub.ingestion.source.sql.snowflake:114} - Checking current version\n'
           '[2022-07-02 01:02:06,031] ERROR    {snowflake.connector.network:920} - 000403: HTTP 403: Forbidden\n'

hi, I try to ingest some hive tables into datahub using the datahub cli, but I get the erro that tells me the queue is full, how can I change it ？

Regarding encoding, within our Java application, we are taking data from BigQuery, which returns it in UTF-8, using the MetadataChangeProposalWrapper to make a request and then send it with RestEmitter. We don’t make any particular configuration of these classes regarding encoding and are currently assuming that they use UTF-8 by default. The error only occurs when we change a property value from ascii only to include Japanese characters (business requirement).

Sample Code:
------------
MetadataChangeProposalWrapper mcpw = MetadataChangeProposalWrapper.builder()
                .entityType("dataset")
                .entityUrn("urn:li:dataset:(urn:li:dataPlatform:bigquery,<REMOVED>,TEST)")
                .upsert()
                .aspect(new DatasetProperties().setDescription("Sample Data - 商品ブランドコード"))
                .aspectName("datasetProperties")
                .build();
emitAspectsToDataHub(mcpw);


Error Log:
------------

 Ingestion failed: EMIT_METADATA_ERROR_RESPONSE : Failed to emit entity type: dataset, entity urn: urn:li:dataset:(urn:li:dataPlatform:bigquery,<REMOVED>,TEST), aspect: datasetProperties with status code: 400 retry started...

Hello everyone, I have a very strange problem. Can only 10000 entities be deleted at a time.

FYI GitHub issue was raised https://github.com/datahub-project/datahub/issues/5367

Hi! Can anyone help me understand which ingestion sources support population of the "Features" tab, for an MLModel entity? In the screenshot below, I created this model programatically, for testing. But I'm interested in any sources which would link models and features together. Thanks!

SNAG-0058.png

IMG_20220721_155142.jpg

Thanks, Swaroop. Will give that a shot. @creamy-van-28626 , FYI.

IMG_20220726_145827.jpg

IMG_20220728_135220.jpg,IMG_20220728_135101.jpg

Will the new Bulk Edits via the UI help to solve this. Or is it at table level only @orange-night-91387?

So at this time I am not sure if it is YAML specification error or a bug in the

allow_deny_patterns

for profiling. A quick confirmation would really help. Thanks! CC @little-megabyte-1074, @mammoth-bear-12532

hi anyone here can help?

Hi team, I would appreciate some help on this if you have a chance

hi @dazzling-judge-80093, I have a question that my s3 bucket has a thousand files. If I ingest all in the file level, and build lineage graph between them. It will be very messy. So how do I combine them in the lineage graph? I know that if I specify {table} placeholder, all these files will be counted as a dataset with actual table_name. But I need to trace the log of each file. Do you have any potential approaching? Thanks 🙂 For example:

<s3://bucket/foo/bar/folder/table_name/year=2022/month=08/day=04/hour=09/file1.json.gz>
<s3://bucket/foo/bar/folder/table_name/year=2022/month=08/day=04/hour=09/file2.json.gz>
<s3://bucket/foo/bar/folder/table_name/year=2022/month=08/day=04/hour=09/file3.json.gz>
<s3://bucket/foo/bar/folder/table_name/year=2022/month=08/day=04/hour=09/file4.json.gz>
<s3://bucket/foo/bar/folder/table_name/year=2022/month=08/day=04/hour=09/file5.json.gz>
<s3://bucket/foo/bar/folder/table_name/year=2022/month=08/day=04/hour=09/file6.json.gz>
<s3://bucket/foo/bar/folder/table_name/year=2022/month=08/day=04/hour=09/file7.json.gz>
...

@big-carpet-38439 I have been trying to get off the custom image for datahub frontend and datahub-actions but couldn't get an example to mount a custom file...would you be able to share an example of the same... We use kustomize but i can try based on helm examples as well. Thanks!

I have some metadata got from csv files, I want to ingest that to datahub

image.png

hello, coming back to this PowerBI ingestion, I have registered an azure app, and given it

application

type permission of Tenant.Read.All, But I am still getting 401 error during ingestion, any suggestion on what to look next?

I'm using the docker quick start container. Trying to perform an S3 ingest. I installed pyspark, openjdk-8 java and set env variable SPARK_VERSION=3.2.0 on the datahub-actions image to get past the ingest error "Please set env variable SPARK_VERSION". The ingest runs without error yet no information is retrieved from my S3 bucket. Recipe:

sink:

type: datahub-rest

config:

server: 'http://<redacted>/api/gms'

source:

type: s3

config:

profiling:

enabled: true

path_specs:

-

include: 'https://<redacted>.<http://s3.amazonaws.com/branch-data/*.*|s3.amazonaws.com/branch-data/*.*>'

env: PROD

aws_config:

aws_access_key_id: <redacted>

aws_region: us-east-1

aws_secret_access_key: <redacted>

pipeline_name: 'urn:li:dataHubIngestionSource:<redacted>'

As per documentation (https://datahubproject.io/docs/metadata-ingestion#recipes) the yaml plugin for vscode does syntax validation and type checking. The ingestion itself works, but the validation/auto completion is a nice feature to help you author the recipes...

I am looking for same connection

Could anyone give me advice here, please?

ingestion error.PNG,ingestionerror1.PNG

image.png