blue-beach-27940
07/01/2022, 2:26 AMprofiling
in the ingestion action, I want to get the column statistics? I got some error with blow recipe.ymlblue-beach-27940
07/01/2022, 2:27 AMbrave-tomato-16287
07/01/2022, 11:15 AMproud-baker-56489
07/01/2022, 11:58 AMgray-river-37120
07/02/2022, 1:36 AM403
error. Here’s the recipe:
``source:`
type: snowflake
config:
check_role_grants: true
provision_role:
enabled: false
dry_run: true
run_ingestion: false
admin_username: '${SNOWFLAKE_USER}'
admin_password: '${SNOWFLAKE_PASS}'
account_id: ak43980
warehouse: COMPUTE_WH
username: '${SNOWFLAKE_USER}'
password: '${SNOWFLAKE_PASS}'
role: ROLENAME
`ignore_start_time_lineage: true``
And, from the logs:
'[2022-07-02 00:59:43,674] INFO {datahub.cli.ingest_cli:99} - DataHub CLI version: 0.8.40\n'
'[2022-07-02 00:59:44,598] INFO {datahub.ingestion.source_config.sql.snowflake:236} - using authenticator type '
"'DEFAULT_AUTHENTICATOR'\n"
'[2022-07-02 00:59:44,742] INFO {datahub.cli.ingest_cli:115} - Starting metadata ingestion\n'
'[2022-07-02 00:59:44,743] INFO {datahub.ingestion.source.sql.snowflake:114} - Checking current version\n'
'[2022-07-02 01:02:06,031] ERROR {snowflake.connector.network:920} - 000403: HTTP 403: Forbidden\n'
proud-baker-56489
07/06/2022, 3:42 AMtall-butcher-30509
07/07/2022, 7:55 AMSample Code:
------------
MetadataChangeProposalWrapper mcpw = MetadataChangeProposalWrapper.builder()
.entityType("dataset")
.entityUrn("urn:li:dataset:(urn:li:dataPlatform:bigquery,<REMOVED>,TEST)")
.upsert()
.aspect(new DatasetProperties().setDescription("Sample Data - 商品ブランドコード"))
.aspectName("datasetProperties")
.build();
emitAspectsToDataHub(mcpw);
Error Log:
------------
Ingestion failed: EMIT_METADATA_ERROR_RESPONSE : Failed to emit entity type: dataset, entity urn: urn:li:dataset:(urn:li:dataPlatform:bigquery,<REMOVED>,TEST), aspect: datasetProperties with status code: 400 retry started...
lemon-zoo-63387
07/11/2022, 1:32 AMtall-butcher-30509
07/11/2022, 11:36 PMbumpy-camera-96689
07/14/2022, 6:56 AMsparse-barista-40860
07/18/2022, 6:16 PMcool-vr-73109
07/21/2022, 10:22 AMchilly-carpet-99599
07/21/2022, 7:23 PMcool-vr-73109
07/26/2022, 9:29 AMcool-vr-73109
07/28/2022, 8:26 AMgifted-kite-59905
08/01/2022, 6:24 AMdazzling-insurance-83303
08/05/2022, 7:31 PMallow_deny_patterns
for profiling.
A quick confirmation would really help. Thanks!
CC @little-megabyte-1074, @mammoth-bear-12532gifted-knife-16120
08/08/2022, 9:31 AMrapid-house-76230
08/12/2022, 5:29 PMfamous-florist-7218
08/17/2022, 7:24 AM<s3://bucket/foo/bar/folder/table_name/year=2022/month=08/day=04/hour=09/file1.json.gz>
<s3://bucket/foo/bar/folder/table_name/year=2022/month=08/day=04/hour=09/file2.json.gz>
<s3://bucket/foo/bar/folder/table_name/year=2022/month=08/day=04/hour=09/file3.json.gz>
<s3://bucket/foo/bar/folder/table_name/year=2022/month=08/day=04/hour=09/file4.json.gz>
<s3://bucket/foo/bar/folder/table_name/year=2022/month=08/day=04/hour=09/file5.json.gz>
<s3://bucket/foo/bar/folder/table_name/year=2022/month=08/day=04/hour=09/file6.json.gz>
<s3://bucket/foo/bar/folder/table_name/year=2022/month=08/day=04/hour=09/file7.json.gz>
...
nutritious-bird-77396
08/17/2022, 4:11 PMsparse-forest-98608
08/18/2022, 7:32 AMsparse-forest-98608
08/18/2022, 7:34 AMcurved-magazine-23582
08/19/2022, 6:57 PMapplication
type permission of Tenant.Read.All, But I am still getting 401 error during ingestion, any suggestion on what to look next?bright-motherboard-35257
08/20/2022, 10:01 PMsink:
type: datahub-rest
config:
server: 'http://<redacted>/api/gms'
source:
type: s3
config:
profiling:
enabled: true
path_specs:
-
include: 'https://<redacted>.<http://s3.amazonaws.com/branch-data/*.*|s3.amazonaws.com/branch-data/*.*>'
env: PROD
aws_config:
aws_access_key_id: <redacted>
aws_region: us-east-1
aws_secret_access_key: <redacted>
pipeline_name: 'urn:li:dataHubIngestionSource:<redacted>'
gifted-bird-57147
08/22/2022, 6:15 AMhappy-twilight-44865
08/25/2022, 1:03 PMgreat-account-95406
08/29/2022, 5:47 AMbrave-nail-85388
08/30/2022, 8:18 PMbrave-nail-85388
08/30/2022, 8:41 PM