limited-library-89060
03/31/2023, 10:15 AMfailing. Any idea how to show all the test results if all the tests are passed ?limited-library-89060
03/31/2023, 10:15 AMimport logging
from datahub.emitter.mcp import MetadataChangeProposalWrapper
from datahub.ingestion.graph.client import DatahubClientConfig, DataHubGraph
from datahub.metadata.schema_classes import (
TestResultsClass,
TestResultClass,
TestResultTypeClass,
TestInfoClass,
TestDefinitionClass,
TestDefinitionTypeClass
)
log = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)
gms_endpoint = "http://host..."
graph = DataHubGraph(
config=DatahubClientConfig(
server=gms_endpoint
)
)
test_properties = TestInfoClass(
category="lineage",
description="Simple test description 2",
name="lineage-test",
definition=TestDefinitionClass(
type=TestDefinitionTypeClass.JSON,
json="..."
)
)
event: MetadataChangeProposalWrapper = MetadataChangeProposalWrapper(
entityUrn="urn:li:test:lineage-test-2",
aspect=test_properties,
)
graph.emit(event)
test_result_properties = TestResultClass(
type=TestResultTypeClass.SUCCESS,
test="urn:li:test:lineage-test-2"
)
test_results = TestResultsClass(
failing=[],
passing=[test_result_properties]
)
event: MetadataChangeProposalWrapper = MetadataChangeProposalWrapper(
entityUrn="urn:li:dataset:(urn:li:dataPlatform:bigquery,xxx,PROD)",
aspect=test_results,
)
graph.emit(event)limited-library-89060
03/31/2023, 10:16 AMtest_results to
test_results = TestResultsClass(
passing=[test_result_properties],
failing=[test_result_properties]
)
, then it will give me thislimited-library-89060
03/31/2023, 10:17 AMtest_results = TestResultsClass(
passing=[],
failing=[test_result_properties]
)
then it will give meastonishing-answer-96712
04/03/2023, 9:21 PMlimited-library-89060
04/04/2023, 12:41 AMingestion state (local python logs)
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): dummy_host:8082
DEBUG:urllib3.connectionpool:http://dummy_host:8082 "GET /config HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:http://dummy_host:8082 "GET /aspects/urn%3Ali%3Atelemetry%3AclientId?aspect=telemetryClientId&version=0 HTTP/1.1" 404 8642
DEBUG:datahub.emitter.rest_emitter:Attempting to emit to DataHub GMS; using curl equivalent to:
curl -X POST -H 'User-Agent: python-requests/2.28.1' -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'Connection: keep-alive' -H 'X-RestLi-Protocol-Version: 2.0.0' -H 'Content-Type: application/json' --data '{"proposal": {"entityType": "test", "entityUrn": "urn:li:test:lineage-test-2", "changeType": "UPSERT", "aspectName": "testInfo", "aspect": {"value": "{\"name\": \"lineage-test\", \"category\": \"lineage\", \"description\": \"Simple test description 2\", \"definition\": {\"type\": \"JSON\", \"json\": \"\"}}", "contentType": "application/json"}}}' 'http://dummy_host:8082/aspects?action=ingestProposal'
DEBUG:urllib3.connectionpool:http://dummy_host:8082 "POST /aspects?action=ingestProposal HTTP/1.1" 200 38
DEBUG:datahub.emitter.rest_emitter:Attempting to emit to DataHub GMS; using curl equivalent to:
curl -X POST -H 'User-Agent: python-requests/2.28.1' -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'Connection: keep-alive' -H 'X-RestLi-Protocol-Version: 2.0.0' -H 'Content-Type: application/json' --data '{"proposal": {"entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:postgres,dummy_dataset,PROD)", "changeType": "UPSERT", "aspectName": "testResults", "aspect": {"value": "{\"failing\": [], \"passing\": [{\"test\": \"urn:li:test:lineage-test-2\", \"type\": \"SUCCESS\"}]}", "contentType": "application/json"}}}' 'http://dummy_host:8082/aspects?action=ingestProposal'
DEBUG:urllib3.connectionpool:http://dummy_host:8082 "POST /aspects?action=ingestProposal HTTP/1.1" 200 93limited-library-89060
04/04/2023, 12:45 AM00:42:24.706 [qtp522764626-38929] INFO c.l.m.r.entity.AspectResource:143 - INGEST PROPOSAL proposal: {aspectName=testResults, entityUrn=urn:li:dataset:(urn:li:dataPlatform:postgres,dummy_dataset,PROD), entityType=dataset, aspect={contentType=application/json, value=ByteString(length=87,bytes=7b226661...227d5d7d)}, changeType=UPSERT}
00:42:24.710 [pool-11-thread-1] INFO c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 4ms
00:43:18.712 [pool-11-thread-1] INFO c.l.m.filter.RestliLoggingFilter:55 - GET /entitiesV2?ids=List(urn%3Ali%3Acorpuser%3Adatahub) - batchGet - 200 - 2ms
00:43:18.715 [pool-11-thread-1] INFO c.l.m.filter.RestliLoggingFilter:55 - GET /entitiesV2?ids=List(urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Apostgres%2Cdummy_dataset%2CPROD%29) - batchGet - 200 - 2ms
00:43:18.717 [pool-11-thread-1] INFO c.l.m.filter.RestliLoggingFilter:55 - GET /entitiesV2?ids=List(urn%3Ali%3Acorpuser%3Adatahub) - batchGet - 200 - 1ms
00:43:19.051 [qtp522764626-36516] INFO c.l.m.resources.usage.UsageStats:291 - Attempting to query usage stats
00:43:19.058 [pool-11-thread-1] INFO c.l.m.filter.RestliLoggingFilter:55 - POST /usageStats?action=queryRange - queryRange - 200 - 7ms
00:43:19.362 [I/O dispatcher 1] INFO c.l.m.s.e.update.BulkListener:47 - Successfully fed bulk request. Number of events: 2 Took time ms: -1limited-library-89060
04/04/2023, 12:53 AMAre you able to query the data in GMS via the GraphQL api?Yes, I am able to access the dataset details (including
TestResults ) using graphql. I am using the default graphiqllimited-library-89060
04/04/2023, 6:57 AMaloof-gpu-11378
04/04/2023, 7:00 AMlimited-library-89060
04/04/2023, 7:01 AMquery getDetail{
dataset(urn: "urn:li:dataset:(urn:li:dataPlatform:postgres,dummy_dataset,PROD)") {
urn
testResults{
passing{
test{
urn
type
name
category
description
definition{
json
}
}
}
failing{
test{
urn
type
name
category
description
definition{
json
}
}
}
}
}
}limited-library-89060
04/04/2023, 7:03 AMlimited-library-89060
04/04/2023, 7:04 AMhundreds-photographer-13496
04/04/2023, 7:06 AMlimited-library-89060
04/04/2023, 7:17 AM0.9.5hundreds-photographer-13496
04/04/2023, 9:35 AMfilteredFailing.length > 0 needs to be updated by totalTests > 0
Would you be able to create a PR for this ?limited-library-89060
04/04/2023, 9:36 AMaloof-gpu-11378
04/04/2023, 9:39 AMlimited-library-89060
04/04/2023, 9:53 AMhundreds-photographer-13496
04/04/2023, 10:09 AMechoing-airport-49548
04/05/2023, 12:57 AMechoing-airport-49548
04/05/2023, 12:58 AMechoing-airport-49548
04/05/2023, 12:58 AM