gifted-queen-80042
03/03/2022, 7:43 PMprofiling.enabled: True
in my recipe. I can see aspect names datasetProfile
in the output, along with the values in the respective aspects being produced, in terms of rowCount
, columnCount
, fieldProfiles
, etc. However, when I ingest onto the UI, with sink type set to datahub-kafka
, I don't see the data. The Stats
tab remains disabled. Any idea why this might be happening?hundreds-photographer-13496
03/04/2022, 12:51 PMgifted-queen-80042
03/04/2022, 3:34 PMsource:
type: "mysql"
config:
host_port: "<host port>"
database: "<database name>"
username: "<username>"
password: "<password>"
schema_pattern:
allow:
- "<regex pattern allowed>"
table_pattern:
allow:
- "<regex pattern allowed>"
profiling:
enabled: True
# profile_table_level_only: true
query_combiner_enabled: False
turn_off_expensive_profiling_metrics: True
max_number_of_fields_to_profile: 10
profile_pattern:
allow:
- "<regex pattern allowed>"
sink:
type: "datahub-kafka"
config:
connection:
bootstrap: "<some url>"
schema_registry_url: "<some url>"
topic_routes:
MetadataChangeEvent: <topic route>
MetadataChangeProposal: <topic route>
mammoth-bear-12532
topic_routes
?early-lamp-41924
03/05/2022, 12:40 AMgifted-queen-80042
03/07/2022, 2:48 PMtopic_routes:
MetadataChangeEvent: ...
MetadataChangeProposal: ...
gifted-queen-80042
03/07/2022, 4:24 PMprofiling.query_combiner_enabled: False
. Otherwise, it shows issues.
Whats strange is I can see datasetProfile aspects when ingested locally on to a file but I don't see the Stats tab being enabled for those datasets when ingesting onto the Datahub UI.gifted-queen-80042
03/07/2022, 5:17 PM0.8.27
and the one we have hosted is version v0.8.14
Also, here are some snapshots from the gms logs :
16:47:10.102 [qtp1933493643-23495] INFO c.l.m.r.entity.AspectResource:126 - INGEST PROPOSAL proposal: {aspectName=container, systemMetadata={lastObserved=...., runId=mysql-....}, entityUrn=urn:li:dataset:(urn:li:dataPlatform:mysql,...,PROD), entityType=dataset, aspect={contentType=application/json, value=ByteString(length=66,bytes=7b22636f...6532227d)}, changeType=UPSERT}
16:47:10.116 [qtp1933493643-23495] INFO c.l.m.filter.RestliLoggingFilter:56 - POST /aspects?action=ingestProposal - ingestProposal - 500 - 14ms
16:47:10.117 [qtp1933493643-23495] ERROR c.l.m.filter.RestliLoggingFilter:38 - java.lang.RuntimeException: Unknown aspect container for entity dataset
16:47:10.118 [pool-11-thread-1] ERROR c.l.common.callback.CallbackAdapter:90 - Failed to convert callback error, original exception follows:
com.linkedin.r2.message.rest.RestException: Received error 500 from server for URI <http://localhost:8080/aspects>
at com.linkedin.r2.transport.http.common.HttpBridge$1.onResponse(HttpBridge.java:76)
at com.linkedin.r2.transport.http.client.rest.ExecutionCallback.lambda$onResponse$0(ExecutionCallback.java:64)
at io.opentelemetry.javaagent.instrumentation.api.concurrent.RunnableWrapper.run(RunnableWrapper.java:28)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
16:47:10.119 [generic-mce-consumer-job-client-0-C-1] ERROR c.l.m.k.MetadataChangeProposalsProcessor:68 - MCP Processor Error
com.linkedin.r2.RemoteInvocationException: com.linkedin.data.template.RequiredFieldNotPresentException: Field "value" is required but it is not present
.
.
.
.
.
.
.
.
.
16:47:09.759 [generic-mce-consumer-job-client-0-C-1] ERROR c.l.m.k.MetadataChangeProposalsProcessor:69 - Message: {"auditHeader": null, "entityType": "dataset", "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:mysql,.....,PROD)", "entityKeyAspect": null, "changeType": "UPSERT", "aspectName": "container", "aspect": {"value": "{\"container\": \"urn:li:container:.....\"}", "contentType": "application/json"}, "systemMetadata": {"lastObserved": ...., "runId": "mysql-....", "registryName": null, "registryVersion": null, "properties": null}}
16:47:09.760 [generic-mce-consumer-job-client-0-C-1] INFO c.l.m.k.MetadataChangeProposalsProcessor:79 - Error while processing FMCP: FailedMetadataChangeProposal - {error=com.linkedin.r2.RemoteInvocationException: com.linkedin.data.template.RequiredFieldNotPresentException: Field "value" is required but it is not present
square-activity-64562
03/07/2022, 5:22 PMv0.8.27
and server is v0.8.14
? There have been many changes between those versions. You can see the releases in https://github.com/linkedin/datahub/releases?page=2
Strongly recommend to update the server. There have been many changes and they almost certainly will not work together.gifted-queen-80042
03/07/2022, 5:26 PMv0.8.14
?square-activity-64562
03/07/2022, 5:32 PMgifted-queen-80042
03/07/2022, 6:42 PMv0.8.20
square-activity-64562
03/08/2022, 6:14 AMgifted-queen-80042
03/08/2022, 8:10 PM