https://datahubproject.io logo
Join Slack
Powered by
:wave: Hoping someone can help me debug this erro...
# troubleshoot
w
👋 Hoping someone can help me debug this error message when I try to post a MCP event to GMS 🧵
👍 1
Error:
Copy code
{
  "error": "MCP request failed because\n> Request to <http://localhost:8080/aspects?action=ingestProposal> failed! with status code: 500. because\n> unexpected response code=500, resp[:2000]= {\"exceptionClass\":\"com.linkedin.restli.server.RestLiServiceException\",\"stackTrace\":\"com.linkedin.restli.server.RestLiServiceException [HTTP Status:500]: Unknown dereferenced type STRING for DataMap's schema \\\"string\\\"\\n\\tat com.linkedin.restli.server.RestLiServiceException.fromThrowable(RestLiServiceException.java:321)\\n\\tat com.linkedin.restli.server.BaseRestLiServer.buildPreRoutingError(BaseRestLiServer.java:202)\\n\\tat com.linkedin.restli.server.BaseRestLiServer.handleResourceRequest(BaseRestLiServer.java:242)\\n\\tat com.linkedin.restli.server.RestRestLiServer.handleResourceRequestWithRestLiResponse(RestRestLiServer.java:294)\\n\\tat com.linkedin.restli.server.RestRestLiServer.handleResourceRequest(RestRestLiServer.java:262)\\n\\tat com.linkedin.restli.server.RestRestLiServer.handleResourceRequest(RestRestLiServer.java:232)\\n\\tat com.linkedin.restli.server.RestRestLiServer.doHandleRequest(RestRestLiServer.java:215)\\n\\tat com.linkedin.restli.server.RestRestLiServer.handleRequest(RestRestLiServer.java:171)\\n\\tat com.linkedin.restli.server.RestLiServer.handleRequest(RestLiServer.java:130)\\n\\tat com.linkedin.restli.server.DelegatingTransportDispatcher.handleRestRequest(DelegatingTransportDispatcher.java:70)\\n\\tat com.linkedin.r2.filter.transport.DispatcherRequestFilter.onRestRequest(DispatcherRequestFilter.java:70)\\n\\tat com.linkedin.r2.filter.TimedRestFilter.onRestRequest(TimedRestFilter.java:76)\\n\\tat com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnRequest(FilterChainIterator.java:146)\\n\\tat com.linkedin.r2.filter.FilterChainIterator$FilterChainRestIterator.doOnRequest(FilterChainIterator.java:132)\\n\\tat com.linkedin.r2.filter.FilterChainIterator.onRequest(FilterChainIterator.java:62)\\n\\tat com.linkedin.r2.filter.TimedNextFilter.onRequest(TimedNextFilter.java:55)\\n\\tat com.linkedin.r2.filter.transport.ServerQueryTunnelFilter.onRestRequest(ServerQueryTunnelFilter.java:58)\\n\\tat com.linkedin.r2.filter.TimedRestFilter.onRestRequest(TimedRestFilter.jav, request: <http://localhost:8080/aspects?action=ingestProposal>"
}
Payload:
Copy code
{
  "proposal": {
    "aspect": {
      "com.linkedin.pegasus2avro.mxe.GenericAspect": {
        "contentType": "application/json",
        "value": "{\"customProperties\":null,\"description\":null,\"externalUrl\":null,\"name\":\"test_dag\",\"project\":null}"
      }
    },
    "aspectName": {
      "string": "dataFlowInfo"
    },
    "auditHeader": null,
    "changeType": "UPSERT",
    "entityKeyAspect": null,
    "entityType": "dataFlow",
    "entityUrn": {
      "string": "urn:li:dataFlow:(airflow,test_dag,prod)"
    },
    "systemMetadata": null
  }
}
o
Copy code
"aspectName": {
      "string": "dataFlowInfo"
    },
What are you using to generate the MCP? This looks wrong. aspectName is not a map type, it's a string.
Same with entityUrn
w
Thanks Ryan. Yea, I suspected that was the case. • I have a Golang service that will be publishing such MCP events to GMS. • I generated go models from avro using gogen-avro • The payload is essentially the auto-generated JSON xform But it seems like this is not compatible with the expected JSON schema. Is there a schema spec somewhere that I can use instead?
o
So one thing you can try is generating from our new OpenAPI spec: https://demo.datahubproject.io/openapi/v3/api-docs (and Swagger: https://demo.datahubproject.io/openapi/swagger-ui/index.html). I have not tested out generating Go models from it, but there should be better support for actual Json Schemas. My experience has been that generating code from our Avro schemas directly requires some extra work, largely due to the extensive use of Union types. I struggled with that a lot while I was writing the OpenAPI generation 😅
w
This is super useful. Yea, I'll abandon my avro->json route and use these instead 🙌 Thanks again!
🎉 1
o
Let me know how it goes! OpenAPI is all new with 0.8.34 so we're very much looking for feedback on improvements and I'm happy to help out with troubleshooting 😄
b
Hey @wide-dawn-46249 did you ever get the GoLang OpenAPI gen working? https://github.com/deepmap/oapi-codegen seems to be struggling pretty hard on the latest opeanpi schema 😕
w
Yea I didn’t get very far. Gave up and hand-rolled Go structs to match the OpenAPI spec instead.
b
Thanks for the reply 😁. How have you found that, if you don’t mind my asking. Pain to maintain, or not so bad?
w
I think it has worked pretty well for us in prod. Once I got get past the initial setup (the error responses from GMS are not super helpful), the API has remained fairly stable.
b
Amazing, thanks! I may eventually try to submit some diffs to the oapi-codegen package and get that working…. but that’s good to hear 🙂
5 Views
Open in Slack
PreviousNext
Powered by