Hi, I am trying to integrate datahub with great ex...
# troubleshoot
w
Hi, I am trying to integrate datahub with great expectations. I have a script which do some validation and send its result to datahub. I am having issues with integration. According the outputs printed in my script
action_list
contains Datahub configs and I investigate the traffic with nmap tool on port 8080 which is the gms port, and I can see that every requests are responded with HTTP status code 200. You can see some examples of the output:
Copy code
{"proposal": {"entityType": "assertion", "entityUrn": "urn:li:assertion:81f38ff71a20153a0291b5f087e75f55", "changeType": "UPSERT", "aspectName": "assertionInfo", "aspect": {"value": "{\"customProperties\": {\"expectation_suite_name\": \"test_suite_3\"}, \"type\": \"DATASET\", \"datasetAssertion\": {\"dataset\": \"urn:li:dataset:(urn:li:dataPlatform:mysql,<my_table>)\", \"scope\": \"DATASET_COLUMN\", \"fields\": [\"urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:mysql,london.logging_bid,PROD),id)\"], \"aggregation\": \"IDENTITY\", \"operator\": \"NOT_NULL\", \"nativeType\": \"expect_column_values_to_not_be_null\", \"nativeParameters\": {\"column\": \"id\", \"mostly\": \"1.0\"}}}", "contentType": "application/json"}}}

HTTP/1.1 200 OK.
Date: Fri, 07 Apr 2023 07:39:55 GMT.
Content-Type: application/json.
X-RestLi-Protocol-Version: 2.0.0.
Content-Length: 61.
Server: Jetty(9.4.46.v20220331).
.
{"value":"urn:li:assertion:81f38ff71a20153a0291b5f087e75f55"}

{"proposal": {"entityType": "assertion", "entityUrn": "urn:li:assertion:81f38ff71a20153a0291b5f087e75f55", "changeType": "UPSERT", "aspectName": "dataPlatformInstance", "aspect": {"value": "{\"platform\": \"urn:li:dataPlatform:great-expectations\"}", "contentType": "application/json"}}}

POST /aspects?action=ingestProposal HTTP/1.1.
Host: <my-host>:8080.
User-Agent: python-requests/2.26.0.
Accept-Encoding: gzip, deflate.
Accept: */*.
Connection: keep-alive.
X-RestLi-Protocol-Version: 2.0.0.
Content-Type: application/json.
Content-Length: 287.
What I expect now is that I should see the expectations result in Datahub UI in tab called
Validation
. Yet it looks disabled. So, maybe you can guide me what should be the reason for that.
đź“– 1
🔍 1
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!
a
Hi, could you confirm via graphQL that the data was actually ingested?
w
Hi Paul, I don’t know how to do it yet but I am going to search. By the way I’ve realized something. I am connecting to mariadb to validate data with great expectations. When I open datahub ui, the urn shows in the url bar is something like this
urn:li:dataset:(urn:li:dataPlatform:mariadb,<my-table-name>,PROD)
. Yet when I execute my validation script and listen the traffic in datahub-gms port 8080 I saw something like that.
Copy code
{
  "proposal": {
    "entityType": "assertion",
    "entityUrn": "urn:li:assertion:a922925e5e864e2567b926ec79a44750",
    "changeType": "UPSERT",
    "aspectName": "assertionInfo",
    "aspect": {
      "value": "{\"customProperties\": {\"expectation_suite_name\": \"test_suite\"}, \"type\": \"DATASET\", \"datasetAssertion\": {\"dataset\": \"urn:li:dataset:(urn:li:dataPlatform:mysql,<my-tabĹźe>,PROD)\", \"scope\": \"DATASET_COLUMN\", \"fields\": [\"urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:mysql,<my-table>,PROD),id)\"], \"aggregation\": \"IDENTITY\", \"operator\": \"BETWEEN\", \"parameters\": {\"maxValue\": {\"value\": \"null\", \"type\": \"UNKNOWN\"}, \"minValue\": {\"value\": \"100010\", \"type\": \"UNKNOWN\"}}, \"nativeType\": \"expect_column_values_to_be_between\", \"nativeParameters\": {\"column\": \"id\", \"min_value\": \"100010\"}}}",
      "contentType": "application/json"
    }
  }
}
As you can see urn starts with mysql not mariadb. Do you think that would be the problem?
Hi, Paul, I think the problem looks like the reason for different urn. I’ve solved this problem by changing the recipe that I’ve used in Datahub by changing the
type
from
mariadb
to
mysql
. Now I can see the test results in Datahub. Yet another problem occured 🙂 I couldn’t see any Assertions have run. Trying to figure it out why it gives me this error.
a
Hmm, this could indeed be a problem with the urn prefix, but it’s not super likely
What DH version are you using?
w
Here it is:
Copy code
DataHub CLI version: 0.10.1
Python version: 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110]
a
Ah OK, in the actions container can you see in the logs whether the assertions are initiated?