Hi Team, Is there a way to enable queries tab on h...
# ingestion
s
Hi Team, Is there a way to enable queries tab on hive datasets. We capture table usage from hive-metastore using hooks and we would like push this information to datahub. I was able to ingest some sample data using file source but the queries tab is still disabled on UI. Can you please suggest how to enable queries tab in UI on datasets ingested using file source ? sample data ingest using file source
Copy code
{
  "auditHeader": null,
  "entityType": "dataset",
  "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)",
  "entityKeyAspect": null,
  "changeType": "UPSERT",
  "aspectName": "datasetUsageStatistics",
  "aspect": {
    "value": "{\"timestampMillis\": 1623888000000, \"eventGranularity\": {\"unit\": \"DAY\", \"multiple\": 1}, \"partitionSpec\": {\"type\": \"FULL_TABLE\", \"partition\": \"FULL_TABLE_SNAPSHOT\"}, \"uniqueUserCount\": 1, \"totalSqlQueries\": 2, \"topSqlQueries\": [\"select * from `test`\"], \"userCounts\": [{\"user\": \"urn:li:corpuser:xx\", \"count\": 2, \"userEmail\": \"xx\"}], \"fieldCounts\": [{\"fieldPath\": \"complaint_description\", \"count\": 2}, {\"fieldPath\": \"last_update_date\", \"count\": 2}, {\"fieldPath\": \"complaint_type\", \"count\": 2}, {\"fieldPath\": \"unique_key\", \"count\": 2}, {\"fieldPath\": \"source\", \"count\": 1}, {\"fieldPath\": \"city\", \"count\": 1}, {\"fieldPath\": \"map_tile\", \"count\": 1}, {\"fieldPath\": \"longitude\", \"count\": 1}, {\"fieldPath\": \"state_plane_y_coordinate\", \"count\": 1}, {\"fieldPath\": \"map_page\", \"count\": 1}, {\"fieldPath\": \"status_change_date\", \"count\": 1}, {\"fieldPath\": \"latitude\", \"count\": 1}, {\"fieldPath\": \"incident_zip\", \"count\": 1}, {\"fieldPath\": \"status\", \"count\": 1}, {\"fieldPath\": \"created_date\", \"count\": 1}, {\"fieldPath\": \"county\", \"count\": 1}, {\"fieldPath\": \"owning_department\", \"count\": 1}, {\"fieldPath\": \"street_name\", \"count\": 1}, {\"fieldPath\": \"close_date\", \"count\": 1}, {\"fieldPath\": \"street_number\", \"count\": 1}, {\"fieldPath\": \"incident_address\", \"count\": 1}, {\"fieldPath\": \"state_plane_x_coordinate\", \"count\": 1}, {\"fieldPath\": \"council_district_code\", \"count\": 1}, {\"fieldPath\": \"location\", \"count\": 1}]}",
    "contentType": "application/json"
  },
  "systemMetadata": {
    "lastObserved": 1626739200000,
    "runId": "test-hivequery-usage",
    "registryName": null,
    "registryVersion": null,
    "properties": null
  }
}
b
what you captured in the file belongs to the Stats tab. Queries is not automatically generated for Hive at the moment.
s
thanks @better-orange-49102. is it possible to ingest queries data through file source ? i know queries is not supported for hive source. can you please share a sample data for queries similar to above json ?
g
Actually, what is the queries will show?