A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hello, when running the hive crawler is it normal to have the following warnings:
• `unable to map type DATE to metadata schema`
• `unable to map type TIMESTAMP to metadata schema`
• `unable to map type DECIMAL to metadata schema`

Isn't the `platformSchema` (platform-specific schema) meant to be generated?
Here is a sample of one of the crawled tables:
```"com.linkedin.pegasus2avro.schema.SchemaMetadata": {
                        "schemaName": "dev.bi.active_time_by_crowdmember_01_days",
                        "platform": "urn:li:dataPlatform:hive",
                        "version": 0,
                        "created": {
                            "time": 1614685268000,
                            "actor": "urn:li:corpuser:etl",
                            "impersonator": null
                        },
                        "lastModified": {
                            "time": 1614685268000,
                            "actor": "urn:li:corpuser:etl",
                            "impersonator": null
                        },
                        "deleted": null,
                        "dataset": null,
                        "cluster": null,
                        "hash": "",
                        "platformSchema": {
                            "com.linkedin.pegasus2avro.schema.MySqlDDL": {
                                "tableSchema": ""
                            }
                        },
...}```
Nothing appears in the field

<@U01L2QD243Z> when we introduced the new ingestion scripts, we tried to be “bug compatible” with the previous scripts that existed. That’s probably why we didn’t fill out the platformSchema field. 

So it is a bug that this field is not filled?

In my opinion, yes. The ui doesn’t break if you don’t fill it out though. 

You just see an empty json schema / raw schema tab. 

Could you file an issue and we can take a look at fixing this. 

With regard to your original question about type mapping, the script tries to canonicalize the platform schema to a common schema model. The warning tells us that it was unable to create an equivalent field (with the right type) in the canonical schema. This is the schema that you see when you view the dataset in the Schema tab in tabular format. 

If it can not map to the right type it will be String?

I will file the issue related to native schemas in the meantime

It looks like it maps it to NullType if it can’t find a match. <https://github.com/linkedin/datahub/blob/9e7379402265241f0580c3bf28dd4f8496907ee1/metadata-ingestion/src/datahub/ingestion/source/sql_common.py#L111|https://github.com/linkedin/datahub/blob/9e7379402265241f0580c3bf28dd4f8496907ee1/metadata-ingestion/src/datahub/ingestion/source/sql_common.py#L111>

Here is the bug report:
<https://github.com/linkedin/datahub/issues/2155>