Hi! I haven’t found any mention to partition colum...
# ingestion
w
Hi! I haven’t found any mention to partition columns in the SchemaField metadata or any other schema-related PDL https://github.com/linkedin/datahub/blob/master/metadata-models/src/main/pegasus/com/linkedin/schema/SchemaField.pdl Is there any plan to include such info in the model?
s
I see it when I search for https://demo.datahubproject.io/search?page=1&query=partition It says
Copy code
Matches column [type=PartitionSpec].partitionSpec.[type=string].partition
So it in the hierarchy somewhere of dataset
w
It’s related to the time series thingy recently added https://github.com/linkedin/datahub/search?q=PartitionSpec but I cannot find how that links to the
SchemaMetedata
m
Hey @witty-butcher-82399 Partitioned Datasets are on our short-term roadmap; as you rightly pointed out, they are not in the metadata model yet.
I expect we will get to it when we do Spark / delta-lake integration.
👍 2