A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

&gt; 2. What is the difference between PDLs under `metadata-models` &amp; graphqls under `datahub-graphql-core`? E.g. <https://datahubproject.io/docs/generated/metamodel/entities/dataprocessinstance|DataProcessInstance> entity looks very promising to our needs. However, we noticed there are types only in graphql but not in PDL (<https://github.com/datahub-project/datahub/blob/master/datahub-graphql-core/src/main/resources/entity.graphql#L986-L1006|DataProcessInstanceResult>), types only in PDL but not in graphql (<https://github.com/treff7es/datahub/blob/master/metadata-models/src/main/pegasus/com/linkedin/dataprocess/DataProcessInfo.pdl|DataProcessInfo>) and types in both places with slight differences (<https://github.com/datahub-project/datahub/blob/master/datahub-graphql-core/src/main/resources/entity.graphql#L4639-L4654|DataProcessInstanceRunResultType> and <https://github.com/treff7es/datahub/blob/master/metadata-models/src/main/pegasus/com/linkedin/dataprocess/DataProcessInstanceRunResult.pdl#L8-L21|RunResultType>). If we need to create a new aspect, how do I configure both places? :thread: 


The PDLs under `metadata-models` form the "base metadata model" for DataHub. They result in auto-generated Avro schemas, support for generic Rest-ful endpoints and auto-indexing / graph+search. We are close to merging in automatic OpenAPI spec generation from this as well.

The graphql types found in `datahub-graphql-core` form a sort of "business-logic layer" on top of this raw metadata model, to
1. Solve for modeling mistakes / regret that we might have in the base layer
2. Simplify certain common operations that might be too verbose to express using the base model. 

Work in the "business-logic" layer typically is manual, and involves a lot of hand-written mappers from the graphql types to the underlying pegasus types with some "transformation logic" embedded.

We have plans to support auto-generation of GraphQL types from the metadata model (as a bootstrapping way to reduce the cost of spinning up that first graphql resolver)... but we expect that as the application needs evolve, some manual work will be done in the biz-logic layer and that will make the "business model" drift from the "data model".

We don't exactly have a silver bullet here, curious what your thoughts are on this particular subject and what sort of solution you are looking for in this area.

To answer your specific question -&gt; To create a new aspect, you start with PDL, and then to surface it in GQL you usually create a shadow GQL type by hand and wire up the GraphQL resolver manually to have it become serve-able by the GQL API. There is a dynamic GQL thing that we did for auto-rendering aspects that requires no additional code to be written which <@U01JUBAPZPS> is best positioned to explain.