Hi team What's the difference between module file...
# all-things-deployment
c
Hi team What's the difference between module file and module datahub lineage file ? And in which case we prefer one over other ?
i
Hello Karishma, What files are you referring to? Could you link them here?
c
There are two types of sources - file and file based lineage ?
i
Could you send the link to the documentation?
If you’re talking about these: • file: http://datahubproject.io/docs/generated/ingestion/sources/file • file-based lineage: http://datahubproject.io/docs/generated/ingestion/sources/file-based-lineage/ The differences are that file ingestion is a generic input that accepts MCP definitions as json from a local filesystem (see example here: https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/mce_files/bootstrap_mce.json). You can certainly define MCPs that produce lineage metadata but that would be quite big files. File-based lineage is a local file with yml content (not json) with a simplified structured (not MCP) that defines lineage between pre-existing entities (see example here: https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/bootstrap_data/file_lineage.yml)
You want to use file ingestion when you want to ingest any metadata that datahub supports but you have to write the low-level MCP for it. It is generic (you can define anything there that datahub supports). File-based lineage is when you want to define the lineage (ONLY) between pre-existing entities. That is all that file-based lineage allows.