Hi Everyone, I have successfully imported my metadata from some s3 paths in datahub using

from datahub.ingestion.run.pipeline import Pipeline.

Now I want to ingest the data dictionaries of the various metada ingested. I see the option to do this via the datahub interface but I can't find any documentation for doing this programmatically. Has anyone done this in the past? Is there any suggestion?

Hi Manrof, you simply can do:

pipeline = Pipeline.create(recipe)
pipeline.run()

where

recipe

is just a dictionary containing the recipe configuration as described in the documentation 🙂

That's for the reply @curved-planet-99787. In the documentation, i see the dictionary is being uploaded for hive. Is this going to be the same for all all other sources? https://datahubproject.io/docs/api/tutorials/datasets/

that example creates a hive table. In your case if the schema is already inside datahub already, you just need to programmatically add descriptions, tags and terms to the existing table like https://datahubproject.io/docs/api/tutorials/descriptions

Sorry, @hallowed-kilobyte-916 I didn't get your question, so my answer is probably not of help

@curved-planet-99787 It's fine...you led me down the path to the solution