hi all, I want to ask how to add a scheduler if I'm trying to ingest data using package from

datahub.ingestion.run.pipeline

the whole code is

from datahub.ingestion.run.pipeline import Pipeline

# The pipeline configuration is similar to the recipe YAML files provided to the CLI tool.

pipeline = Pipeline.create(

{

   

'source':{

     

"type":"postgres",

     

"config":{

         

"username":"postgres",

         

"password":"strongpass",

         

"database":"northwind",

         

"host_port":"localhost:5432",

         

"database_alias":"test",

         

"schema_pattern":{

           

"allow":{

               

"public"

           

}

         

},

         

"table_pattern":{

           

"allow":[

               

"test.public.region",

               

"test.public.suppliers"

           

]

         

}

     

}

   

},

   

"sink":{

     

"type":"datahub-rest",

     

"config":{

         

"server":"<http://localhost:8080>"

     

}

   

}

}

)

# Run the pipeline and report the results.

pipeline.run()

pipeline.pretty_print_summary()

and I want to try to make a scheduler from running that config, is it possible to do that? thank you.

Hi @rhythmic-kitchen-64860! Gentle reminder to please use Threads when posting large blocks of code - it’s a HUGE help for us while we’re trying to keep track of which questions are still unanswered across channels! teamwork

Hi @rhythmic-kitchen-64860, please check this out: https://datahubproject.io/docs/ui-ingestion

We have a scheduling guide https://datahubproject.io/docs/metadata-ingestion/schedule_docs/intro