Is it possible to schedule an ingestion to happen ...
# ingestion
t
Is it possible to schedule an ingestion to happen on a schedule by sending requests to GMS? I know that this is possible through the UI but we would like to use the UI for viewing data and not any management tasks.
l
Yes please use a scheduler like airflow to schedule the ingestion jobs using the appropriate connector
t
I was looking for a way that didn't involve adding another tool to our stack. Out of curiosity how does the UI schedule jobs? Would it be possible to mimic that behavior from outside of the UI?
s
You could do that by creating a corresponding DataHubExecutionRequest entity with a dataHubExecutionRequestInput​ aspect (https://datahubproject.io/docs/generated/metamodel/entities/datahubexecutionrequest). Alternatively you could use the GraphQL API and the createIngestionExecutionRequest mutation the UI is using (https://github.com/datahub-project/datahub/blob/7b5f28b919372a41e8ef90d20d8dc48411578de4/datahub-web-react/src/app/ingest/source/IngestionSourceList.tsx#L174 and https://github.com/datahub-project/datahub/blob/7b5f28b919372a41e8ef90d20d8dc48411578de4/datahub-web-react/src/graphql/ingestion.graphql#L125), then you only have to provide the URN of the ingestion source you want to execute...but I am not sure how stable this particular mutation is or if it might be changed in the future. There is also a documentation about how to use the GraphQL API (https://datahubproject.io/docs/api/graphql/getting-started). If you instead want to "manually" create the DataHubExecutionRequest entity you could look into the resolver of the mutation (https://github.com/datahub-project/datahub/blob/master/datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/resolvers/ingest/execution/CreateIngestionExecutionRequestResolver.java) and check how the resolver is building the DataHubExecutionRequest entity (e.g. the receipe and the config is coming from the DataHubIngestionSource using its URN).