how many folks would like a UI flow for setting up ingestion? are there concerns people would have with this?

Can you explain this a bit more? is it setting up a new ingestion, running ad-hoc ingestions, something else?

What I get from @big-carpet-38439 is a no-code ingestion builder for most of the users and advance users can still use low-code implementation

So configuring ingestion via the UI for say BigQuery, Redshift, Snowflake. Probably also the ability to schedule ingestion periodically. What Rahul mentioned is on^

That sounds really interesting! 👍 Currently, connectors are just sort of trusted components… and even in that scenario, errors occur! 😅 Being able to deploy connectors from a UI so users can do it selfserve would be great but IMO requires to implement RBAC in the ingestion pipeline too.

Some inspiration from tableau server. Is quite straightfoward there for things like snowflake or redshift

wont be able to use this, am using only custom connectors that requires external credential files (kerberos)🤷‍♂️

There is more than credential in there 😉 There is also scheduling and observability and retries… But maybe that’s for a second iteration

It would be important to be able to disable the feature (perhaps via policies) so users wouldn't unexpectedly set up ingestion flows outside of a ci/cd workflow (which would be the primary path at my company)

Yeah this makes sense. We'd definitely have privileges dictating who can view and manage ingestion jobs

We currently leverage COTS data catalog that has ingestion and its scheduling built in its UI. While the UI for configuring ingestion may be useful (it already is pretty low code currently) we have found the scheduling within the tool to be not helpful. There are cases where you want to run your ingestion based on external dependencies.. if it is built in the tool, then integrating with those external dependencies becomes difficult. I would rather propose exposing an endpoint to trigger ingestion flows from outside which the enterprise scheduler that manages all dependencies can trigger.. this is what we have resorted to in our current deployment.

Hi folks! Just a heads up that this has been migrated to our Feature Request portal, and it’s currently in-flight! Feel free to add additional context in the post and/or subscribe to it to receive updates as work progresses https://feature-requests.datahubproject.io/admin/p/ui-based-ingestion-82fd2c65-11b6-48e3-921c-249dd1c753f7