https://datahubproject.io logo
Join Slack
Powered by
Hello. I've got a question about the RBAC feature ...
# ingestion
w
Hello. I've got a question about the RBAC feature on the roadmap. Will this include a solution for preventing different teams of overwriting each others metadata which might otherwise happen in a well federated metadata production landscape?
b
Can you elaborate on an example case? RBAC will help determine who can write what metadata through the UI and API at a granular level (asset level)
w
In my head I was thinking of some kind of namespace concept where permissions to create metadata would be granted on a namespace level.
b
That's something we hope to achieve using "domains"
But it will likely be the case that conflicting writes cannot be prevented at ingest time via the "ingest" apis, but rather via the UI and GraphQL api
w
Hmm. Ok. Preventing it at the ingest time would have been optimal ofc.
b
The default disposition is that batch ingest clients should be trusted to do the right thing.. Is that not the case for you folks?
We are actively working to implement RBAC so this feedback is quite useful
w
I don't think people would actively misbehave, just that accidents might happen.
I don't think we can assume batch ingests. By batch do you mean actual batches or just the kafka route?
b
basically anything being reported from the metadata ingestion framework, which hits a dedicated set of "ingest" endpoints
in the future, we will likely consider these endpoints as framework internal. and our public api will be GraphQL
w
Ok. Thanks! Do you know if the GraphQL api includes/will include async endpoints?
b
by async you mean "fire and forget"?
w
yup
@big-carpet-38439 Regarding
The default disposition is that batch ingest clients should be trusted to do the right thing
I'd like to check again if there are any plans to require authentication and perhaps a permission model on top of the gms rest api?
b
Authentication - yes. Permissions model, likely not
Unless there are strong use cases. What do you have in mind?
w
Some kind of namespacing of entities as a means of avoiding different teams accidentally overwriting each others entities. But getting authentication would be a great feature on its own.
b
Hmm got it...where does the risk of conflict come from? Is it multiple teams ingestin metadata from the same SQL db?
We really want to understand if this will be common or not
The thing about the Rest.li ingestion APIs: They are fairly low-level, coupled closely to how we actually store the data. Going forward we are trying to lighten the dependencies directly on storage so we have an easier time with public API model changes
s
@big-carpet-38439 e.g. We have 2 country teams - one using mysql with our company name as database name, another using mariadb with our company name as database name. As I was ingesting it I held on to ingesting tables and changed browse paths so there was no confusion. But if country teams themselves were doing the ingestion and there were common table names it would have caused problems
b
This to me is more a problem with how we identify individual data assets, which is going to improve soon enough
s
yes data instances would be great to have
➕ 1
w
Is it multiple teams ingestin metadata from the same SQL db
Well, probably not from the same physical db no. But potentially two different domains in an organisation could have named both their dbs the same thing without knowing. Maybe I'm grasping for straws here. Let's leave that part for now and I'll try to formulate more realistic examples if I can come up with them. Thanks for your patience. Regarding the auth, do you think that is something coming in Q3, Q4 or later?
Open in Slack
PreviousNext
Powered by