Hello Everyone, i’m exploring datahub capabilities, and try to set custom properties on a dataset. Looking at the official documentation at https://datahubproject.io/docs/api/datahub-apis/#datahub-api-comparison it says that custom properties on dataset can be added/removed/replaced using OpenApi. Now, if i come to the swagger exposed by my datahub instance, I cannot figure out where this rest api is located: the dataset seems only to support the POST (creation) method and as well, I cannot find the create aspect custom properties for that dataset. Anyone is able to help me finding this? Thanks

For this, you can use the python api -> https://datahubproject.io/docs/api/tutorials/custom-properties#add-and-remove-custom-properties-programmatically @big-carpet-38439 can help you answer how you can do it with the open API endpoint.

ok good. I’m in a first phase of this exploring, so currently, i’ve deployed it in GKE using your official helm chart. Of course now, building and deploying a java application would require time to setup etc. so it would be very very nice if i can do it with rest api

yeah, I can understand, we have also option to use the Python API where maybe the setup can be easier.

setting up another pod with the python runtime, python app and so on would require time, not a java specific problem, but I need to keep my environment reproducible, so setting up a new application requires pipelines, building, testing etc . Rest api or graphql can be called directly from my computer

Custom properties can be modified via the generic /entities/v1 endpoint. This endpoint is exceedingly generic but very powerful. The drawback is that it is not intuitive and we’re building out other more explicit endpoints over time. In version v0.11.0, there is a way to interact directly with the openapi entities endpoint to manipulate any aspect however patching is not supported. This means to remove/add 1 property on a list of properties, rather then setting the entire list, requires a read before write pattern unless you can readily compute all properties at once. Other SDKs rely on the rest.li endpoint which include support for patch operations. Since you are familiar with java and the openapi documentation is lacking let me show you the java backend. The entities endpoint is actually accepting a list of UpsertAspectRequests. This includes a field called

aspect

which is a massive oneOf which includes

datasetProperties

Here is a concrete example and as mentioned it will replace the customProperties on a dataset.

[{
	"entityType": "dataset",
	"entityUrn": "urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)",
	"aspect": {
		"__type": "DatasetProperties",
		"customProperties": {
			"foo": "bar"
		}
	}
}]