Hello, I have two questions: 1. I am trying to in...
# ingestion
c
Hello, I have two questions: 1. I am trying to install
datahub-protobuf
module described in this documentation (https://github.com/datahub-project/datahub/tree/master/metadata-integration/java/datahub-protobuf) but can't seem to find it on maven; can someone advise? 2. After running the code below with java emitter (using Scala below), I have found that some of the methods on
DatasetProperties
(e.g.
setTags
,
setQualifiedName
) don't alter anything in the user interface; can someone advise? Follow up: I notice, based on maven, that the
datahub-client
is fairly new; would it be fair to say that it's functionality is still fairly limited?
Copy code
val emitter: RestEmitter = RestEmitter.create(b => b
          .server("<http://localhost:8080>")
          .extraHeaders(Collections.singletonMap("Custom-Header", "custom-val"))
  )
  // emitter.testConnection()

  val tags = new StringArray()
  tags.add("featureStore")
  tags.add("bi")

  val url = new Url("<https://www.denofgeek.com/>")

  val customProperties = new StringMap()
  customProperties.put("governance", "disabled")
  customProperties.put("otherProp", "someValue")

  val mcpw = MetadataChangeProposalWrapper.builder()
    .entityType("dataset")
    .entityUrn("urn:li:dataset:(urn:li:dataPlatform:delta-lake,fraud.feature-stores.feature-store-v1,PROD)")
    .upsert
    .aspect(
      new DatasetProperties()
        .setName("feature-store")
        .setDescription("some feature store desc")
        .setTags(tags, SetMode.DISALLOW_NULL) // SetMode.IGNORE_NULL
        .setQualifiedName("fraudFeatureStore")
        .setExternalUrl(url)
    //    .setUri(new URI("<https://www.geeksforgeeks.org/>"))
        .setCustomProperties(customProperties)
    )
    .build

val requestFuture = emitter.emit(mcpw, null).get()
c
1. protobuf is not published on maven. Compilation instructions are provided in the document you are looking at. I hope you have referred to example here 2. On ui GlobalTags are displayed. They need to set using
globalTags
aspect of the dataset. Please refer to this and this. ( these are in python but you will get fair idea on how to do it) 3. Java emitter is fully functional. Let us know if anything is not working for you. We have wide range of example of how to emit thing in python at this location. you can always refer there.
c
Hi @careful-pilot-86309 thanks for your answer. W.r.t. #2 and #3, I will keep playing around with the emitter and leverage the examples. Regarding #1, I am new to gradle and so maybe I am not following the instructions correctly: I have just opened up a new gradle project via Intellij, can you tell me the steps I need to do to be able to import the following
import datahub.protobuf.ProtobufDataset;
?
Additionally: is it possible to update a schema via the Python emitter?
NVM: figured it out
c
Glad to hear that it worked out for you. 👍
thanks bear 1
m
hey @colossal-sandwich-50049: we finally published the datahub-protobuf artifact, so you can start depending on it from maven now.
c
Awesome! Happy to hear this, thanks @mammoth-bear-12532