Hi team, for our use case, we are writing a

terraform provider

for datahub (and yes, we will opensource once it is matured). Something like below will add dataset for bigquery with schemas defined.

resource "datahub_dataset" "test" {
  platform = "bigquery"
  name = "testdataset"
  origin = "DEV"
  owner = "dinesh"
  schema_name = "test"
  
  field {
    field_path = "name"
    native_datatype = "String()"
    recursive = false
    nullable = true
  }

  field {
    field_path = "address"
    native_datatype = "String()"
    recursive = true
    nullable = false
  }

  tags = ["tag1", "tag2"]
  upstreams = [
    datahub_dataset.gcs.id,
    "urn:li:dataset:(urn:li:dataPlatform:gcs,testgcs,DEV)"
  ]
}

Question: For this terraform provider, we are using rest-api based results. Will it scale ? Should we use graphql based apis ?

Hi @victorious-dream-46349: this is quite cool. Curious what you are doing with the rest-api results.

Using rest api, we will read/create/update/delete resource based on the terraform resource files.

Was curious whether a one-way push api over Kafka will not work for your use-case and why

Reading this thread with much interest @victorious-dream-46349 😄 have you been able to come up with a solution for this? I am particularly interested with a solution for governance items: Policies, Groups and domains.