Hi! I’ve noted there is a new

DataPlatformInstance

aspect for datasets. Is that somehow related to the “domains” feature that was mentioned at some moment?

Hey @witty-butcher-82399! We did not have a way of standardizing the concept of platforms across all entities, which is why we added this new aspect. (For instance, it was orchestrator for data pipelines, dashboardTool for charts/dashboards)

Having those aspects and being able to filter entities sounds nice @early-lamp-41924. However, if the platform instance is not part of the URN, then we cannot differentiate two datasets with the same name in different platform instances. Eg: two kafka clusters having the same topic name would share the same URN:

urn:li:dataset:(urn:li:dataPlatform:kafka,TopicName,PROD)

Right now we have a custom transform to overwrite data platform so we can have:

urn:li:dataset:(urn:li:dataPlatform:kafka@clusterX,TopicName,PROD)

and

urn:li:dataset:(urn:li:dataPlatform:kafka@clusterY,TopicName,PROD)

Is this something that will be covered with the domains feature?

For sure. We created this aspect as a backward compatible stop gap. We are planning a major update with backward incompatible changes to our model which will include this! we will create migration strategies once that happens!

That sounds great! Could you please share a rough estimation for that? Thanks!

Hi @witty-butcher-82399 this work is planned for this quarter. We want to get this done before end of this calendar year.

That’s great news, thanks! CC: @dry-lizard-54588 @breezy-guitar-97226

Just to be conservative: assume it might slip to early Jan given holidays and all.

yes, all plannings in December has some uncertainty…. we also suffer that 😅

Hi @early-lamp-41924. I came across this thread and (https://datahubspace.slack.com/archives/CUMUWQU66/p1628520633160700), searching for the solution for the same use case. I am also using a custom transformer to change dataset name in urn(as suggested in linked thread), which worked for me. Now I want to run sql profiling on the same database, but instead of adding that info to existing records, new records are being created in DataHub and it looks like transformer is not being applied when profiling is enabled. E.g. datasetKey created through ingestion (transformers are applied) urn:

urn:li:dataset:(urn:li:dataPlatform:mysql,my_machine@classicmodels.products,PROD)

aspect:

datasetKey

metadata:

{"origin":"PROD","name":"my_machine@classicmodels.products","platform":"urn:li:dataPlatform:mysql"}

When same recipe is executed with profiling enabled. urn:

urn:li:dataset:(urn:li:dataPlatform:mysql,classicmodels.products,PROD)

aspect:

datasetKey

metadata:

{"name":"classicmodels.products","platform":"urn:li:dataPlatform:mysql","origin":"PROD"}

My recipe looks like this.

source:

 

type: mysql

 

config:

   

username: my_username

   

password: my_password

   

host_port: localhost:15000

   

database: classicmodels

   

profiling:

     

enabled: true

sink:

 

type: datahub-rest

 

config:

   

server: <http://localhost:8080>

transformers:

- type: example_transformer.ChangeDataset

 

config:

   

instance_id: "my_machine"

Is there something that I am missing here, that is resulting in duplication of all my tables with new urn. I would like to mention that if I don't modify the urn with my transformer, it works fine. TLDR; looking for a way to profile datasets with custom name. Thanks in advance.

This was already noted here https://github.com/linkedin/datahub/issues/3300 In our case we have a transform that adds the instance information to the dataPlatform part of the URN, instead of the dataset name: eg

urn:li:dataset:(urn:li:dataPlatform:kafka@my-cluster,my-topic,PROD)

. And we have a couple of redshift recipes with

profile

enabled and it works. We overcome the issue by handling the different envelopes in the transform:

def transform(
        self, record_envelopes: Iterable[RecordEnvelope]
    ) -> Iterable[RecordEnvelope]:

        # loop over envelopes
        for envelope in record_envelopes:
            # if envelope is an MCE, MCP or MCPW add the cluster name
            # see <https://github.com/linkedin/datahub/issues/3300> for an explanation
            if isinstance(envelope.record, MetadataChangeEventClass):
                envelope.record = self.transform_one(envelope.record)
            elif isinstance(envelope.record, MetadataChangeProposalClass):
                envelope.record = self.transform_one_proposal(envelope.record)
            elif isinstance(envelope.record, MetadataChangeProposalWrapper):
                envelope.record = self.transform_one_proposal_wrapper(envelope.record)
            yield envelope

Hope this helps.

Yes that makes it clear. Thank you.