set the channel description: Channel dedicated to conversation & collaboration to support protobuf in DataHub

Hi folks! Great to catch up this afternoon 🙂 I wasn’t able to find Rashmi - is she in the Slack community yet?

Probably not yet

👍

I haven’t had the chance to really dig into the datahub metadata model yet, however I am anticipating some need to extend it in order to handle the nesting of schema that is quite common in protobuf. While the nested structure could be represented as a struct in each dataset, that makes updating a nested schema a pain to identify all datasets with a specific nesting. At a glance the inter-dataset relationships mostly involve lineage or foreignkey relations. I am wondering if any work or thought put into this?

Here was the output to the protobuf integration investigation: https://github.com/linkedin/datahub/pull/4178 see the README for overview.

I am starting to consider the next steps around protobuf. 1. Domain support. For example, annotating a protobuf message to associate the dataset with a domain. 2. Ownership. Same idea but add the ownership. 3. Improve flexibility of the protobuf metadata annotations, for example using annotations like `meta.ownwership.team`and

meta.ownership.organization

instead of the

meta.msg.team

or

meta.msg.organization

. This would work exactly the same way as before but provide a way to group related annotations visually in the proto schema text. 4. List properties, allow repeated values to be collected into a json array string and stored as a property.

@mammoth-bear-12532 Interested in your perspective on the above ideas whenever you have a moment to consider. Thanks!

I don't quite understand why this file is checked into git: https://github.com/datahub-project/datahub/blob/b9068ffd2ef527a2b24b5d3828a9cc6c0f[…]gration/java/datahub-protobuf-example/libs/datahub-protobuf.jar

As far as I can tell only the first message in any protobuf file is being ingested, is this expected? can I somehow change this so it is not the case?

I think ideally https://github.com/datahub-project/datahub/blob/master/metadata-integration/java/datahub-protobuf-example/schema/protobuf/meta/meta.proto should be in https://buf.build/explore - have you considered adding it there? And should I just log an issue for "feature" requests like this?

Also still, this is really annoying:

java -cp /home/iwana/d.x/github.com/datahub-project/datahub/metadata-integration/java/datahub-protobuf/build/libs/datahub-protobuf-0.8.45-SNAPSHOT.jar <http://datahub.protobuf.App|datahub.protobuf.App> var/generated/main.dsc proto/protobufmessage/price-scraper.proto
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See <http://www.slf4j.org/codes.html#StaticLoggerBinder> for further details.

the SLF4J warnings

@future-smartphone-53257: we have some improvements in the ergonomics of using the standalone app based on some of your feedback. Specifically, support for custom platforms and subtypes and you can now point it to the root of the directory where your schema files live. Docs here. Latest jar is : here

Hi everyone, I haven't got any answer in

ingestion

channel so I hope I can get some here. Is there a way to specify which message to emit to Datahub if we have multiple non-nested schemas in

*.proto

source file? I can see this line in the documentation:

In addition, you can supply the root message in cases where a single protobuf source file includes multiple non-nested messages.

However I don't see any parameter to specify this. I am using

0.8.45

version of the datahub-protobuf

@brainy-tent-14503 might know the answer to this

has renamed the channel from "integration-protobuf" to "integrate-protobuf-datahub"

Hello! Has anyone managed to ingest proto schema recently?

Hello, I created an issue because there was a bug in the UI when ingest protobuf. Can you check it?

@mammoth-bear-12532 I'm having trouble with the protobuf code and raised a PR, but haven't heard back yet, is there any chance you can check it out? https://github.com/datahub-project/datahub/pull/9318