• g

    gentle-night-56466

    8 months ago
    I haven’t had the chance to really dig into the datahub metadata model yet, however I am anticipating some need to extend it in order to handle the nesting of schema that is quite common in protobuf. While the nested structure could be represented as a struct in each dataset, that makes updating a nested schema a pain to identify all datasets with a specific nesting. At a glance the inter-dataset relationships mostly involve lineage or foreignkey relations. I am wondering if any work or thought put into this?
    g
    w
    3 replies
    Copy to Clipboard
  • g

    gentle-night-56466

    6 months ago
    I am starting to consider the next steps around protobuf. 1. Domain support. For example, annotating a protobuf message to associate the dataset with a domain. 2. Ownership. Same idea but add the ownership. 3. Improve flexibility of the protobuf metadata annotations, for example using annotations like meta.ownwership.teamand
    meta.ownership.organization
    instead of the
    meta.msg.team
    or
    meta.msg.organization
    . This would work exactly the same way as before but provide a way to group related annotations visually in the proto schema text. 4. List properties, allow repeated values to be collected into a json array string and stored as a property.
    g
    m
    3 replies
    Copy to Clipboard
  • f

    future-smartphone-53257

    2 weeks ago
  • f

    future-smartphone-53257

    2 weeks ago
    As far as I can tell only the first message in any protobuf file is being ingested, is this expected? can I somehow change this so it is not the case?
    f
    m
    2 replies
    Copy to Clipboard
  • f

    future-smartphone-53257

    2 weeks ago
    I think ideally https://github.com/datahub-project/datahub/blob/master/metadata-integration/java/datahub-protobuf-example/schema/protobuf/meta/meta.proto should be in https://buf.build/explore - have you considered adding it there? And should I just log an issue for "feature" requests like this?
    f
    m
    6 replies
    Copy to Clipboard
  • f

    future-smartphone-53257

    2 weeks ago
    Also still, this is really annoying:
    java -cp /home/iwana/d.x/github.com/datahub-project/datahub/metadata-integration/java/datahub-protobuf/build/libs/datahub-protobuf-0.8.45-SNAPSHOT.jar <http://datahub.protobuf.App|datahub.protobuf.App> var/generated/main.dsc proto/protobufmessage/price-scraper.proto
    SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
    SLF4J: Defaulting to no-operation (NOP) logger implementation
    SLF4J: See <http://www.slf4j.org/codes.html#StaticLoggerBinder> for further details.
    the SLF4J warnings
    f
    m
    4 replies
    Copy to Clipboard
  • m

    mammoth-bear-12532

    5 days ago
    @future-smartphone-53257: we have some improvements in the ergonomics of using the standalone app based on some of your feedback. Specifically, support for custom platforms and subtypes and you can now point it to the root of the directory where your schema files live. Docs here. Latest jar is : here