https://datahubproject.io logo
Join SlackCommunities
Powered by
Hello, I was ingesting some kafka topics, and so, ...
# ui
v
Hello, I was ingesting some kafka topics, and so, using: "platformSchema":{"documentSchema": schema}, everything works as expected, but on the front end, it appeared as such: I am using latest master from https://github.com/linkedin/datahub. Nothing too crazy appeared in the javascript console (just some 404) Am I doing something wrong / is it the intended behaviour? I was expecting to have all first level fields visible when viewing as a table.
b
Could you share the MCE you sent?
v
Copy code
{
  "auditHeader": null,
  "proposedSnapshot": {
    "com.linkedin.pegasus2avro.metadata.snapshot.DatasetSnapshot": {
      "urn": "urn:li:dataset:(urn:li:dataPlatform:kafka,whichevertopic,dev)",
      "aspects": [
        {
          "com.linkedin.pegasus2avro.dataset.DatasetProperties": {
            "description": {
              "string": "some description"
            },
            "uri": null,
            "tags": [
              "exampletag1",
              "exampletag2"
            ],
            "customProperties": {
              "producer": "fxt-data-store",
              "consumer": "fxt-data-bi"
            }
          }
        },
        {
          "com.linkedin.pegasus2avro.schema.SchemaMetadata": {
            "schemaName": "whichevertopic",
            "platform": "urn:li:dataPlatform:kafka",
            "version": 1,
            "created": {
              "time": 1599049279,
              "actor": "urn:li:corpuser:",
              "impersonator": null
            },
            "lastModified": {
              "time": 1599049279,
              "actor": "urn:li:corpuser:",
              "impersonator": null
            },
            "deleted": null,
            "dataset": null,
            "cluster": null,
            "hash": "",
            "platformSchema": {
              "com.linkedin.pegasus2avro.schema.KafkaSchema": {
                "documentSchema": "{\"type\": \"record\", \"name\": \"Envelope\", \"namespace\": \"fxt_data_stream_raw_landing_android_staging_order_download_hashes\", \"fields\": [{\"type\": \"string\", \"name\": \"hash\"}, {\"type\": \"long\", \"name\": \"_metadata_order\"}, {\"type\": \"string\", \"name\": \"_metadata_source\"}, {\"type\": \"boolean\", \"name\": \"_metadata_created\"}, {\"type\": \"boolean\", \"name\": \"_metadata_deleted\"}, {\"type\": \"long\", \"name\": \"_metadata_ingest_time\"}]}"
              }
            },
            "fields": [
              {
                "fieldPath": "",
                "jsonPath": null,
                "nullable": false,
                "description": {
                  "string": ""
                },
                "type": {
                  "type": {
                    "com.linkedin.pegasus2avro.schema.BooleanType": {}
                  }
                },
                "nativeDataType": "string",
                "recursive": false
              }
            ],
            "primaryKeys": null,
            "foreignKeysSpecs": null
          }
        }
      ]
    }
  },
  "proposedDelta": null
}
b
I see. LGTM. Could you create a ticket on GH so I can loop in the right person to take a look?
👍 1
v
https://github.com/linkedin/datahub/issues/1845 Issue created. I find it overall a bit confusing the correlation between ingestion & UI (For instance, all CustomProperties not represented in Datahub, etc.), if there are some videos about how people use datahub, that would maybe really help me 🙂
m
@victorious-dawn-14920 Thanks for the feedback. Videos/Clear step-by-step documentation for most specific/common tasks is something which we will focus on soon. For this particular usecase we can get you in touch with one of our UI experts.
@breezy-forest-52817 and @witty-hydrogen-79411 FYI
b
The issue has been resolved.
2 Views
Open in Slack
PreviousNext
Powered by