posting this here because it is probably a better channel fo DataHub #troubleshoot

posting this here because it is probably a better ...

ambitious-shoe-92590

02/21/2023, 1:33 AM

posting this here because it is probably a better channel for it. This might have been answered in the past but I have a quick question regarding the

Struct

type with "pull" based ingestion. I have a nested field called

data

which contains a number of child key:values. When I ingest this data with datahub, the outputted dataset will have the data field, but it is un-expandable. I've read into Field Paths and the differences between v1 and v2 paths, but I am a bit confused on how to actually get to the point of being able to "expand" the nested struct. Seems like emitters are used in some examples but from my understanding that is if you want to manually add fields to the schema? Any help would be appreciated, the data is coming from a S3 source if that makes a difference.

bulky-soccer-26729

02/21/2023, 3:28 PM

hey charlie! i might be able to help you out here. first, could you post what your data looks like after you ingest it? when using nested fields, you need to make sure that a new

SchemaField

gets created for each field on the dataset's

schemaMetadata

aspect in the

fields

field of that aspect. Then, for V1 field paths, we know it's nested using dot notation based on the fields

fieldPath

- so if

data

is a parent schema field then its child

fieldPath

would be

data.child_name

ambitious-shoe-92590

02/21/2023, 4:51 PM

@bulky-soccer-26729 Thanks for the reply. When you say "post what my data looks like" after ingestion, what exactly do you mean? Do i have to query this using datahubs api? Sorry for the basic question, I am still fairly new to Datahub's ecosystem

bulky-soccer-26729

02/21/2023, 4:55 PM

oh no worries! sorry I wasn't more specific. but yeah, a query with datahub's api would work to see what you're getting back on

schemaMetadata

for a dataset. Or you could use a tool like Sequel Ace to look at your mysql DB and show me the row for that aspect on the dataset you care about

ambitious-shoe-92590

02/21/2023, 5:11 PM

working on getting this through the datahub api. the data source is coming from s3 in parquet format.

ambitious-shoe-92590

02/21/2023, 5:42 PM

@bulky-soccer-26729 trying to use the api to get the schema information. From this sample request:

Copy code

curl  '<http://localhost:8080/entitiesV2/><url-encoded-entity-urn>'

I get this response:

Copy code

{
  "urn": "the urn",
  "aspects": {
    "datasetKey": {
      "created": {
        "actor": "urn:li:corpuser:__datahub_system",
        "time": 1677001110711
      },
      "name": "datasetKey",
      "type": "VERSIONED",
      "version": 0,
      "value": {
        "origin": "PROD",
        "name": "the name",
        "platform": "urn:li:dataPlatform:s3"
      }
    }
  },
  "entityName": "dataset"
}

I would expect to also see

schemaMetadata

under

aspects

is this a limitation of the s3 source?

ambitious-shoe-92590

02/21/2023, 5:45 PM

from the datahub docs for "s3 data lake" sources:

Copy code

Schemas for Parquet and Avro files are extracted as provided.

ambitious-shoe-92590

02/21/2023, 5:49 PM

from my knowledge, parquet is self describing so the schema can be inferred from the parquet files themselves.

bulky-soccer-26729

02/21/2023, 7:01 PM

hmm that's weird.. that's the full response you get back? that should include all aspects in the response so the fact that you're only getting a key aspect means there's no other data about the entity - such as its schema

ambitious-shoe-92590

02/21/2023, 7:08 PM

yeah thats it. Seems strange that that is the only aspect returned. I am able to view the top level key:values in the ui just fine so i would expect to get some sort of

schemaMetadata

back for that at least

bulky-soccer-26729

02/21/2023, 7:08 PM

yeah definitely. and the key:values are in your schema tab right?

ambitious-shoe-92590

02/21/2023, 7:09 PM

correct

ambitious-shoe-92590

02/21/2023, 7:09 PM

let me try and query again. I noticed some of the dependancies (acryl-actions) seemed to be in a bad state and maybe that has something to do with it

bulky-soccer-26729

02/21/2023, 7:12 PM

okay yeah definitely. I'll also drop some docs around field paths and getting nested fields here for you to check out as well to add context to what I was saying above: https://datahubproject.io/docs/advanced/field-path-spec-v2/#the-fieldpath-encoding-schemev2

ambitious-shoe-92590

02/21/2023, 7:19 PM

just did a test ingestion from an elasticsearch source and it works just fine out of the box. full schema expansion and when I query for that entity I get the full response including

schemaMetadata

. Seems like something specific to the datalake/s3 source.

ambitious-shoe-92590

02/21/2023, 7:20 PM

thanks for the links. Ill be digging into this more and will see if i can find anything

bulky-soccer-26729

02/21/2023, 7:21 PM

okay sounds good! yeah that might be it. please let us know if you find anything else 🙂

4 Views

Open in Slack

Previous Next