https://datahubproject.io logo
Join Slack
Powered by
Hello! I'm working on extending the metadata model...
# troubleshoot
b
Hello! I'm working on extending the metadata models, but I'm having an issue trying to load data. I created .pdl files in 
metadata-models-ext/com/example/
for my model's key, entity and snapshot. 
./gradelw build
picks up the models and generates code. When I create a work unit and attempt ingestion using 
datahub ingest -c
, I get the error 
AvroException: ('Datum union type not in schema: %s', 'com.example.TeamSnapshot')
. Any tips on what this error actually means?
b
Hey there! You likely need to be rebuild the 
metadata-ingestion
module and then use the local copy of the DataHub Cli during ingest
Under the hood, our ingestion framework uses strongly typed, auto generated Python classes to make writing integrations with other systems easier. The downside to this is that you need to regenerate those strongly typed bindings. The PyPI distribution of the CLI will not have the classes for the new model you've added
So
You can doing the following:
Copy code
cd metadata-ingestion 
source venv/bin/activate 
datahub ingest -c <your-recipe-file>
b
I have been using the local version of metadata-ingestion, exactly as you suggested. When I checked the code generation, the extended models are missing. It appears that gradle tasks from metadata-events/mxe-schemas do not pick up new models.
b
Ah I see. And the new models are present in 
gms/api/mainGeneratedDataTemplate
?
b
no, they are not there.
b
please run a
Copy code
./gradlew gms:war:clean && ./gradlew gms:war:build
and see if they are generated. they should be
b
they are not.
I also tried moving the models into metadata-models/src/main/pegasus/com/linkedin/example (and updated the namespace accordingly), also without results.
b
I just realized that you mentioned metadata-models-ext
You should indeed be defining in the metadata-models module
That is very very strange
Remember that you must additionally update the "Snapshot.pdl" file to include your new snapshot
b
oh. I did not know about the Snapshot.pdl update.
did I miss that line in the "Extending" instructions?
So moving files to 
metadata-models/src/main/pegasus/com/linkedin
and adding new snapshots to 
metadata-models/src/main/pegasus/com/linkedin/metadata/snapshot/Snapshot.pdl
helped. There are more things built, but still no new models in 
gms/api/mainGeneratedDataTemplate
b
No this is a documentation bug. It is not in there as a step. Will take this as an action item to add
TBH at this point I’m unsure of what it could be. We may just need to schedule a meeting time
2 Views
Open in Slack
PreviousNext
Powered by