https://datahubproject.io logo
Join Slack
Powered by
Trying to figure out how to add a custom metadata ...
# troubleshoot
g
Trying to figure out how to add a custom metadata model following these instructions: A Custom Metadata Model I've cloned the datahub repo and used ./gradlew build to build out the project. Now that it's built, I have been trying to get the custom aspect "customDataQualityRules" built. When I check localhost:8080/config I don't have any models currently. What I'm not sure about is how the model gets built. I will cd into the metadata-models-custom folder and then run ../gradlew build which will give me the output in the first screenshot. But the model doesn't update when I check localhost:8080/config. I was watching

Upcoming Feature Alert!! No-Code Extensions in Datahubā–¾

for help and saw that Shirshanka took a generated folder and put it in ~/.datahub/plugins/models/ I think I found the generated file in the dist folder, but when I copied it into ~/.datahub/plugins/models/ my localhost:8080/config still looks the same. I am not familiar with gradle or Java, so there could be something obvious that I'm missing. I also noticed the src folder updated with some new files. Not sure if there's anything I'm supposed to do with those either. I also confirmed that the aspect doesn't exist by running insert_custom_aspect.py. Anyway, I tried to explain as much as possible so hopefully its easy to see what I'm doing wrong. Any help would be appreciated to get this test sample working, thanks!
āœ… 2
o
Hiya! This guide is more complete and up to date, can you try out the steps there?
g
Hey @orange-night-91387, thanks for the reply! Hmm, so that's the page I was following, it has a link in step 4 that redirects to the 'A Custom Metadata Model' page (the link is called 'metadata-models-custom'). Should I just follow the options for the open-source fork instead? I am just cloning the repo instead of forking it
o
A couple of things:
but when I copied it into ~/.datahub/plugins/models/
Did you do this with the install command?
localhost:8080/config still looks the same
Did you try restarting your local docker container? Also make sure that in your GMS Docker container config you can see: 
/ETC/DATAHUB/PLUGINS
/Users/myuser/.datahub/plugins
under the mounts in the inspect tab
If you're following everything from that top level guide, this one should be working, you shouldn't need to maintain a fork. I just tested it out
If you can give a bit more context on how you're deploying, what system you're running on, etc. that might be helpful for seeing where you're going wrong
g
Did you do this with the install command?
No, I can do that though
Did you try restarting your local docker container?
Didn't do that either šŸ˜…
Also make sure that in your GMS Docker container config you can see: 
/ETC/DATAHUB/PLUGINS
/Users/myuser/.datahub/plugins
under the mounts in the inspect tab
Ok, I can do that
If you're following everything from that top level guide, this one should be working, you shouldn't need to maintain a fork. I just tested it out
@orange-night-91387 Did you define your entity in the metadata-models-custom folder or did you put it in the entity-registry folder? And did you rebuild datahub from the source folder or just from within the metadata-models-custom folder? I will go ahead and follow your instructions, but answering those questions would also help me understand
o
I just ran the gradle build task for metadata-models-custom, ran the install task, verified this added the zip to my plugins directory, then restarted GMS and checked the config endpoint. I didn't define a new one though, just used the example one
g
@orange-night-91387 Ok awesome, that makes sense. Let me try that
Yeah Iā€™m just trying to get the example one rn too as a first step
@orange-night-91387 Ok, I can see that it is under the mounts tab It tries to load it 5 times and then fails šŸ˜•
Here is the error:
Copy code
ERROR c.l.m.m.r.PluginEntityRegistryLoader:182 - Tried 5 times. Failed to load registry mycompany-dq-model with java.lang.IllegalArgumentException: Error while reading config file in path java.io.FileInputStream@55b67e5c: java.io.CharConversionException: Invalid UTF-8 start byte 0x9b (at char #11, byte #-1)

 at [Source: (FileInputStream); line: 1, column: 1]
As well as
Copy code
Caused by: org.yaml.snakeyaml.error.YAMLException: java.io.CharConversionException: Invalid UTF-8 start byte 0x9b (at char #11, byte #-1)
I am copying it into a folder called 
mycompany-dq-model
. I can also try it outside of that folder. Here was my install command:
Copy code
install build/dist/metadata-models-custom-0.0.0-dev.zip ~/.datahub/plugins/models/mycompany-dq-model
Also http://localhost:8080/config isn't pulling up anything for me rn, just getting a failed to load pg
Ah, it could be b/c I renamed the YAML at one point for entity registry:
Copy code
id: mycompany-dq-model-am
entities:
  - name: dataset
    aspects:
      - customDataQualityRules
o
Could be picking up an invalid character in the copy & paste maybe?
Try restoring the file through git and making the modification directly if you want to change the id in the registry file
Could also be getting added by your editor maybe if you're using a special text editor. To rule that out you can try editing the file in vi/vim/nano/emacs or an IDE like IntelliJ/Eclipse
g
I'm using VS code, but I'll go ahead and restore and try again
@orange-night-91387 Still getting the same error. Found this online: https://stackoverflow.com/questions/6352861/json-invalid-utf-8-middle-byte
Something about the YAML parsing it seems like
o
Hmm, this happens even with the fully default file? What OS are you using?
Also make sure to do a clean build and reinstall it along with restarting the docker container
g
Also make sure to do a clean build and reinstall it along with restarting the docker container
Yeah I've been doing that - I will do it one more time to be sure tho
Hmm, this happens even with the fully default file?
Yeah
What OS are you using?
@orange-night-91387 I'll re-clone the git repo and start from scratch and see if that solves it
o
Yeah that's very weird šŸ˜• Let me know how that goes
g
@orange-night-91387 Dang, even after reinstalling. This is what localhost:8080/config looks like
image.png
metadata-models-custom-0.0.0-dev.zip
metadata-models-custom-0.0.0-dev.zip
@orange-night-91387 You're the man, thank you so much!!! For all those that find this thread, I was installing it incorrectly (using the bash command instead of this gradle install). The correct command is 
./gradlew :metadata-models-custom:install
o
Nice! Glad you were able to get it working šŸ˜„
g
@orange-night-91387 I updated the docs with the instructions and opened a PR so if anyone else has issues that isn't familiar with Java like me, then they will still be able to get it working: https://github.com/datahub-project/datahub/pull/7400 If you could look it over and see if it makes sense to add or not that would be great, thanks
Open in Slack
PreviousNext
Powered by