https://datahubproject.io logo
Join Slack
Powered by
Hi, all! I added a source to datahub, and now I wa...
# ingestion
d
Hi, all! I added a source to datahub, and now I want to expand 
SchemaFieldDataType
. What steps should I do. I found the code related to schema is generated by avro_codegen.py, so I think the best way to expand 
SchemaFieldDataType
is adding a new 
.pdl
file to 
metadata-models/src/main/pegasus/com/linkedin/schema/
and then rebuild the project, right?
e
Yes!
You should just change pdls. Everything else is generated from the pdl model files
d
Oh, I see. But sometimes I think the process is too heavy, because building the project takes too much time and it often gets trouble. Is there a simple a way? For example, I just add a class in a 
.py
file?
e
The PDL file has to change for backend to understand the new type! For me, I try not to build the whole project, but just the schemas by running from metadata-ingestion directory!
Copy code
(cd .. && ./gradlew :metadata-events:mxe-schemas:build) && ./scripts/codegen.sh
This one should take less than a minute!
d
Ok, thank you very much
And the last question, For example, I am going to add a `SchemaFieldDataType`，I added a 
TextType.pdl
to 
metadata-models/src/main/pegasus/com/linkedin/schema/
, added 
TextType
to 
SchemaFieldDataType.pdl
, but it still doesn't work. is there anywhere else to change?
The error is "com.linkedin.schema.TextType" is not a member type of union[{"type":"record", "name" : "ArrayType"...}]
e
Is the error on python side? Do you see the generated class for TextType ?
d
Year, I saw the class in the 
schemaclass.py
, I don't think the error is on python side, because it also said "`exceptionClass: com.linkedin.restli.server.RestLiServiceException`". But it works well while ingesting other source so I think 
gms
is ok. So strange!
The failure is in sink report
e
Did you rebuild gms and deploy? It needs to pick up your mode changes!
d
Year, I get it. Some files in gms and frontend-react also need to be modified
e
No files. Just rebuilding!
Just need to pick up your model changes so our validation logic understands that whatever you are ingesting is a vlid input
d
But some error occurred if I don't modify some files. eg: 
metadata-service/restli-api/src/main/snapshot/com/linkedin/entity/entities/snapshot.json
and 
aspects/snapshot.json
and 
TypeIcon.tsx
in datahub-web-react
Is there any logic code to modify these files automatically?
e
ah interesting. tagging @green-football-43791 or @big-carpet-38439 on this one for TypeIcon.tsx
for snapshot.json, it should have been generated when you run gradlew build
was this not the case?
can you try running 
./gradlew clean :metadata-service:restli-api:build
?
d
Oh, thank you , I'll try it later!
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':metadata-service:restli-impl:checkRestModel'.
> See output for :metadata-service:restli-impl:checkRestModel. Summary written to /tmp/test-auto-compile/datahub-0.8.17/metadata-service/restli-impl/build/reports/checkRestModel/summary.txt
the error occurred while gradlew build
b
You may need to ignore backwards compatibility checks with
Copy code
-Prest.model.compatibility=ignore
d
ok, I'll try it later!
Now , 
com.linkedin.entity.entities.snapshot.json
and 
com.linkedin.entity.aspects.snapshot.json
have modified. But 
SchemaFieldMapper.java
in 
datahub-graphql-core
and 
Typelcon.tsx
in 
datahub-web-react
haven't changed. is it alright or a bug?
e
Yeah those files are not generated
and need to be updated. @green-football-43791 do you know if it would still work without changing this? He is trying to add a new schema field data type.
g
Hey @damp-minister-31834 - in order to add a new schema field data type you will need to manually update those two files as you have pointed out!
d
Ok, thank you very much!
Open in Slack
PreviousNext
Powered by