https://datahubproject.io logo
Join Slack
Powered by
Hello folks!!! can you please guide me through int...
# ingestion
g
Hello folks!!! can you please guide me through into my lab to create a kafka emmitter please? I'm trying to construct datahub object with SchemaMetadata, but I received some errors, so do you know or help to try fix the code please? into my kafka emmitter python have : # Construcao dataset propriedades do objeto schema_metadata = SchemaMetadata( schemaName="dataset_name", platform="sql" version="0", hash="", platformSchema=MySqlDDL(tableSchema=""), fields="idproduto", ) # Construcao do MetadataChangeProposalWrapper objeto metadata_event = MetadataChangeProposalWrapper( entityType="dataset", changeType=ChangeTypeClass.UPSERT, entityUrn=builder.make_dataset_urn("mysql", "fabio-mysql.fabio-dataset.user-table"), aspectName="schemaMetadata", aspect=schema_metadata, ) but show the error: File "/home/fabiocastro/datahub/metadata-ingestion/src/datahub/emitter/fabiocastro_kafka_emitter.py", line 54 hash="", ^ SyntaxError: invalid syntax
m
@gentle-florist-49869, just wondering if you could be missing a "," after 
platform="sql"
g
@miniature-tiger-96062 tks , I fix this but continue not working with a lot errors - Do you have any schemametada py code to example, please? Where code construct the schematada object
m
Could you please share the new errors you are getting? I'll also look for an example in the meatime.
g
@miniature-tiger-96062 sure , below you can see the error and my py code , if you can help me I will appreciate tks
fabiocastro_kafka_emitter.py
m
@gentle-florist-49869: are you using any IDE to write your python code?
I highly recommend using one (e.g. VisualStudio)
g
@mammoth-bear-12532,yes I using VScode
I always prefer VsCode
🙂
m
@gentle-florist-49869: are you using a 
venv
for your development?
I wonder if you have the right avro packages in your pip
g
@mammoth-bear-12532 I'm not sure , in this initial lab I used: datahub docker quickstart --graph-service-impl elasticsearch
evertyhing in my local machine by docker
in my linux the pip list showing : avro 1.11.0 avro-gen3 0.7.1
m
could you swap out the kafka emitter for the rest emitter
to see if avro is the issue or the event content itself
g
@mammoth-bear-12532 yes , I can try swap and see the results ... I will do this tomorrow and back here 🙂
m
@gentle-florist-49869 was able to figure this out
Copy code
# Construcao dataset propriedades do objeto
schema_metadata = SchemaMetadata(
    schemaName="dataset_name",
    platform="urn:li:dataplatform:mysql",
    version=0,
    hash="",
    platformSchema=MySqlDDL(tableSchema=""),
    fields=[
        SchemaFieldClass(fieldPath="idproduto", type=SchemaFieldDataTypeClass(type=StringTypeClass()), nativeDataType="varchar(100)")
    ]
)
two differences ^ version is an 
int
not a string
and 
fields
is a 
List[SchemaFieldClass..]
if you replace your schema_metadata variable with what I pasted above, it will work.
g
@mammoth-bear-12532 I replaced the schema like you said - it works - but now my datahub lab ( by docker) experience issue (code 500) - so I did (datahub docker nuke) , prune docker images, containers , volume and network - so start new lab , make the emmiter and error happened
fabiocastro_kafka_emitter.py
m
@gentle-florist-49869 can you please look at the datahub-gms logs from your docker and see what the error is ?
g
@miniature-tiger-96062 sure, I can do that - tks for back and help
@miniature-tiger-96062 and @mammoth-bear-12532 follow the gms docker log ..... in this case my kafka emitter works fine and the web interface showing error code = 500 ...like into gms log too
gms_logs.txt
m
Seems like this is the culprit -
Copy code
Caused by: com.linkedin.data.template.TemplateOutputCastException: Invalid URN syntax: Urn entity type should be 'dataPlatform'.: urn:li:dataplatform:mysql
g
let me try with upper letter P , just a second please
@miniature-tiger-96062 it's works - thank you
Now I will try to populate the schema/fields 🙂
m
great! glad it worked đź‘Ť
đź‘Ť 1
Open in Slack
PreviousNext
Powered by