I didn t see the `status` but `removed false` just showed up DataHub #getting-started

I didn't see the `status`, but `removed:false` jus...

acceptable-architect-70237

06/03/2020, 4:16 AM

I didn't see the

status

, but

removed:false

just showed up

bumpy-keyboard-50565

06/03/2020, 4:30 PM

If I read this correctly,

Ownerships

was ingested correctly but not

DataProcessInfo

, and

Status

was magically set?

acceptable-architect-70237

06/03/2020, 5:37 PM

that's correct. any chance know where I did wrong? Apparently Kafka schema doesn't pick up

DataProcessInfo

bumpy-keyboard-50565

06/03/2020, 5:41 PM

I assume you also rebuilt

mce-consumer

after updating the PDL models?

acceptable-architect-70237

06/03/2020, 6:05 PM

yes. I rebuild two . one is gms, the other is the

mce-consumer

acceptable-architect-70237

06/03/2020, 6:10 PM

here is my fork branch in case you need it. https://github.com/liangjun-jiang/datahub/tree/data-process-entity. thanks!

bumpy-keyboard-50565

06/03/2020, 7:09 PM

Okay looks like some weirdness with Avro serialization in Python. Looks like when there's a schema mismatch, instead of failing loudly, the

avro-python3

lib does something weird (i.e. randomly picks a type from the union). Simply drop the extra

dataProcessInfo

in your MCE should fix the issue.

Copy code

{"auditHeader": None, "proposedSnapshot": ("com.linkedin.pegasus2avro.metadata.snapshot.DataProcessSnapshot", {"urn": "urn:li:dataprocess:(sqoop2,4DEMO2,PROD)", "aspects": [{"owners": [{"owner": "urn:li:corpuser:datahub", "type": "DATAOWNER"}], "lastModified": {"time": 0, "actor": "urn:li:corpuser:datahub"}}, { "outputs": [ "urn:li:dataset:(urn:li:dataPlatform:cassandra,barEarth,DEV)", "urn:li:dataset:(urn:li:dataPlatform:cassandra,barMars,DEV)" ], "inputs": [ "urn:li:dataset:(urn:li:dataPlatform:hbase,barSky,PROD)", "urn:li:dataset:(urn:li:dataPlatform:hbase,barOcean,PROD)" ] } ]}), "proposedDelta": None}

acceptable-architect-70237

06/03/2020, 7:34 PM

thanks. I think I have tried this. and tried again, still not see it. want to make sure the

mce-cli.py

is running with python 3, right?

bumpy-keyboard-50565

06/03/2020, 7:39 PM

Yes it should work with python 3. There's probably other validation errors, try to wrap

DataProcessInfo

in a tuple with the type info as the first element and the value as the second like this

Copy code

("com.linkedin.pegasus2avro.dataprocess.DataProcessInfo, {"inputs":[...], "outputs": [...]})

acceptable-architect-70237

06/03/2020, 8:07 PM

still not luck.

bumpy-keyboard-50565

06/03/2020, 8:25 PM

Hum. What's the error you're getting?

acceptable-architect-70237

06/03/2020, 8:36 PM

not getting any error. the

dataprocessinfo

aspect doesn't show. Since you think it's the python

avro

package causes this problem, I think I can use other tools, maybe a

java client

or approach like this https://stackoverflow.com/questions/51664191/pushing-avro-file-to-kafka to generate a valid message?

acceptable-architect-70237

06/03/2020, 8:36 PM

my point is that I don't have to use this python lib.

bumpy-keyboard-50565

06/03/2020, 8:37 PM

Could you share the latest MCE with me again? Thanks

acceptable-architect-70237

06/03/2020, 8:38 PM

Copy code

{"auditHeader": None, "proposedSnapshot": ("com.linkedin.pegasus2avro.metadata.snapshot.DataProcessSnapshot", {"urn": "urn:li:dataprocess:(21sqoop121,4DEMO3,PROD)", "aspects": [{"owners": [{"owner": "urn:li:corpuser:datahub", "type": "DATAOWNER"}], "lastModified": {"time": 0, "actor": "urn:li:corpuser:datahub"}}, { "com.linkedin.pegasus2avro.dataprocess.DataProcessInfo": { "outputs": [ "urn:li:dataset:(urn:li:dataPlatform:cassandra,barEarth,DEV)", "urn:li:dataset:(urn:li:dataPlatform:cassandra,barMars,DEV)" ], "inputs": [ "urn:li:dataset:(urn:li:dataPlatform:hbase,barSky,PROD)", "urn:li:dataset:(urn:li:dataPlatform:hbase,barOcean,PROD)" ] } }]}), "proposedDelta": None}

bumpy-keyboard-50565

06/03/2020, 8:40 PM

You're not using a tuple, which is specified using

(...)

instead of

{...}

. This is how you should do it

Copy code

{"auditHeader": None, "proposedSnapshot": ("com.linkedin.pegasus2avro.metadata.snapshot.DataProcessSnapshot", {"urn": "urn:li:dataprocess:(21sqoop121,4DEMO3,PROD)", "aspects": [{"owners": [{"owner": "urn:li:corpuser:datahub", "type": "DATAOWNER"}], "lastModified": {"time": 0, "actor": "urn:li:corpuser:datahub"}}, ( "com.linkedin.pegasus2avro.dataprocess.DataProcessInfo": { "outputs": [ "urn:li:dataset:(urn:li:dataPlatform:cassandra,barEarth,DEV)", "urn:li:dataset:(urn:li:dataPlatform:cassandra,barMars,DEV)" ], "inputs": [ "urn:li:dataset:(urn:li:dataPlatform:hbase,barSky,PROD)", "urn:li:dataset:(urn:li:dataPlatform:hbase,barOcean,PROD)" ] } )]}), "proposedDelta": None}

acceptable-architect-70237

06/03/2020, 8:40 PM

let me give it a try. thakns

bumpy-keyboard-50565

06/03/2020, 8:41 PM

Here's the doc: https://fastavro.readthedocs.io/en/latest/writer.html#using-the-tuple-notation-to-specify-which-branch-of-a-union-to-take

acceptable-architect-70237

06/03/2020, 8:43 PM

Copy code

Traceback (most recent call last):
  File "mce_cli.py", line 108, in <module>
    main(parser.parse_args())
  File "mce_cli.py", line 88, in main
    produce(conf, args.data_file, args.schema_record)
  File "mce_cli.py", line 31, in produce
    content = ast.literal_eval(sample.strip())
  File "/Users/liajiang/opt/anaconda3/lib/python3.7/ast.py", line 46, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/Users/liajiang/opt/anaconda3/lib/python3.7/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 1
    {"auditHeader": None, "proposedSnapshot": ("com.linkedin.pegasus2avro.metadata.snapshot.DataProcessSnapshot", {"urn": "urn:li:dataprocess:(21sqoop121,4DEMO3,PROD)", "aspects": [{"owners": [{"owner": "urn:li:corpuser:datahub", "type": "DATAOWNER"}], "lastModified": {"time": 0, "actor": "urn:li:corpuser:datahub"}}, ( "com.linkedin.pegasus2avro.dataprocess.DataProcessInfo": { "outputs": [ "urn:li:dataset:(urn:li:dataPlatform:cassandra,barEarth,DEV)", "urn:li:dataset:(urn:li:dataPlatform:cassandra,barMars,DEV)" ], "inputs": [ "urn:li:dataset:(urn:li:dataPlatform:hbase,barSky,PROD)", "urn:li:dataset:(urn:li:dataPlatform:hbase,barOcean,PROD)" ] } )]}), "proposedDelta": None}
                                                                                                                                                                                                                                                                                                                                                                                        ^
SyntaxError: invalid syntax

bumpy-keyboard-50565

06/03/2020, 8:44 PM

Oops sorry, my bad. Should also change

"com.linkedin.pegasus2avro.dataprocess.DataProcessInfo":

"com.linkedin.pegasus2avro.dataprocess.DataProcessInfo",

Copy code

{"auditHeader": None, "proposedSnapshot": ("com.linkedin.pegasus2avro.metadata.snapshot.DataProcessSnapshot", {"urn": "urn:li:dataprocess:(21sqoop121,4DEMO3,PROD)", "aspects": [{"owners": [{"owner": "urn:li:corpuser:datahub", "type": "DATAOWNER"}], "lastModified": {"time": 0, "actor": "urn:li:corpuser:datahub"}}, ( "com.linkedin.pegasus2avro.dataprocess.DataProcessInfo", { "outputs": [ "urn:li:dataset:(urn:li:dataPlatform:cassandra,barEarth,DEV)", "urn:li:dataset:(urn:li:dataPlatform:cassandra,barMars,DEV)" ], "inputs": [ "urn:li:dataset:(urn:li:dataPlatform:hbase,barSky,PROD)", "urn:li:dataset:(urn:li:dataPlatform:hbase,barOcean,PROD)" ] } )]}), "proposedDelta": None}

acceptable-architect-70237

06/03/2020, 8:47 PM

it works!

bumpy-keyboard-50565

06/03/2020, 8:49 PM

Yup. Updating the document now

acceptable-architect-70237

06/03/2020, 8:49 PM

acceptable-architect-70237

06/03/2020, 8:53 PM

apparnetly

com.linkedin.pegasus2avro.dataprocess.DataProcessInfo

is also needed. I couldn't just use

dataProcessInfo

, in case you have doubt about it

bumpy-keyboard-50565

06/03/2020, 8:54 PM

Yeah that's expected. It needs to be fully qualified type unless the namespace matches the default namespace of the schema

2 Views

Open in Slack

Previous Next