Hi <@U01GCJKA8P9> and <@U01TCN40JKV> and datahub t...
# ingestion
b
Hi @big-carpet-38439 and @better-orange-49102 and datahub team, @tall-butcher-30509 I have tried to ingest the below string to dataset description but i could nt do it for unknown reason. But if i try modifying from the datahub UI i can able to do that. Can you guys help me on this issue. I have faced the similar issue while ingesting the description of dataset columns. Failed description:
Copy code
{"tbl_lgcl_name_eng":"Stock Header","tbl_lgcl_name_lcl":"ストックヘッダ","tbl_desc":"entity name:Stock Header\\nlayer:cleansed\\nnote:Cleansed table of loaded.t_r_oiv_stk_hdr_jp\\n\\n#interface_logical_name_english:Stock Information Raw Data"}}
metadatachangeproposalwrapper to string:
Copy code
MetadataChangeProposalWrapper(entityType=dataset, entityUrn=urn:li:dataset:(urn:li:dataPlatform:bigquery,dataset.bq.datalake.stg_dataset.table,STG), changeType=UPSERT, aspect={externalUrl="", customProperties={created_time=2022-02-24 10:47:08.296, created_by=event_driven}, description={"tbl_lgcl_name_eng":"Stock Header","tbl_lgcl_name_lcl":"ストックヘッダ","tbl_desc":"entity name:Stock Header\\nlayer:cleansed\\nnote:Cleansed table of loaded.t_r_oiv_stk_hdr_jp\\n\\n#interface_logical_name_english:Stock Information Raw Data"}}, aspectName=datasetProperties)
b
just to isolate the cause, does it work if its pure english characters? I see some japanese characters inside
t
@bitter-dusk-52400 In addition to above, please confirm the encoding setting of the response from BQ and the request to datahub. I’m guessing BQ res. should be in UTF8 anyway. Request to datahub should also be.
b
@better-orange-49102 For pure english ingestion its working.
b
Yes are we sure encoding is UTF-8?
In the payload
I’m presuming you’re using Python Ingestion by itself directly from BQ
t
We are using Java
b
Let me check confirm, which encoding type does datahub accepts
@big-carpet-38439 Encoding setting is UTF-8.
@big-carpet-38439 Any idea on this problem