Hi guys, I ingested some test data, but UI, I get ...
# uib
brief-toothbrush-55766
03/05/2021, 12:31 PMHi guys, I ingested some test data, but UI, I get this when trying to view them:
b
big-carpet-38439
03/05/2021, 3:44 PMHey Moses! So all of the datasets you've ingested must include a valid "data platform" reference. The valid options lives in the files
DataPlatformInfo.jsonbig-carpet-38439
03/05/2021, 3:44 PMUnfortunately this validation doesn't today exist on the ingest side 😞 But we do validate on read
b
brief-toothbrush-55766
03/06/2021, 11:21 AM@big-carpet-38439 Thanks for the feedback, my source is a postgres DB and I am sinking to the datahub-rest, where should I declare the 'platform' field
brief-toothbrush-55766
03/06/2021, 11:43 AMFrom what i can see in the
DataPlatformInfo.jsonbrief-toothbrush-55766
03/06/2021, 11:46 AM Copy code
source:
type: postgres
config:
username: <user>
password: <pw>
host_port: <host:port>
database: <dbname>
sink:
type: "datahub-rest"
config:
server: 'server:port'brief-toothbrush-55766
03/06/2021, 12:34 PM@big-carpet-38439 I wonder how I can fix this, or am I referencing in a wrong manner?
brief-toothbrush-55766
03/06/2021, 1:10 PMThis is a bit strange because retrieving a dataset via curl seems to work fine:
brief-toothbrush-55766
03/06/2021, 1:11 PM Copy code
curl -H 'X-RestLi-Protocol-Version:2.0.0' -H 'X-RestLi-Method: get' '<http://host>:port/datasets/($params:(),name:gama-pg-db.public.ken_roads,origin:PROD,platform:urn%3Ali%3AdataPlatform%3Apostgresql)' | jqb
big-carpet-38439
03/06/2021, 5:29 PMlooking into this more
big-carpet-38439
03/06/2021, 5:29 PMi think this hs to do with the fact that the platform name you show above^ is "postgresql"
big-carpet-38439
03/06/2021, 5:30 PMand in https://github.com/linkedin/datahub/blob/master/gms/impl/src/main/resources/DataPlatformInfo.json#L91 you can see it defined as "postgres"
big-carpet-38439
03/06/2021, 5:30 PMmust need a change in the ingestion flow
b
brief-toothbrush-55766
03/08/2021, 8:52 AM@big-carpet-38439 sorry I dint catch exactly what you mean by this - change in the ingesion flow. Do I have to modify the DataPlatformInfo.json, because i dont know where it got the type 'postgresql', I was just testing and discovered that if I use 'postgresql' for platform name, it works, however, if i use 'postgres' it doesnt work.
b
big-carpet-38439
03/08/2021, 1:02 PMHi Moses. I've put a PR up to try to address the UI issue: https://github.com/linkedin/datahub/pull/2185 Can you pull the latest? The type should be set to "postgres" as shown in gms/impl/src/main/resources/DataPlatformInfo.json:91
b
brief-toothbrush-55766
03/08/2021, 5:38 PM@big-carpet-38439 Works like charm, thanks!
b
big-carpet-38439
03/08/2021, 6:04 PMWonderful to hear Moses, sorry about that 😛
big-carpet-38439
03/08/2021, 6:04 PMPlease do let me know if you run into any other trouble / have questions
b
brief-toothbrush-55766
03/08/2021, 7:17 PMWill do, great work @big-carpet-38439!!!
🙏 1
2 Views