Hi guys, I ingested some test data, but UI, I get ...
# ui
b
Hi guys, I ingested some test data, but UI, I get this when trying to view them:
b
Hey Moses! So all of the datasets you've ingested must include a valid "data platform" reference. The valid options lives in the files
DataPlatformInfo.json
... Basically just need to make sure the ingested data's "platform" field matches one of the entries in DataPlatformInfo.json
Unfortunately this validation doesn't today exist on the ingest side 😞 But we do validate on read
b
@big-carpet-38439 Thanks for the feedback, my source is a postgres DB and I am sinking to the datahub-rest, where should I declare the 'platform' field
From what i can see in the
DataPlatformInfo.json
file, postgres is included. And my recipe looks like this:
Copy code
source:
  type: postgres
  config:
    username: <user>
    password: <pw>
    host_port: <host:port>
    database: <dbname>

sink:
  type: "datahub-rest"
  config:
    server: 'server:port'
@big-carpet-38439 I wonder how I can fix this, or am I referencing in a wrong manner?
This is a bit strange because retrieving a dataset via curl seems to work fine:
Copy code
curl -H 'X-RestLi-Protocol-Version:2.0.0' -H 'X-RestLi-Method: get' '<http://host>:port/datasets/($params:(),name:gama-pg-db.public.ken_roads,origin:PROD,platform:urn%3Ali%3AdataPlatform%3Apostgresql)' | jq
b
looking into this more
i think this hs to do with the fact that the platform name you show above^ is "postgresql"
must need a change in the ingestion flow
b
@big-carpet-38439 sorry I dint catch exactly what you mean by this - change in the ingesion flow. Do I have to modify the DataPlatformInfo.json, because i dont know where it got the type 'postgresql', I was just testing and discovered that if I use 'postgresql' for platform name, it works, however, if i use 'postgres' it doesnt work.
b
Hi Moses. I've put a PR up to try to address the UI issue: https://github.com/linkedin/datahub/pull/2185 Can you pull the latest? The type should be set to "postgres" as shown in gms/impl/src/main/resources/DataPlatformInfo.json:91
b
@big-carpet-38439 Works like charm, thanks!
b
Wonderful to hear Moses, sorry about that 😛
Please do let me know if you run into any other trouble / have questions
b
Will do, great work @big-carpet-38439!!!
🙏 1