https://datahubproject.io logo
Join Slack
Powered by
Hi all! just a quick question... I'm running `data...
# getting-started
n
Hi all! just a quick question... I'm running 
datahub
locally (using docker images) and I ran a ingestion to load some metadata from postgres... everything seem to be executed properly... no failures or warnings in the sink, plus about 116 records written. But I cannot see any dataset in the frontend. Do I need to run something else beyond 
datahub ingest run
? I'm using the default 
datahub
user, so I don't know how this user is related to the data ingested...
b
could you share more about the recipe that you used? just wondering if you wrote to console or file sink instead ie datahub ingest -c <recipe>.yml
n
Copy code
source:
  type: postgres
  config:
    # Coordinates
    host_port: <....>.<http://rds.amazonaws.com:5432|rds.amazonaws.com:5432>
    database: mydb

    # Credentials
    username: myuser
    password: <....>

    # Options
    database_alias: mydb
    env: PROD
sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:8080>"
thanks xL for your reply!
this is how the process ended
however, I'm seeing a bunch of warnings in the source part related to 
unable to map type UUID() to metadata schema
we have several primary keys defined as uuid
b
hmm you're writing to datahub from the looks of it. since u cant see in the UI, i would try to peer into the mySQL DB and see if there is the datasets inside as the next step? using something like
Copy code
docker exec -it mysql /bin/sh
mysql -u datahub -h localhost -p
or u could try looking into the ES index and see if there are any records. though frankly if there are records ingested inside, i've no idea whats causing the lack of datasets in the UI. maybe some rendering bug?
n
well, I only see two tables in 
databub
database... 
metadata_aspect_v2
with 577 records and 
metadata_index
with 0 records... maybe I need to run some index process to fill that table..
and I can confirm that some of the records in 
metadata_aspect_v2
are actually the metadata from my postgres database
I found this issue in Github https://github.com/linkedin/datahub/issues/2355 I've increased Docker memory to 11 Gb + 2 Gb for swap, but still no luck
b
It is normal for Metadata-index to be empty. Metadata_aspect_v2 is the table that keeps the records.
@early-lamp-41924 perhaps you guys can help him instead?
e
Can you try searching “*” in the search bar? That searches for all entities.
This is very weird. Can we also check whether elasticsearch is running okay?
n
well, now it's working! After Dexter recommendation I took a look at the 
elasticsearch
logs and I found that indexes were read only due to the lack of space in docker's disk configuration... so I increase it and now ingestion worked properly! thank you so much!
e
Glad to hear!!
Open in Slack
PreviousNext
Powered by