https://datahubproject.io logo
Join Slack
Powered by
:wave: Hi team! I have a problem with `dbt` ingest...
# troubleshoot
s
👋 Hi team! I have a problem with 
dbt
ingestion. I was running the ingestion and the cronjob seems ok, but I cannot visualize 
dbt
platform in UI. Attaching the output log of the ingestion job. And this is the configuration
Copy code
source:
  type: "dbt"
  config:
    # Coordinates
    manifest_path: "<s3://bucket_name/manifest.json>"
    catalog_path: "<s3://bucket_name/catalog.json>"
    sources_path: "<s3://bucket_name/sources.json>"

    aws_connection:
      aws_region: "eu-west-2"

    # Options
    target_platform: "snowflake"
    load_schemas: True
    env: STG
    node_name_pattern:
      allow:
        - ".*branch_activities.*"
      deny:
        - ".*test.*"

sink:
  type: "datahub-rest"
  config:
    server: "<http://datahub-datahub-gms:8080>"
Any clue? Thanks!
output.txt
c
@salmon-area-51650 by visualize you are referring to view urn in ui ?
can you check in db if the entry exists? with urn in logs
if so then there might connection error with db from frontend service
s
I can see entries for 
dbt
in mysql but not in UI. I’d expect something like this. I think that it’s not a database problem as I can see the rest of the platforms
c
hmm..., can try opening in incognito mode and can you share logs for frontend after dbt ingestion?
s
There is not logs in frontend after ingestion but I can see that the list of the platforms don’t contain `dbt`(See screenshot)
c
so when you refresh there must be some log ensuring your are hitting api, or session management, as you have attached the screenshot, these api are getting hit so there must exists
s
so there must exists
But there is not anything related to dbt in the response as you can see in the screenshot. Do you know what is the query executed? I can try to executed manually in the database to know why it’s not returning dbt
c
there is one table fire something like this select * from table where id = 'urnlidataPlatform:dbt'
s
graphQl is returning the platform as well…. I don’t understand why the ui is not showing the platform
c
try searching dbt
s
Empty
c
seems ui is unable to connect with es and db and returning cached data
can you try restarting ui pod/ container
new connection will get setup
datahub-frontend service
s
done, but the same result 😞
There are a lot of entries for dbt datasets
b
hey Miguel! so are you seeing the datasets in the UI, but just not listed under dbt? or are they not showing up at all?
s
it’s not showing up at all 😞. I can see these datasets only in the database
b
okay gotcha well that's a problem.. let me ask around and get back to you soon
s
ok thanks!!!
b
okay so this is not related to another issue I thought it might be. what happens if you try to go to a profile for one of those datasets by pasting in the urn to 
/dataset/<urn>
?
s
yeah I can access directly to the dataset. Then why can’t I see dbt platform in the main page? It’s weird 🤔
b
okay good so we know we can get to them, but the real problem is that dbt isn't showing up as a platform option on the home page. this might actually be related to another issue now that I understand this one a bit better.. i'll get back to you soon!
thankyou1 1
Open in Slack
PreviousNext
Powered by