https://datahubproject.io logo
Join Slack
Powered by
Hi, I am trying to test out the `datahub-rest` si...
# getting-started
p
Hi, I am trying to test out the 
datahub-rest
sink using the docker setup. I am using the 
datahub ingest -c demo.yml
command and can run the job with the 
file
based sink, but with 
datahub-rest
sink, I get 
'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x13d2ba940>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /config
Any suggestions on what should I do to debug this issue?
e
Hi @purple-tailor-57675 would you mind sharing your recipe?
p
Hi Aditya, I was able to make it work from the CLI, but from the datahub console, I am seeing some errors.
Copy code
source:
    type: snowflake
    config:
        check_role_grants: true
        ignore_start_time_lineage: true
        account_id: <account_id>
        warehouse: <warehouse>
        username: <username>
        password: <password>
        role: <role>
        database_pattern:
            allow:
                - DEV_DWH
        schema_pattern:
            deny:
                - 'information_schema.*'
            allow:
                - DWH
        profiling:
            enabled: true
        profile_pattern:
            deny:
                - '.*information_schema.*'

sink: 
    type: datahub-rest 
    config: 
        server: "<http://localhost:9002/api/gms>"
This is when I add the source from the datahub UI
e
Ah I see the UI ingestion isn’t working properly?
p
Yes, that's correct. Also from the above logs, it seems that the dependencies are being installed again, but I had installed all the required dependencies locally from CLI, so wanted to confirm about that issue as well.
e
I suspect your GMS server isn’t being configured correctly
could you change the server field to 
<http://localhost:8080>
? I’m assuming you’re running this locally
p
Yes, using docker
Failed again
logslocalhost.txt
e
can you quickly verify that the 
datahub-gms
docker pod is running on 
localhost:8080
? or is it a different url?
p
Copy code
5d038dcd4c05   linkedin/datahub-gms:head                "/bin/sh -c /datahub…"   3 hours ago   Up 3 hours (healthy)   0.0.0.0:8080->8080/tcp                             datahub-gms
seems to working on 
localhost:8080
e
got it that’s good to know
and you said the same recipe seems to be working locally?
p
I think I got it working using the link 
<http://datahub-gms:8080>
from the UI. All my previous attempts failed in less than 60 secs and this one is has crossed that
Yes, it works locally with the 
localhost
url
quick questions - 1. Why does it need different server urls between the UI and local as all the containers are working locally? 2. Why does each execution of the recipe downloads all the dependencies from scratch? 3. Why are the 
queries
, 
stats
and 
validation
options menu options are not enabled, even though I am using the 
datahub
user to login to the UI?
e
1. Let me get back to you on this one 2. UI based ingestion uses the 
datahub-actions
container which will create it’s own python virtual environment to run UI based ingestion 3. If those options are not enabled, it may mean that usage statistics were not collected. Have you run the 
snowflake-usage
plugin as well? If not, queries and stats will not show up, unfortunately
p
I haven't executed the 
snowflake-usage
plugin, will give it a try now.
Thanks for unblocking me Aditya 🙂
e
Happy to help!
2 Views
Open in Slack
PreviousNext
Powered by