https://datahubproject.io logo
Join Slack
Powered by
Hey everyone. I have started up DataHub 0.8.27 bot...
# troubleshoot
a
Hey everyone. I have started up DataHub 0.8.27 both with the Docker QuickStart and on Minikube, and with both I am getting an error when executing a Snowflake ingestion source. Recipe and logs are in the reply thread.
Recipe:
Copy code
source:
    type: snowflake
    config:
        host_port: [REDACTED]
        warehouse: [REDACTED]
        username: [REDACTED]
        password: [REDACTED]
sink:
    type: datahub-rest
    config:
        server: '<http://datahub-gms:8080>'
It seems like a bug, that something is trying to connect to '/config' as a host
I also tried this on 0.8.26 and had the same result.
s
Hello, try server localhost:8080 if you ingest from commandline, for me that works, and datahub-gms:8080 works if you use the GUI
b
@adamant-kilobyte-90981 Hmm are you able to access localhost:8080/config?
We are running this on helm. Let me check our configs
Yeah as far as I can tell this is work on ours... And yes we do hit the /config from the ingestion code to verify some information about the server version - so this part is expected
What's not is that we are failing to find that host... Is the actions pod deployed on the same docker network as the datahub-gms container? If not, then this won't work
In the docker quickstart, it should be by default
Is the error the same when running using Docker Quickstart? If yes then that's very concerning
Yes so this error seems to be docker network related. Questions: 1. You are running 
datahub docker quickstart
with no modifications? 2. You have not changed the default port of datahub-gms (8080)?
a
Thanks a bunch @big-carpet-38439 for pointing me in the right direction. It turns out that it was a hostname issue on both Docker QuickStart and K8s: 1. The docs seem to say the port on localhost in Docker is 9002 but it's actually 8080 2. The docs for K8s say the hostname for gms is 
datahub-gms
but it's actually 
datahub-datahub-gms
Changing to the right hostname fixed this error, and moved me on to the next error 😄
b
Chris - please link out the docs you were following and we will promptly update.. Want to prevent others from running into this. What is the issue you are currently facing? For snowflake, we most often see these issues: 1. host_port config set to a full URL like "company.aws.us-west-2.snowflakecomputing.com" - It should be just the first part - "company" 2. Privileges - The user executing ingestion must have the requisite privileges for all tables in the warehouse (should be detailed in Snowflake source doc)
a
I sorted out the other errors, one of which was the first one you mentioned about host_port. Let me grab a couple of links and screenshots of the URLs that didn't work out

https://datahubproject.io/assets/images/example-mysql-recipe-09bb4120c603b5476bc0d5171a2b4e44.pngâ–¾

That's where I got the 
<http://datahub-gms:8080>
address from on https://datahubproject.io/docs/ui-ingestion
And "http://localhost:9002/api/gms" is what shows up in the recipe by default
Since it had localhost:9002 in it and that's how the Docker QuickStart worked, I assumed the gms url had been correctly populated automatically
b
Got it - yes so we do need to update the defaults.. The "http://localhost:9002/api/gms" works if you've enabled Metadata Service Auth, which requires that all reqs to the backend are secured by a token.. but for quickstart we still do not enable this by default, to make testing DataHub easier (even though in this case that doesn't seem to be making things easier 😛) More information here: https://blog.datahubproject.io/tech-deep-dive-introducing-datahub-metadata-service-authentication-661e3aabbad0
Open in Slack
PreviousNext
Powered by