https://datahubproject.io logo
Join Slack
Powered by
Hi! I'm new to DataHub and trying to deploy it us...
# troubleshoot
f
Hi! I'm new to DataHub and trying to deploy it using the recommended K8S deployment spec. The deployed services do not match the ones as specified in the spec. I would appreciate it someone can help with this. Here's a list of my deployed services.
Copy code
NAME                                               READY   STATUS      RESTARTS   AGE
datahub-acryl-datahub-actions-74c674fb9d-rm5rq     1/1     Running     0          6h17m
datahub-datahub-frontend-bd5c8677c-84nrf           1/1     Running     0          6h17m
datahub-datahub-gms-54f994fdf5-vgjzc               1/1     Running     0          6h17m
datahub-datahub-upgrade-job-rsln9                  0/1     Error       0          6h17m
datahub-datahub-upgrade-job-zx2sk                  0/1     Completed   0          6h16m
datahub-elasticsearch-setup-job-t59k2              0/1     Completed   0          6h18m
datahub-kafka-setup-job-zxsjc                      0/1     Completed   0          6h18m
datahub-mysql-setup-job-7glgk                      0/1     Completed   0          6h17m
elasticsearch-master-0                             1/1     Running     0          6h21m
elasticsearch-master-1                             1/1     Running     0          6h21m
elasticsearch-master-2                             1/1     Running     0          6h21m
prerequisites-cp-schema-registry-cf79bfccf-mx25m   2/2     Running     0          6h21m
prerequisites-kafka-0                              1/1     Running     1          6h21m
prerequisites-mysql-0                              1/1     Running     0          6h21m
prerequisites-neo4j-community-0                    1/1     Running     0          6h21m
prerequisites-zookeeper-0                          1/1     Running     0          6h21m
ubuntu                                             1/1     Running     0          71m
It seems like the upgrade tasks failed and caused the gms consumers to fail?
I was able to bring up the console and created a Kafka data source, but it failed to consume.
b
Can you extract the logs from the datahub-gms pod?
f
Not sure where the logs are kept. Looked at the usual location (/var/log) and its empty
Copy code
bash-5.1$ pwd
/var/log
bash-5.1$ ls -la
total 0
drwxr-xr-x 2 root root  6 Apr  4 16:07 .
drwxr-xr-x 1 root root 19 Apr  4 16:07 ..
bash-5.1$
b
f
b
oh wow - I've not seen this yet. And you are indeed able to log into DataHub?
But no ingestion?
f
yes, though, I didn't know the actual user/pwd... I used datahub/datahub and the console mentioned that it's invalid, but let me in regardless
From there I tried to create a Kafka data source and got similar except as you seen in the log
b
datahub/datahub should work - it should not say invalid
can you share your kafka recipe configuration?
the fact that you can get in and look around means it is at minimum partially responsive
f
What's the correct user/pwd to the console
b
it is datahub/datahub - you used the correct one that we launch with
f
here's the ingestion data log
b
Okay - can you try redirecting your requests to this url (sink block)
Copy code
<http://datahub-datahub-gms:8080>
It's unable to connect to your GMS host, most likely because of the Helm DNS
f
trying
Still failed
b
Same reason/
?
Or different
f
seems to be different this time
Does DataHub consumer support SASL? Based on popup diagram, it seems to be the case.
b
To ingest from kafka you need to provide the "schema registry url" parameter. I'm seeing this in the log:
Copy code
'1 validation error for KafkaSourceConfig\n'
           'connection -> schema_registry_url\n'
           '  none is not an allowed value (type=type_error.none.not_allowed)\n',
That is a required config 🙂 Source docs don't mention this explicitly, but they should
f
I see. I don't have a schema registry
b
oh - so that's what we are extracting in this source - the schemas for your topics
f
Is another a way to provide topic metadata information without source and sink?
b
You can emit metadata directly using our Python / Java Metadata Emitter SDKs!
Open in Slack
PreviousNext
Powered by