https://datahubproject.io logo
Join Slack
Powered by
Hello, After installing the latest version of dat...
# troubleshoot
m
Hello, After installing the latest version of datahub and trying to ingest data from an Oracle DB, I realized that I don't have read access to metadata tables (dba_*). After searching in this forum, I found out that it used to be possible to ingest data without access to dba tables until version 0.8.43. So I am now trying to run this version. I nuked the existing datahub containers, created a new Python virtual environment in which I installed datahub 0.8.43, and I run the program with the following commands: 
datahub docker nuke
python -m pip install --upgrade pip wheel setuptools
python -m pip install acryl-datahub==0.8.43
datahub version
DataHub CLI version: 0.8.43 Python version: 3.7.15 (default, Nov 24 2022, 184454) [MSC v.1916 64 bit (AMD64)] 
datahub docker quickstart --version v0.8.43 --mysql-port 53306
But it fails to start after a few minutes, telling me that datahub-gms is not running. The logs for datahub-gms container are as follows. I don't know what the problem is (the latest version worked without any problem). 
2022-12-13 16:52:58 2022/12/13 07:52:58 Waiting for: <http://elasticsearch:9200>
2022-12-13 16:52:58 2022/12/13 07:52:58 Waiting for: <tcp://mysql:3306>
2022-12-13 16:52:58 2022/12/13 07:52:58 Waiting for: <tcp://broker:29092>
2022-12-13 16:52:58 2022/12/13 07:52:58 Problem with dial: dial tcp 172.28.0.5:29092: connect: connection refused. Sleeping 1s
2022-12-13 16:52:58 2022/12/13 07:52:58 Connected to <tcp://mysql:3306>
2022-12-13 16:52:58 2022/12/13 07:52:58 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.28.0.3:9200: connect: connection refused. Sleeping 1s
2022-12-13 16:52:59 2022/12/13 07:52:59 Problem with dial: dial tcp 172.28.0.5:29092: connect: connection refused. Sleeping 1s
2022-12-13 16:52:59 2022/12/13 07:52:59 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.28.0.3:9200: connect: connection refused. Sleeping 1s
2022-12-13 16:53:00 2022/12/13 07:53:00 Problem with dial: dial tcp 172.28.0.5:29092: connect: connection refused. Sleeping 1s
2022-12-13 16:53:00 2022/12/13 07:53:00 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.28.0.3:9200: connect: connection refused. Sleeping 1s
2022-12-13 16:53:01 2022/12/13 07:53:01 Problem with dial: dial tcp 172.28.0.5:29092: connect: connection refused. Sleeping 1s
2022-12-13 16:53:01 2022/12/13 07:53:01 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.28.0.3:9200: connect: connection refused. Sleeping 1s
2022-12-13 16:53:02 2022/12/13 07:53:02 Problem with dial: dial tcp 172.28.0.5:29092: connect: connection refused. Sleeping 1s
2022-12-13 16:53:02 2022/12/13 07:53:02 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.28.0.3:9200: connect: connection refused. Sleeping 1s
2022-12-13 16:53:03 2022/12/13 07:53:03 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.28.0.3:9200: connect: connection refused. Sleeping 1s
2022-12-13 16:53:03 2022/12/13 07:53:03 Problem with dial: dial tcp 172.28.0.5:29092: connect: connection refused. Sleeping 1s
2022-12-13 16:53:04 2022/12/13 07:53:04 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.28.0.3:9200: connect: connection refused. Sleeping 1s
2022-12-13 16:53:04 2022/12/13 07:53:04 Problem with dial: dial tcp 172.28.0.5:29092: connect: connection refused. Sleeping 1s
2022-12-13 16:53:05 2022/12/13 07:53:05 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.28.0.3:9200: connect: connection refused. Sleeping 1s
2022-12-13 16:53:05 2022/12/13 07:53:05 Connected to <tcp://broker:29092>
2022-12-13 16:53:06 2022/12/13 07:53:06 Problem with request: Get "<http://elasticsearch:9200>": dial tcp 172.28.0.3:9200: connect: connection refused. Sleeping 1s
2022-12-13 16:53:07 2022/12/13 07:53:07 Received 200 from <http://elasticsearch:9200>
2022-12-13 16:53:08 2022-12-13 07:53:08.265:INFO::main: Logging initialized @277ms to org.eclipse.jetty.util.log.StdErrLog
2022-12-13 16:53:08 WARNING: jetty-runner is deprecated.
2022-12-13 16:53:08          See Jetty Documentation for startup options
2022-12-13 16:53:08          <https://www.eclipse.org/jetty/documentation/>
2022-12-13 16:53:08 ERROR: No such classes directory file:///etc/datahub/plugins/auth/resources
2022-12-13 16:53:08 Usage: java [-Djetty.home=dir] -jar jetty-runner.jar [--help|--version] [ server opts] [[ context opts] context ...]
2022-12-13 16:53:08 Server opts:
2022-12-13 16:53:08  --version                           - display version and exit
2022-12-13 16:53:08  --log file                          - request log filename (with optional 'yyyy_mm_dd' wildcard
2022-12-13 16:53:08  --out file                          - info/warn/debug log filename (with optional 'yyyy_mm_dd' wildcard
2022-12-13 16:53:08  --host name|ip                      - interface to listen on (default is all interfaces)
2022-12-13 16:53:08  --port n                            - port to listen on (default 8080)
2022-12-13 16:53:08  --stop-port n                       - port to listen for stop command (or -DSTOP.PORT=n)
2022-12-13 16:53:08  --stop-key n                        - security string for stop command (required if --stop-port is present) (or -DSTOP.KEY=n)
2022-12-13 16:53:08  [--jar file]*n                      - each tuple specifies an extra jar to be added to the classloader
2022-12-13 16:53:08  [--lib dir]*n                       - each tuple specifies an extra directory of jars to be added to the classloader
2022-12-13 16:53:08  [--classes dir]*n                   - each tuple specifies an extra directory of classes to be added to the classloader
2022-12-13 16:53:08  --stats [unsecure|realm.properties] - enable stats gathering servlet context
2022-12-13 16:53:08  [--config file]*n                   - each tuple specifies the name of a jetty xml config file to apply (in the order defined)
2022-12-13 16:53:08 Context opts:
2022-12-13 16:53:08  [[--path /path] context]*n          - WAR file, web app dir or context xml file, optionally with a context path
2022-12-13 16:53:08 2022/12/13 07:53:08 Command exited with error: exit status 1
2022-12-13 16:53:27 + echo
2022-12-13 16:53:27 + grep -q ://
2022-12-13 16:53:27 + NEO4J_HOST=http://
2022-12-13 16:53:27 + [[ ! -z '' ]]
2022-12-13 16:53:27 + [[ -z '' ]]
2022-12-13 16:53:27 + ELASTICSEARCH_AUTH_HEADER='Accept: */*'
2022-12-13 16:53:27 + [[ '' == true ]]
2022-12-13 16:53:27 + ELASTICSEARCH_PROTOCOL=http
2022-12-13 16:53:27 + WAIT_FOR_EBEAN=
2022-12-13 16:53:27 + [[ '' != true ]]
2022-12-13 16:53:27 + [[ '' == ebean ]]
2022-12-13 16:53:27 + [[ -z '' ]]
2022-12-13 16:53:27 + WAIT_FOR_EBEAN=' -wait <tcp://mysql:3306> '
2022-12-13 16:53:27 + WAIT_FOR_CASSANDRA=
2022-12-13 16:53:27 + [[ '' == cassandra ]]
2022-12-13 16:53:27 + WAIT_FOR_KAFKA=
2022-12-13 16:53:27 + [[ '' != true ]]
2022-12-13 16:53:27 ++ echo broker:29092
2022-12-13 16:53:27 ++ sed 's/,/ -wait tcp:\/\//g'
2022-12-13 16:53:27 + WAIT_FOR_KAFKA=' -wait <tcp://broker:29092> '
2022-12-13 16:53:27 + WAIT_FOR_NEO4J=
2022-12-13 16:53:27 + [[ elasticsearch != elasticsearch ]]
2022-12-13 16:53:27 + OTEL_AGENT=
2022-12-13 16:53:27 + [[ '' == true ]]
2022-12-13 16:53:27 + PROMETHEUS_AGENT=
2022-12-13 16:53:27 + [[ '' == true ]]
2022-12-13 16:53:27 + auth_resource_dir=/etc/datahub/plugins/auth/resources
2022-12-13 16:53:27 + COMMON='
2022-12-13 16:53:27      -wait <tcp://mysql:3306>            -wait <tcp://broker:29092>           -timeout 240s     java -Xms1g -Xmx1g                -jar /jetty-runner.jar     --jar jetty-util.jar     --jar jetty-jmx.jar --classes /etc/datahub/plugins/auth/resources     --config /datahub/datahub-gms/scripts/jetty.xml     /datahub/datahub-gms/bin/war.war'
2022-12-13 16:53:27 + [[ '' != true ]]
2022-12-13 16:53:27 + exec dockerize -wait <http://elasticsearch:9200> -wait-http-header 'Accept: */*' -wait <tcp://mysql:3306> -wait <tcp://broker:29092> -timeout 240s java -Xms1g -Xmx1g -jar /jetty-runner.jar --jar jetty-util.jar --jar jetty-jmx.jar --classes /etc/datahub/plugins/auth/resources --config /datahub/datahub-gms/scripts/jetty.xml /datahub/datahub-gms/bin/war.war
2022-12-13 16:53:27 2022/12/13 07:53:27 Waiting for: <http://elasticsearch:9200>
2022-12-13 16:53:27 2022/12/13 07:53:27 Waiting for: <tcp://mysql:3306>
2022-12-13 16:53:27 2022/12/13 07:53:27 Waiting for: <tcp://broker:29092>
2022-12-13 16:53:27 2022/12/13 07:53:27 Connected to <tcp://mysql:3306>
2022-12-13 16:53:27 2022/12/13 07:53:27 Connected to <tcp://broker:29092>
2022-12-13 16:53:27 2022/12/13 07:53:27 Received 200 from <http://elasticsearch:9200>
2022-12-13 16:53:27 2022-12-13 07:53:27.370:INFO::main: Logging initialized @224ms to org.eclipse.jetty.util.log.StdErrLog
2022-12-13 16:53:27 WARNING: jetty-runner is deprecated.
2022-12-13 16:53:27          See Jetty Documentation for startup options
2022-12-13 16:53:27          <https://www.eclipse.org/jetty/documentation/>
2022-12-13 16:53:27 ERROR: No such classes directory file:///etc/datahub/plugins/auth/resources
2022-12-13 16:53:27 Usage: java [-Djetty.home=dir] -jar jetty-runner.jar [--help|--version] [ server opts] [[ context opts] context ...]
2022-12-13 16:53:27 Server opts:
2022-12-13 16:53:27  --version                           - display version and exit
2022-12-13 16:53:27  --log file                          - request log filename (with optional 'yyyy_mm_dd' wildcard
2022-12-13 16:53:27  --out file                          - info/warn/debug log filename (with optional 'yyyy_mm_dd' wildcard
2022-12-13 16:53:27  --host name|ip                      - interface to listen on (default is all interfaces)
2022-12-13 16:53:27  --port n                            - port to listen on (default 8080)
2022-12-13 16:53:27  --stop-port n                       - port to listen for stop command (or -DSTOP.PORT=n)
2022-12-13 16:53:27  --stop-key n                        - security string for stop command (required if --stop-port is present) (or -DSTOP.KEY=n)
2022-12-13 16:53:27  [--jar file]*n                      - each tuple specifies an extra jar to be added to the classloader
2022-12-13 16:53:27  [--lib dir]*n                       - each tuple specifies an extra directory of jars to be added to the classloader
2022-12-13 16:53:27  [--classes dir]*n                   - each tuple specifies an extra directory of classes to be added to the classloader
2022-12-13 16:53:27  --stats [unsecure|realm.properties] - enable stats gathering servlet context
2022-12-13 16:53:27  [--config file]*n                   - each tuple specifies the name of a jetty xml config file to apply (in the order defined)
2022-12-13 16:53:27 Context opts:
2022-12-13 16:53:27  [[--path /path] context]*n          - WAR file, web app dir or context xml file, optionally with a context path
2022-12-13 16:53:27 2022/12/13 07:53:27 Command exited with error: exit status 1
👀 1
b
Hey, • From the error you posted, I thinkg the problem is the a missing props file at 
${HOME}/.datahub/plugins/auth/resources/
which is addressable by running this command 
echo "datahub:datahub" >>  ~/.datahub/plugins/frontend/auth/user.props
• @dazzling-judge-80093 might have more context about the change on the oracle ingestion.
d
This error is not related to oracle source as gms fails to come up
b
Yes, that's unrelated, but the reason why Thierry had downgrade is because of the change in the ingestion.
d
@melodic-telephone-26568 can you grant access to dba_* tables? Yes, I can see it was changed recently to get metadata from those system tables.
m
@melodic-telephone-26568 you should be able to use the latest datahub quickstart ... along with an older datahub ingestion source version.
so you can just run the latest datahub quickstart like you were doing previously
and only run Oracle ingestion using a venv with 
acryl-datahub[oracle]==0.8.43
however, I'm curious why you aren't able to grant access to dba_* tables.
m
@mammoth-bear-12532 I just run the ingestion from a venv with version 0.8.43 and it worked, thank you. I will eventually request access to dba_* tables, but in my company every change requires a series of approvals, which takes days (with no guarantee of success), and I wanted to do a quick demo for my team.
9 Views
Open in Slack
PreviousNext
Powered by