Hi I am trying to upgrade our datahub server to the new rele DataHub #troubleshoot

Hi I am trying to upgrade our datahub server to th...

most-nightfall-36645

08/04/2022, 9:59 AM

Hi I am trying to upgrade our datahub server to the new release

v0.8.42

from

v0.8.41

. We have deployed datahub using the k8s helm chart. During the upgrade all jobs succeeed and the front-end pod is replaced. However the gms pod is failing to start with following error:

Copy code

ERROR: No such classes directory file:///etc/datahub/plugins/auth/resources
Usage: java [-Djetty.home=dir] -jar jetty-runner.jar [--help|--version] [ server opts] [[ context opts] context ...]
Server opts:
 --version                           - display version and exit
 --log file                          - request log filename (with optional 'yyyy_mm_dd' wildcard
 --out file                          - info/warn/debug log filename (with optional 'yyyy_mm_dd' wildcard
 --host name|ip                      - interface to listen on (default is all interfaces)
 --port n                            - port to listen on (default 8080)
 --stop-port n                       - port to listen for stop command (or -DSTOP.PORT=n)
 --stop-key n                        - security string for stop command (required if --stop-port is present) (or -DSTOP.KEY=n)
 [--jar file]*n                      - each tuple specifies an extra jar to be added to the classloader
 [--lib dir]*n                       - each tuple specifies an extra directory of jars to be added to the classloader
 [--classes dir]*n                   - each tuple specifies an extra directory of classes to be added to the classloader
 --stats [unsecure|realm.properties] - enable stats gathering servlet context
 [--config file]*n                   - each tuple specifies the name of a jetty xml config file to apply (in the order defined)
Context opts:
 [[--path /path] context]*n          - WAR file, web app dir or context xml file, optionally with a context path
2022/08/04 09:53:25 Command exited with error: exit status 1

In short it seems like the jar is missing some classes? I checked out the pod manifests and we are using the

linkedin/datahub-gms:v0.8.42

container image.

plus1 1

incalculable-ocean-74010

08/04/2022, 10:32 AM

Hello Dominic, We are aware of this bug, which got fixed in this PR: https://github.com/datahub-project/datahub/pull/5554

square-activity-64562

08/04/2022, 10:32 AM

@incalculable-ocean-74010 I think that only fixed it for quickstart not kubernetes

incalculable-ocean-74010

08/04/2022, 10:33 AM

I see… in that case it is a known bug 😞

incalculable-ocean-74010

08/04/2022, 10:33 AM

Sorry Dominic, let me try and repro this in a minikube instance

most-nightfall-36645

08/04/2022, 10:34 AM

No problem, thanks for your help 🙂

incalculable-ocean-74010

08/04/2022, 2:37 PM

Hey Dominic, Can you try deploying GMS with this image tag: 2eef7e4 ? Thanks

most-nightfall-36645

08/04/2022, 3:28 PM

Still got the same error on image

linkedin/datahub-gms:2eef7e4

Copy code

ERROR: No such classes directory file:///etc/datahub/plugins/auth/resources

incalculable-ocean-74010

08/04/2022, 3:31 PM

cc @square-activity-64562

square-activity-64562

08/04/2022, 3:35 PM

@most-nightfall-36645 Can you please send the full stacktrace? Maybe there is some difference before that?

most-nightfall-36645

08/04/2022, 3:40 PM

Copy code

% kubectl logs -n datahub datahub-datahub-gms-<omitted>
+ echo
+ grep -q ://
+ NEO4J_HOST=http://
+ [[ ! -z '' ]]
+ [[ -z '' ]]
+ ELASTICSEARCH_AUTH_HEADER='Accept: */*'
+ [[ true == true ]]
+ ELASTICSEARCH_PROTOCOL=https
+ WAIT_FOR_EBEAN=
+ [[ '' != true ]]
+ [[ '' == ebean ]]
+ [[ -z '' ]]
+ WAIT_FOR_EBEAN=' -wait tcp://<omitted> '
+ WAIT_FOR_CASSANDRA=
+ [[ '' == cassandra ]]
+ WAIT_FOR_KAFKA=
+ [[ '' != true ]]
++ echo <omitted>
++ sed 's/,/ -wait tcp:\/\//g'
+ WAIT_FOR_KAFKA=' -wait <omitted> '
+ WAIT_FOR_NEO4J=
+ [[ elasticsearch != elasticsearch ]]
+ OTEL_AGENT=
+ [[ '' == true ]]
+ PROMETHEUS_AGENT=
+ [[ '' == true ]]
+ auth_resource_dir=/etc/datahub/plugins/auth/resources
+ mkdir -p /etc/datahub/plugins/auth/resources
mkdir: cannot create directory '/etc/datahub': Permission denied
+ COMMON='
     -wait tcp://<omitted> -wait tcp://<omitted>           -timeout 240s     java                 -jar /jetty-runner.jar     --jar jetty-util.jar     --jar jetty-jmx.jar --
classes /etc/datahub/plugins/auth/resources     --config /datahub/datahub-gms/scripts/jetty.xml     /datahub/datahub-gms/bin/war.war'
+ [[ '' != true ]]
+ exec dockerize -wait https://<omitted> -wait-http-header 'Accept: */*' -wait tcp://<omitted> -wait <omitted> -timeo ut 240s java -jar /jetty-runner.jar --jar jetty-util.jar --jar jetty-jmx.jar --classes /etc/datahub/plugins/auth/resources --config /datahub/datahub-gms/scripts/jetty.xml /datahub/datahub-gms/bin/war.war
2022/08/04 15:26:56 Waiting for: <omitted>
2022/08/04 15:26:56 Waiting for: <omitted>
2022/08/04 15:26:56 Waiting for: <omitted>
2022/08/04 15:26:56 Connected to <omitted>
2022/08/04 15:26:56 Connected to <omitted>
2022/08/04 15:26:56 Received 200 from <omitted>
2022-08-04 15:26:56.618:INFO::main: Logging initialized @195ms to org.eclipse.jetty.util.log.StdErrLog
WARNING: jetty-runner is deprecated.
         See Jetty Documentation for startup options
         <https://www.eclipse.org/jetty/documentation/>
ERROR: No such classes directory file:///etc/datahub/plugins/auth/resources
Usage: java [-Djetty.home=dir] -jar jetty-runner.jar [--help|--version] [ server opts] [[ context opts] context ...]
Server opts:
 --version                           - display version and exit
 --log file                          - request log filename (with optional 'yyyy_mm_dd' wildcard
 --out file                          - info/warn/debug log filename (with optional 'yyyy_mm_dd' wildcard
 --host name|ip                      - interface to listen on (default is all interfaces)
 --port n                            - port to listen on (default 8080)
 --stop-port n                       - port to listen for stop command (or -DSTOP.PORT=n)
 --stop-key n                        - security string for stop command (required if --stop-port is present) (or -DSTOP.KEY=n)
 [--jar file]*n                      - each tuple specifies an extra jar to be added to the classloader
 [--lib dir]*n                       - each tuple specifies an extra directory of jars to be added to the classloader
 [--classes dir]*n                   - each tuple specifies an extra directory of classes to be added to the classloader
 --stats [unsecure|realm.properties] - enable stats gathering servlet context
 [--config file]*n                   - each tuple specifies the name of a jetty xml config file to apply (in the order defined)
Context opts:
 [[--path /path] context]*n          - WAR file, web app dir or context xml file, optionally with a context path

square-activity-64562

08/04/2022, 3:41 PM

Seems this did not work but worked in quickstart

mkdir: cannot create directory '/etc/datahub': Permission denied

square-activity-64562

08/04/2022, 3:41 PM

Let me check

square-activity-64562

08/04/2022, 4:56 PM

@most-nightfall-36645 Can you attempt it with GMS tag

4f35a6c

incalculable-ocean-74010

08/04/2022, 5:19 PM

The folder exists now so it should be good

most-nightfall-36645

08/04/2022, 5:45 PM

ill give it a go.

most-nightfall-36645

08/04/2022, 5:55 PM

Looks like it works. Thanks for your help, ill watch out for the next realease

incalculable-ocean-74010

08/04/2022, 11:38 PM

Thank you Dominic, I will release a bug fix version in the coming days.

2 Views

Open in Slack

Previous Next