Hi all I m trying to locally upgrade the datahub from versio DataHub #troubleshoot

Hi all, I'm trying to locally upgrade the datahub ...

few-sunset-43876

11/06/2022, 3:56 PM

Hi all, I'm trying to locally upgrade the datahub from version v0.8.42 to the version v0.9.1 using nocode upgrade docker/datahub-upgrade/nocode/run_upgrade.sh But I got the error and the process was aborted.

Copy code

Starting upgrade with id NoCodeDataMigration...
Cleanup has not been requested.
Skipping Step 1/6: RemoveAspectV2TableStep...
Executing Step 2/6: GMSQualificationStep...
Completed Step 2/6: GMSQualificationStep successfully.
Executing Step 3/6: UpgradeQualificationStep...
-- V1 table does not exist
Failed to qualify upgrade candidate. Aborting the upgrade...
Step with id UpgradeQualificationStep requested an abort of the in-progress update. Aborting the upgrade...
Upgrade NoCodeDataMigration completed with result ABORTED. Exiting...

It seems that a table was missed? What should I do in this case? Thanks!

better-orange-49102

11/06/2022, 11:31 PM

It doesn't matter, really. The upgrade step was for a far older version that didn't use the current metadata_aspect_v2 table

few-sunset-43876

11/07/2022, 2:21 AM

but it's still the old version (e.g v0.8.42) instead of v0.9.1 Is there any step I have missed?

better-orange-49102

11/07/2022, 2:59 AM

if you're using the quickstart CLI, you should just upgrade the acryl-datahub version and run the quickstart again?

few-sunset-43876

11/07/2022, 4:47 AM

Thanks xL! I tried to pull the images using

Copy code

docker-compose down --remove-orphans && docker-compose pull && docker-compose -p datahub up --force-recreate

and I got this issue.

Copy code

Removing network datahub_network
Pulling zookeeper              ... done
Pulling mysql                  ... done
Pulling neo4j                  ... done
Pulling broker                 ... done
Pulling schema-registry        ... done
Pulling kafka-setup            ... done
Pulling elasticsearch          ... done
Pulling elasticsearch-setup    ... done
Pulling datahub-gms            ... done
Pulling datahub-actions        ... done
Pulling datahub-frontend-react ... done
Pulling mysql-setup            ... done
Creating network "datahub_network" with the default driver
Removing schema-registry
Recreating mysql         ... done
Recreating zookeeper     ... done
Recreating elasticsearch ... 
Recreating neo4j         ... 
Recreating elasticsearch ... done
Recreating neo4j               ... done
Recreating mysql-setup         ... done
Recreating broker              ... done
Recreating elasticsearch-setup          ... done
Recreating b8e3b9266356_schema-registry ... 
Recreating b8e3b9266356_schema-registry ... error

ERROR: for b8e3b9266356_schema-registry  Cannot start service schema-registry: driver failed programming external connectivity on endpoint schema-registry (295e93aa8bbcf5cf4d4ad379a0b89019e0dd91cc86bee21a2259250b6d2f72af): Error starting userland proxy: listen tcp4 0.0.0.0:8081: bind: address already in use

ERROR: for schema-registry  Cannot start service schema-registry: driver failed programming external connectivity on endpoint schema-registry (295e93aa8bbcf5cf4d4ad379a0b89019e0dd91cc86bee21a2259250b6d2f72af): Error starting userland proxy: listen tcp4 0.0.0.0:8081: bind: address already in use
ERROR: Encountered errors while bringing up the project.

Is there anyway to modify the port of schema-registry to another port? (when start the quickstart to deploy version v0.8.42, I specified the port of schema-registry to 8082

datahub docker quickstart --version="v0.8.24" --schema-registry-port 8082

)

better-orange-49102

11/07/2022, 5:16 AM

v0.8.24 or 8.42? cos the codebase only had the functionality to set ports via flags in 0.8.41 onwards else, you would have to modify the docker-compose files directly

few-sunset-43876

11/07/2022, 5:46 AM

sorry, it's v0.8.42

few-sunset-43876

11/10/2022, 7:00 AM

hi @better-orange-49102 following the upgrade steps https://datahubproject.io/docs/advanced/no-code-upgrade/#upgrade-steps, I run the pull images

Copy code

docker-compose down --remove-orphans && docker-compose pull && docker-compose -p datahub up --force-recreate

Then to the step 2:

Copy code

cd docker/datahub-upgrade/nocode
./run_upgrade.sh

and I got the same error

Copy code

Starting upgrade with id NoCodeDataMigration...
Cleanup has not been requested.
Skipping Step 1/6: RemoveAspectV2TableStep...
Executing Step 2/6: GMSQualificationStep...
Completed Step 2/6: GMSQualificationStep successfully.
Executing Step 3/6: UpgradeQualificationStep...
-- V1 table does not exist
Failed to qualify upgrade candidate. Aborting the upgrade...
Step with id UpgradeQualificationStep requested an abort of the in-progress update. Aborting the upgrade...
Upgrade NoCodeDataMigration completed with result ABORTED. Exiting...
2022-11-10 06:57:00.566  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutdown requested
2022-11-10 06:57:00.567  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.common.AbstractNettyClient   : Shutting down
2022-11-10 06:57:00.574  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : Shutting down 0 connection pools
2022-11-10 06:57:00.579  INFO 1 --- [ionShutdownHook] c.l.r.t.h.c.c.ChannelPoolManagerImpl     : All connection pools shutdown

I see that the version is upgraded. some data are kept, but I lost the tags, glossary terms... Is there something I missed? I found that the step 2 is only for upgrading the old version v.0.7.x to v0.8.x. we can ignore these logs or skip the step 2, right?

better-orange-49102

11/10/2022, 7:50 AM

hmmm can you check the tags in mysql to see if the entities are still inside (ie the tags you mentioned are missing) data shouldnt be missing unless the volumes got erased.

few-sunset-43876

11/22/2022, 4:15 AM

Thank you @better-orange-49102! It's just because data are not consistent between 2 machines.

7 Views

Open in Slack

Previous Next