Hi all, Trying to run ./run_upgrade.sh script ================== Error trace below ... What are the pre-requisite to run this upgrade ?? TIA and Happy New Year 🙂 Bruno ==== head: Pulling from acryldata/datahub-upgrade Digest: sha256:2691ab0cdf49be771b38e472a1ae61a130194b97b856430ccbc0a170190537c0 Status: Image is up to date for acryldata/datahub-upgrade:head docker.io/acryldata/datahub-upgrade:head ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console... =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.5.12) 2023-01-10 145004.598 INFO 1 --- [ main] io.ebean.EbeanVersion : ebean version: 11.33.3 2023-01-10 145004.619 INFO 1 --- [ main] io.ebean.config.properties.LoadContext : loaded properties from [application.yml] Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 2023-01-10 145004.707 INFO 1 --- [ main] io.ebean.datasource.pool.ConnectionPool : DataSourcePool [gmsEbeanServiceConfig] autoCommit[false] transIsolation[READ_COMMITTED] min[2] max[50] 2023-01-10 145009.999 INFO 1 --- [ main] i.e.d.pool.PooledConnectionQueue : Reseting DataSourcePool [gmsEbeanServiceConfig] min[2] max[50] free[0] busy[0] waiting[0] highWaterMark[0] waitCount[0] hitCount[0] 2023-01-10 145009.999 INFO 1 --- [ main] i.e.d.pool.PooledConnectionQueue : Busy Connections: 2023-01-10 145010.000 ERROR 1 --- [ main] c.l.g.factory.entity.EbeanServerFactory : Failed to connect to the server. Is it up? ERROR SpringApplication Application run failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'upgradeCli': Unsatisfied dependency expressed through field 'noCodeUpgrade'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ebeanServer' defined in class path resource [com/linkedin/gms/factory/entity/EbeanServerFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.ebean.EbeanServer]: Factory method 'createServer' threw exception; nested exception is java.lang.NullPointerException at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:659) $$$$$

Hello Bruno, How are you running the upgrade job? How is DataHub deployed?

Hello Pedro, It's a quickstart install on docker with containers locally instancied.

Using

datahub docker quickstart

?

Yes, but I wanted to keep my data already ingested .

You can backup quickstart by running:

datahub docker quickstart --backup

and then run

datahub docker quickstart --version <updated version> --restore

Thanks it seems there is some error to write the backup file C\Users\JB6407\Anaconda3\lib\site packages\paramiko\transport.py219: CryptographyDeprecationWarning: Blowfish has been deprecated "class": algorithms.Blowfish, [2023-01-10 174525,520] INFO {datahub.cli.docker_cli:259} - Creating directory C:\Users\JB6407/.datahub/quickstart if it doesn't exist [2023-01-10 174525,520] INFO {datahub.cli.docker_cli:261} - Executing backup command /bin/bash: line 1: CUsersJB6407/.datahub/quickstart/backup.sql No such file or directory [2023-01-10 174525,645] INFO {datahub.cli.docker_cli:269} - Backup written to C:\Users\JB6407/.datahub/quickstart/backup.sql with status 1

Does the file exist in

C:\Users\JB6407/.datahub/quickstart/backup.sql

? I don’t see an error message here?

Hi Pedro, my bad , datahub containers were not correctly started , now I have the backup file filed.

Ah, that is important. You can specify the docker-compose file for quickstart which should be the same thing

Finally run it , but try to read an env file , but there is no file in this path , and argument --env-file is not possible will catch the command line and modify it

[2023-01-10 19:59:40,959] INFO     {datahub.cli.docker_cli:375} - Running index restore command: docker pull acryldata/datahub-upgrade:${DATAHUB_VERSION:-head} && docker run --network datahub_network --env-file C:\Users\JB6407\AppData\Local\Temp\tmpe1hy_x37 acryldata/datahub-upgrade:${DATAHUB_VERSION:-head} -u RestoreIndices -a clean

[2023-01-10 19:59:43,316] INFO     {datahub.cli.docker_cli:389} - Index restore command finished with status 125

[2023-01-10 19:59:43,317] INFO     {datahub.cli.docker_cli:393} - b"docker: open C:UsersJB6407AppDataLocalTemptmpe1hy_x37: no such file or directory.\nSee 'docker run --help'.\n"

OUT with docker pull command

Executing Step 1/3: ClearSearchServiceStep...

Completed Step 1/3: ClearSearchServiceStep successfully. Executing Step 2/3: ClearGraphServiceStep... Failed to clear graph indices: org.neo4j.driver.exceptions.ServiceUnavailableException: Unable to connect to neo4j:7687, ensure the database is running and that there is a working network connection to it. Failed to clear graph indices: org.neo4j.driver.exceptions.ServiceUnavailableException: Unable to connect to neo4j:7687, ensure the database is running and that there is a working network connection to it. Failed Step 2/3: ClearGraphServiceStep. Failed after 1 retries. Exiting upgrade RestoreIndices with failure. Upgrade RestoreIndices completed with result FAILED. Exiting...

The logs are confusing, was datahub deployed successfully? It does not seem like it. This is why the

ClearGraphServiceStep

fails.

In fact I deploy without neo4J, finally, trying remove container with docker-compose ... down and up but stuck with version 0.9.1 the tag :latest seems not working...

Latest is not a stable tag. I suggest you going to 0.9.3

Hi @incalculable-ocean-74010 how can I get the correct tag starting from https://hub.docker.com/r/acryldata/datahub-actions/tags ? Thanks .

You can use v0.0.10

I've relaunched but there must be a cache somewhere ? :

←[33m:exclamation:Client-Server Incompatible:exclamation:←[0m ←[36mYour client version 0.9.5.2 is newer than your server version 0.9.1. Downgrading the cli to 0.9.1 is recommended.

`←[0m ←[36m➡️ Downgrade via

"pip install 'acryl-datahub==0.9.1'"←[0m

What command did you execute?

docker-compose --file .\docker-compose-quickstart.yml up

last message comes from manuel ingestion with a cmd like

datahub ingest -c .\ingest_glossary.yml

I don't think the message is an error, just a recommendation. Cc @gray-shoe-75895 @dazzling-judge-80093

Yes I agree, the point is that I have not succeeded in upgrading to v0.9.5 to test the lineage by UI 🙂 !

I’m confused then. If you want to try v0.9.5 features, please update the docker compose file to use v0.9.5 rather than 0.9.3, before you were trying with

latest

tag which is never guaranteed to work and will always return a different codebase whenever someone commits something to master branch.

Thanks , but https://hub.docker.com/r/linkedin/datahub-ingestion/tags?page=1&name=0.9 which repo is the official ? linkedin or acryldata the docker-compose mix both ? :-)

linkedin

Hi @incalculable-ocean-74010,Finally got it worked in 0.9.5 , little detail tag for actions must be v0.0.9 ! Thanks

Awesome, glad you got it working :)

Hello @faint-actor-78390 and @incalculable-ocean-74010, I have the same problem:

(venv_dh) C:\erasmo> datahub docker quickstart --backup
←[32m←[2m[2023-12-12 08:46:27,325]←[0m INFO    ←[0m ←[34m←[2m{datahub.cli.docker_cli:272}←[0m - Creating directory C:\Users\usr.erasmo.senger/.datahub/quickstart if it doesn't exist←[0m
←[32m←[2m[2023-12-12 08:46:27,325]←[0m INFO    ←[0m ←[34m←[2m{datahub.cli.docker_cli:274}←[0m - Executing backup command←[0m
/bin/bash: line 1: C:Userserasmo/.datahub/quickstart/backup.sql: No such file or directory
←[32m←[2m[2023-12-12 08:46:28,867]←[0m INFO    ←[0m ←[34m←[2m{datahub.cli.docker_cli:282}←[0m - Backup written to C:\Users\erasmo/.datahub/quickstart/backup.sql with status 1←[0m

How did you manage to resolve it? In my case the .datahub/quickstart/ folder exists, but the "backup.sql" file is not generated. I'm using version 0.12.0