https://datahubproject.io logo
Join Slack
Powered by
Starting a new thread to try to move forward on th...
# all-things-deployment
s
Starting a new thread to try to move forward on the upgrading issue, started here https://datahubspace.slack.com/archives/CV2UVAPPG/p1678887964885129 Joined office hours today and got some good input from @brainy-tent-14503. I will upgrade the ec2 from t2.large to t2.xlarge tomorrow. I was able to log into it with ssh now, so I can continue with debugging.
📖 1
✅ 1
🔍 1
l
This content can't be displayed.
s
This is what's running now after the quickstart 4 weeks ago.
I restarted the instance, and ran datahub docker quickstart again.
Only the frontend, actions, mysql and elasticsearch stays up. I checked different logs, and the one from zoopkeeper was the one that might be interesting? There was a kafka docker that was pointing to errors with zookeeper as well
@echoing-airport-49548 do you have any ideas about this?
b
Zookeeper’s volume is possibly corrupted, however to fix the permission, execute the following within the zookeeper docker image. 
chown appuser:appuser -R /var/lib/zookeeper/data
s
I get the error: chown: cannot access '/var/lib/zookeeper/data': Permission denied
o
Looks like it should be mapping the folder 
zkdata
to the 
/var/lib/zookeeper
volume being set up for the Docker image, so there should be a folder 
zkdata
that is writeable on the linux box
s
I don't really understand what I do to fix this then? 🙂 But got my hopes up we got a solution here 🙂
image.png
I was able to fix zookeeper with adding user: "0:0" in the docker compose and run the modified docker compose with datahub docker quickstart -f docker-compose_new.yml. Now the problem is kafka I think, it shut downs with the last error: ERROR Shutdown broker because all log dirs in /var/lib/kafka/data have failed
[2023-04-28 100852,797] ERROR Error while loading log dir /var/lib/kafka/data (kafka.server.LogDirFailureChannel) java.nio.file.AccessDeniedException: /var/lib/kafka/data/__consumer_offsets-10/00000000000000000000.log
Got that also fixed by adding user: "0:0" for kafka. Getting errors with gms and schema-registry now that might be the next errors
kafka seems fine?
image.png
schema-registry error:
datahub-gms:
o
GMS is probably not coming up due to the Schema Registry errors since it will try to read from the upgrade topic
s
ok, what seems to be wrong with schema-registry?
it's just strange since this was just a upgrade with datahub docker quickstart, so stuff like this should not happen?
o
It can't find the broker, not sure why that would happen. If you're using a custom compose yml maybe verify the docker network is properly configured?
s
this is what is running and not stopping
image.png
I used a custom compose file to add AUTH_OIDC_CLIENT_ID etc to the react container, running it with docker compose -p datahub -f docker-compose.yml -f docker-compose.yml up -d datahub-frontend-react
nothing is changed with network and so on
I got it working, basically after doing multiple datahub docker quickstarts, and starting up different containers manually with docker start, until it somehow got fixed with the upgrades etc
Thank you for the partial debugs on the way to success ❤️ ❤️ @orange-night-91387 @big-carpet-38439 @brainy-tent-14503 @echoing-airport-49548 @little-megabyte-1074
Open in Slack
PreviousNext
Powered by