Deploy stuck in loading state

Question

I had a successful deploy of my serenity bot yesterday until all of a sudden it seemed to go offline When I tired to check I was unable to run shuttle logs and it would result in a 503 error The stop command was hanging so I figured to try another deploy Now both the original deploy and the new deploy are stuck in loading state the bot is offline and I can t issue a stop command The original deploy has id 250ebd74 62a8 4d1f 889f 4dae3ccb7f29 The second deploy has id 215540d2 1001 4547 a83b 970d0928f541

agreeable-painting-48846 · Answer

Hey thanks for reporting this will query with engi

bulky-lock-83350 · Answer

hey < late sandwich 93145> what s your project name

enough-oil-62271 · Answer

I ve hit something similar once or twice unsure what causes this but generally `cargo shuttle project stop` will take everything down and get the deployments to stop Then you can bring everything back up starting with the project and then get a deployment started

late-sandwich-93145 · Answer

sorew

late-sandwich-93145 · Answer

Is the project name

late-sandwich-93145 · Answer

Ah haven t tried that unfortunately I m stuck at work for a few hours can t try anytime sooner

late-sandwich-93145 · Answer

Yes the project stop project start and deploy got it unstuck and running again

late-sandwich-93145 · Answer

I did loose the persistent storage in the process

agreeable-painting-48846 · Answer

oh dear ok that s not good

late-sandwich-93145 · Answer

it s ok there wasn t much data in there

agreeable-painting-48846 · Answer

Sorry I meant to ask do you mean persistent storage from `shuttle persist`

late-sandwich-93145 · Answer

Yes

agreeable-painting-48846 · Answer

Ok I ll follow this up with engi for you since that s not supposed to happen afaik

late-sandwich-93145 · Answer

I figured its bound to the project so stopping and recreating the project would kill it

late-sandwich-93145 · Answer

Well it could be my error I have an unwrap or default on the deserialization of the file so on any error it will just start with a blank state

late-sandwich-93145 · Answer

Unfortunately it s happening again

late-sandwich-93145 · Answer

After a few hours running the bot suddenly is offline the shuttle status first reports running and then everything returns 503 error

late-sandwich-93145 · Answer

cargo shuttle deployment list Error 503 Service Unavailable message project not ready

late-sandwich-93145 · Answer

Service Name sorew Deployment ID 99007353 a0e0 4ba3 af71 787cec94cbf2 Status running Last Updated 2023 05 25T17 28 11Z

agreeable-painting-48846 · Answer

Is there any chance you can see what made the deployment crash

late-sandwich-93145 · Answer

This was the latest deployment

late-sandwich-93145 · Answer

no the logs command also doesn t respond

late-sandwich-93145 · Answer

also 503

agreeable-painting-48846 · Answer

are you using `cargo shuttle project restart idle minutes 0`

agreeable-painting-48846 · Answer

if you use regular restart it makes the bot go inactive after some time of not using it default 30m if you re not using this flag that maaaaaay cause an issue just a guess but yeah

late-sandwich-93145 · Answer

could be i will try that

late-sandwich-93145 · Answer

its running again unfortunately the persist data is also gone again

agreeable-painting-48846 · Answer

ah dear ok thanks for your patience in the meantime

late-sandwich-93145 · Answer

its fine it helps to improve the product

late-sandwich-93145 · Answer

if needed i ve got the projects open sourced here https github com music for cars sorew

agreeable-painting-48846 · Answer

honestly I don t think `shuttle persist` has been used by more than a few people so this kind of feedback is quite helpful

late-sandwich-93145 · Answer

its a very simple bot that allows members to follow a soundcloud page and it will notify a channel whenever a new track is published on one of the pages

late-sandwich-93145 · Answer

i use shuttle persist to store the soundcloud user profile and latest track uri

late-sandwich-93145 · Answer

I do see that the soundcloud profile checking part is generating a ton of debug log messages almost all coming from the internal html parser

agreeable-painting-48846 · Answer

Hmmmm < frog think 1074922112436949012> yeah that s not great

agreeable-painting-48846 · Answer

i m assuming that s from scraper reqwest

late-sandwich-93145 · Answer

yep

late-sandwich-93145 · Answer

can I change the default loglevel to info or warn

agreeable-painting-48846 · Answer

not at the moment there was a similar issue to this that was raised though specifically with sqlx due to sqlx writing out a log message whenever any query gets called so the team is already aware that this is something that people want

late-sandwich-93145 · Answer

is this something i could contribute to

late-sandwich-93145 · Answer

now that the bot pet project is nearing completion i need something new

agreeable-painting-48846 · Answer

i ll defer to engineering on this one since I m not sure what the most recent update on this is

late-sandwich-93145 · Answer

fyi it just ran for a few hours with only one profile to watch and its been running without any issues i ve now added half the group about 12 profiles that should 12x the amount of debug logging coming from the html parser

late-sandwich-93145 · Answer

as we speak the performance is degrading rapidly

late-sandwich-93145 · Answer

it takes well over seven seconds to respond to a simple command

late-sandwich-93145 · Answer

logs are inaccessible

late-sandwich-93145 · Answer

and its completely gone again

late-sandwich-93145 · Answer

if you ask me its the debug logs piling up somehow

agreeable-painting-48846 · Answer

Yeah it looks like the issues are due to the logging completely destroying the performance lol

agreeable-painting-48846 · Answer

I ll pass the info onto engineering for you

gentle-ice-1561 · Answer

I think this is in the pipeline for the ongoing Shuttle Batch