https://linen.dev logo
Join Discord
Powered by
Deploy stuck in loading state
# help
l
I had a successful deploy of my serenity bot yesterday until all of a sudden it seemed to go offline. When I tired to check I was unable to run "shuttle logs" and it would result in a 503 error. The stop command was hanging so I figured to try another deploy. Now both the original deploy and the new deploy are stuck in "loading" state, the bot is offline and I can't issue a stop command. The original deploy has id 250ebd74-62a8-4d1f-889f-4dae3ccb7f29 The second deploy has id 215540d2-1001-4547-a83b-970d0928f541
a
Hey thanks for reporting this, will query with engi
b
hey @late-sandwich-93145 ! what's your project name?
e
I've hit something similar once or twice, unsure what causes this but generally 
cargo shuttle project stop
will take everything down and get the deployments to stop. Then you can bring everything back up, starting with the project and then get a deployment started.
l
sorew
Is the project name
Ah haven't tried that, unfortunately I'm stuck at work for a few hours, can't try anytime sooner
Yes, the project stop, project start and deploy got it unstuck and running again
I did loose the persistent storage in the process
a
oh dear ok that's not good
l
it's ok, there wasn't much data in there
a
Sorry I meant to ask, do you mean persistent storage from 
shuttle-persist
?
l
Yes
a
Ok I'll follow this up with engi for you since that's not supposed to happen afaik
l
I figured its bound to the project, so stopping and recreating the project would kill it
Well it could be my error, I have an .unwrap_or_default() on the deserialization of the file, so on any error it will just start with a blank state
Unfortunately it's happening again
After a few hours running , the bot suddenly is offline, the shuttle status first reports running and then everything returns 503 error
cargo shuttle deployment list Error: 503 Service Unavailable message: project not ready
Service Name: sorew Deployment ID: 99007353-a0e0-4ba3-af71-787cec94cbf2 Status: running Last Updated: 2023-05-25T17:28:11Z
a
Is there any chance you can see what made the deployment crash?
l
This was the latest deployment
no the logs command also doesn't respond
also 503
a
are you using 
cargo shuttle project restart --idle-minutes 0
?
if you use regular restart it makes the bot go inactive after some time of not using it (default 30m) - if you're not using this flag that maaaaaay cause an issue. just a guess but yeah
l
could be, i will try that
its running again, unfortunately the persist data is also gone again
a
ah dear, ok - thanks for your patience in the meantime
l
its fine 🙂 it helps to improve the product
if needed i've got the projects open sourced here https://github.com/music-for-cars/sorew
a
honestly I don't think 
shuttle-persist
has been used by more than a few people so this kind of feedback is quite helpful
l
its a very simple bot that allows members to !follow a soundcloud page and it will notify a channel whenever a new track is published on one of the pages
i use shuttle persist to store the soundcloud user profile and latest track uri
I do see that the soundcloud profile checking part is generating a ton of debug log messages, almost all coming from the internal html parser
a
Hmmmm yeah that's not great
i'm assuming that's from scraper/reqwest?
l
yep
can I change the default loglevel to info or warn?
a
not at the moment - there was a similar issue to this that was raised though specifically with sqlx due to sqlx writing out a log message whenever any query gets called, so the team is already aware that this is something that people want
l
is this something i could contribute to?
now that the bot pet project is nearing completion i need something new
a
i'll defer to engineering on this one since I'm not sure what the most recent update on this is
l
fyi, it just ran for a few hours with only one profile to watch, and its been running without any issues. i've now added half the group, about 12 profiles, that should 12x the amount of debug logging coming from the html parser
as we speak the performance is degrading rapidly
it takes well over seven seconds to respond to a simple command
logs are inaccessible
and its completely gone again
if you ask me its the debug logs piling up somehow
a
Yeah it looks like the issues are due to the logging completely destroying the performance lol
I'll pass the info onto engineering for you 😄
g
I think this is in the pipeline for the ongoing Shuttle Batch
PreviousNext
Powered by