https://linen.dev logo
Deploy stuck in loading state
l

late-sandwich-93145

05/25/2023, 10:36 AM
I had a successful deploy of my serenity bot yesterday until all of a sudden it seemed to go offline. When I tired to check I was unable to run "shuttle logs" and it would result in a 503 error. The stop command was hanging so I figured to try another deploy. Now both the original deploy and the new deploy are stuck in "loading" state, the bot is offline and I can't issue a stop command. The original deploy has id 250ebd74-62a8-4d1f-889f-4dae3ccb7f29 The second deploy has id 215540d2-1001-4547-a83b-970d0928f541
a

agreeable-painting-48846

05/25/2023, 10:46 AM
Hey thanks for reporting this, will query with engi
b

bulky-lock-83350

05/25/2023, 11:35 AM
hey @late-sandwich-93145 ! what's your project name?
e

enough-oil-62271

05/25/2023, 11:36 AM
I've hit something similar once or twice, unsure what causes this but generally
cargo shuttle project stop
will take everything down and get the deployments to stop. Then you can bring everything back up, starting with the project and then get a deployment started.
l

late-sandwich-93145

05/25/2023, 11:46 AM
sorew
Is the project name
Ah haven't tried that, unfortunately I'm stuck at work for a few hours, can't try anytime sooner
Yes, the project stop, project start and deploy got it unstuck and running again
I did loose the persistent storage in the process
a

agreeable-painting-48846

05/25/2023, 5:18 PM
oh dear ok that's not good
l

late-sandwich-93145

05/25/2023, 5:19 PM
it's ok, there wasn't much data in there
a

agreeable-painting-48846

05/25/2023, 5:21 PM
Sorry I meant to ask, do you mean persistent storage from
shuttle-persist
?
l

late-sandwich-93145

05/25/2023, 5:21 PM
Yes
a

agreeable-painting-48846

05/25/2023, 5:21 PM
Ok I'll follow this up with engi for you since that's not supposed to happen afaik
l

late-sandwich-93145

05/25/2023, 5:22 PM
I figured its bound to the project, so stopping and recreating the project would kill it
Well it could be my error, I have an .unwrap_or_default() on the deserialization of the file, so on any error it will just start with a blank state
Unfortunately it's happening again
After a few hours running , the bot suddenly is offline, the shuttle status first reports running and then everything returns 503 error
cargo shuttle deployment list Error: 503 Service Unavailable message: project not ready
Service Name: sorew Deployment ID: 99007353-a0e0-4ba3-af71-787cec94cbf2 Status: running Last Updated: 2023-05-25T17:28:11Z
a

agreeable-painting-48846

05/25/2023, 8:46 PM
Is there any chance you can see what made the deployment crash?
l

late-sandwich-93145

05/25/2023, 8:46 PM
This was the latest deployment
no the logs command also doesn't respond
also 503
a

agreeable-painting-48846

05/25/2023, 8:48 PM
are you using
cargo shuttle project restart --idle-minutes 0
?
if you use regular restart it makes the bot go inactive after some time of not using it (default 30m) - if you're not using this flag that maaaaaay cause an issue. just a guess but yeah
l

late-sandwich-93145

05/25/2023, 8:51 PM
could be, i will try that
its running again, unfortunately the persist data is also gone again
a

agreeable-painting-48846

05/25/2023, 8:56 PM
ah dear, ok - thanks for your patience in the meantime
l

late-sandwich-93145

05/25/2023, 8:58 PM
its fine 🙂 it helps to improve the product
if needed i've got the projects open sourced here https://github.com/music-for-cars/sorew
a

agreeable-painting-48846

05/25/2023, 8:59 PM
honestly I don't think
shuttle-persist
has been used by more than a few people so this kind of feedback is quite helpful
l

late-sandwich-93145

05/25/2023, 8:59 PM
its a very simple bot that allows members to !follow a soundcloud page and it will notify a channel whenever a new track is published on one of the pages
i use shuttle persist to store the soundcloud user profile and latest track uri
I do see that the soundcloud profile checking part is generating a ton of debug log messages, almost all coming from the internal html parser
a

agreeable-painting-48846

05/25/2023, 9:18 PM
Hmmmm yeah that's not great
i'm assuming that's from scraper/reqwest?
l

late-sandwich-93145

05/25/2023, 9:18 PM
yep
can I change the default loglevel to info or warn?
a

agreeable-painting-48846

05/25/2023, 9:20 PM
not at the moment - there was a similar issue to this that was raised though specifically with sqlx due to sqlx writing out a log message whenever any query gets called, so the team is already aware that this is something that people want
l

late-sandwich-93145

05/25/2023, 9:21 PM
is this something i could contribute to?
now that the bot pet project is nearing completion i need something new
a

agreeable-painting-48846

05/25/2023, 9:27 PM
i'll defer to engineering on this one since I'm not sure what the most recent update on this is
l

late-sandwich-93145

05/26/2023, 1:22 AM
fyi, it just ran for a few hours with only one profile to watch, and its been running without any issues. i've now added half the group, about 12 profiles, that should 12x the amount of debug logging coming from the html parser
as we speak the performance is degrading rapidly
it takes well over seven seconds to respond to a simple command
logs are inaccessible
and its completely gone again
if you ask me its the debug logs piling up somehow
a

agreeable-painting-48846

05/26/2023, 8:13 AM
Yeah it looks like the issues are due to the logging completely destroying the performance lol
I'll pass the info onto engineering for you 😄
g

gentle-ice-1561

05/26/2023, 11:42 PM
I think this is in the pipeline for the ongoing Shuttle Batch