https://linen.dev logo
Join Discord
Powered by
Deployment has not entered the running state
# help
e
I'm using the latest version of shuttle (0.16) I checked logs and I see no problems. This is the terminal output:
Copy code
...
2023-05-16T21:27:44.448632133Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:18971
2023-05-16T21:27:44.451173177Z DEBUG hyper::client::connect::http: connected to 127.0.0.1:18971
2023-05-16T21:27:44.453423195Z DEBUG {service.ready=true} tower::buffer::worker: processing request
2023-05-16T21:27:44.456117210Z  INFO shuttle_deployer::deployment::run: loading project from: /opt/shuttle/shuttle-executables/1861aef5-6744-4fee-bbc2-958b0ec05057
2023-05-16T21:27:44.458121686Z DEBUG shuttle_deployer::deployment::run: loading service
2023-05-16T21:27:44.460324723Z DEBUG {service.ready=true} tower::buffer::worker: processing request
Deployment has not entered the running state
And this is my code:
Copy code
rust
#[shuttle_runtime::main]
async fn actix_web(
    #[shuttle_shared_db::Postgres] pool: sqlx::PgPool,
    #[shuttle_secrets::Secrets] secret_store: shuttle_secrets::SecretStore,
    #[shuttle_static_folder::StaticFolder(folder = "templates")] static_folder: std::path::PathBuf,
) -> ShuttleActixWeb<impl FnOnce(&mut ServiceConfig) + Send + Clone + 'static> {
    pool.execute("CREATE EXTENSION IF NOT EXISTS pgcrypto")
        .await
        .context("cannot create required extension")?;

    sqlx::migrate!()
        .run(&pool)
        .await
        .context("cannot generate migrations")?;

    let secret = secret_store
        .get("COOKIE_SECRET")
        .context("secret was not found")?;
........
I see logs for the extension, but no for the migration. However there are no errors. Any idea why it's failing?
a
were the migrations already generated?
e
When you say that you checked logs, is that from using 
cargo shuttle logs <ID>
? Sometimes the normal deployment log omits the last error(s) if the deployment crashes too early
e
I don't think so, I didn't see any info logs from sqlx regarding that
a
hmmmm, one thing I've tried is just dropping all tables from my db and just trying to run the app again
e
that log were from the terminal, this is the log from 
cargo shuttle logs --latest
https://cdn.discordapp.com/attachments/1108115647944204398/1108120248147787796/message.txt
e
hmm, not much more. What are you generating migrations from?
a
sometimes if there's no error it might be that either you're trying to run migrations that were diffed from your main migrations, or there's some kind of syntax error with your sql migrations (or the migrations are just wrong)
e
I had my project with shuttle 0.8 and now I upgraded to 0.16 , no other changes were made
e
did you upgrade all the shuttle dependencies to that version?
e
the only code change that I did is to move the main function from lib to main.rs
yes, all are upgraded to 0.16
I'm trying to connect to the database and looks like the credentials are invalid, maybe that's the reason
a
shuttle should be able to connect to the db itself without requiring credential login, not sure what exactly has gone wrong here
e
I'll try to completely remove everything
I'm getting a couple 502-503, is this expected?
Copy code
$ cargo shuttle deploy --no-test
2023-05-16T19:01:09.564362Z ERROR cargo_shuttle::client: failed to connect to websocket
Error: could not connect to websocket

Caused by:
    HTTP error: 502 Bad Gateway
or
Copy code
$ cargo shuttle resource list
Error: 503 Service Unavailable
message: project not ready
a
try running 
cargo shuttle project restart
then deploying again
cargo shuttle deploy seems to not have any problems on my end trying to deploy an app
e
after a clean, stop, start, restart... Same problem:
Copy code
2023-05-16T22:13:11.278867497Z  INFO Entering built state

2023-05-16T22:13:11.279072116Z  INFO Entering loading state
2023-05-16T22:13:11.283850000Z TRACE shuttle_deployer::runtime_manager: making new client
2023-05-16T22:13:11.288727800Z DEBUG shuttle_deployer::runtime_manager: Starting alpha runtime at: /opt/shuttle/shuttle-executables/913f1697-5860-4f33-93ce-6fb1444a7f4b
2023-05-16T22:13:13.291955002Z  INFO shuttle_proto::runtime: connecting runtime client
2023-05-16T22:13:13.292057553Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:16185
2023-05-16T22:13:13.294768443Z DEBUG hyper::client::connect::http: connected to 127.0.0.1:16185
2023-05-16T22:13:13.297126122Z DEBUG {service.ready=true} tower::buffer::worker: processing request
2023-05-16T22:13:13.300472354Z  INFO shuttle_deployer::deployment::run: loading project from: /opt/shuttle/shuttle-executables/913f1697-5860-4f33-93ce-6fb1444a7f4b
2023-05-16T22:13:13.302298615Z DEBUG shuttle_deployer::deployment::run: loading service
2023-05-16T22:13:13.305248279Z DEBUG {service.ready=true} tower::buffer::worker: processing request
Deployment has not entered the running state
a
ok, maybe instead of an expect we could try returning an error with pattern match?
e
sure, let me change that
a
if we can see what the exact error is instead of just panicking it would probably help quite a bit as panics tend to not play very nicely with the logs
e
Copy code
rust

    pool.execute("CREATE EXTENSION IF NOT EXISTS pgcrypto")
        .await
        .context("cannot create required extension")?;

    sqlx::migrate!()
        .run(&pool)
        .await
        .context("cannot generate migrations")?;

    let secret = secret_store
        .get("COOKIE_SECRET")
        .context("secret was not found")?;

    TEMPLATES
        .set({
            let mut tera = tera::Tera::new(
                (static_folder
                    .to_str()
                    .context("cannot get static folder")?
                    .to_string()
                    + "/**/*")
                    .as_str(),
            )
            .context("Parsing error while loading template folder")?;
            tera.autoescape_on(vec!["j2"]);
            tera
        })
        .map_err(|x| anyhow::anyhow!("tera instance couldn't initialize due to: {:?}", x))?;
Copy code
2023-05-16T22:22:51.894760521Z TRACE {folder="\"templates\""} shuttle_static_folder: building static folder
2023-05-16T22:22:51.894777190Z TRACE {input_directory="\"/opt/shuttle/shuttle-builds/realworld-fullstack/templates\""} shuttle_static_folder: got input directory
2023-05-16T22:22:51.894790401Z TRACE {output_directory="\"/opt/shuttle/shuttle-storage/realworld-fullstack\""} shuttle_static_folder: got output directory
2023-05-16T22:22:51.895555882Z  INFO sqlx::postgres::notice: extension "pgcrypto" already exists, skipping
2023-05-16T22:22:51.895612181Z  INFO sqlx::query: CREATE EXTENSION IF NOT …; rows affected: 0, rows returned: 0, elapsed: 290.666µs

CREATE EXTENSION IF NOT EXISTS pgcrypto

2023-05-16T22:22:51.899610964Z  INFO sqlx::query: SELECT current_database(); rows affected: 0, rows returned: 1, elapsed: 462.719µs
same situation :/
a
does it automatically stop running after?
e
Copy code
2023-05-16T22:22:51.862386488Z  INFO shuttle_deployer::deployment::run: loading project from: /opt/shuttle/shuttle-executables/7b86e1c8-ac62-45d9-b398-4838ec632915
2023-05-16T22:22:51.865066320Z DEBUG shuttle_deployer::deployment::run: loading service
2023-05-16T22:22:51.868077630Z DEBUG {service.ready=true} tower::buffer::worker: processing request
Deployment has not entered the running state
yes, it stops running
FYI, with 
cargo shuttle run
works flawlessly
a
do you have the static folder in your .gitignore?
probably not obviously, but i literally just ran into a problem with shuttle not detecting my static folder like 5 seconds ago
e
nope, I have 
Secrets.toml
I can't find the root cause of this problem, the project appears as 
ready
and all the deployments as 
loading
😦
a
does your pg extension need anything (like files, etc) installed externally to be able to use it?
e
Not really, it's built in postgres, but some installations need to explicitly enable it
I uploaded my code changes in my repo https://github.com/Bechma/realworld-actix-fullstack maybe there's something in main that I put wrong, but I don't think so...
a
hmmmm, I have a theory
maybe try commenting out the pgcrypto statement then deploying?
e
Copy code
2023-05-16T23:03:01.036393043Z  INFO sqlx::query: SELECT current_database(); rows affected: 0, rows returned: 1, elapsed: 475.652µs
Same situation, but this time that statement is not executed
a
maybe try deleting the [lib] bit from your cargo.toml?
since you're on 0.16.0 you don't need lib.rs anymore as shuttle apps now get compiled into binaries
e
same output :/
a
ok so I don't know what I did, but it worked from my side
have you tried maybe using 
cargo shuttle resource list
to get your prod db then going into that db and then just dropping all the tables?
a lot of the time when i get random errors that cause prod to crash and there's no proper error it's usually the sql what dunnit because it'll be trying to run migrations and something will be wrong so it'll abort
e
I'm getting password authentication error
psql: error: connection to server at "db.shuttle.rs" (3.8.64.23), port 5432 failed: FATAL:  password authentication failed for user "user-realworld-fullstack"
a
i'm assuming you're just copy and pasting the password in from the db conn string right?
e
yes
a
ok i'll query this with engi
s
What is the last few lines of 
cargo shuttle deployment list
?
e
Copy code
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ 0c6a9ed8-c544-4c71-966f-3b99ac1f605a                 ┆         crashed         ┆         2023-05-16T19:26:43Z        │
│ 1861aef5-6744-4fee-bbc2-958b0ec05057                 ┆         stopped         ┆         2023-05-16T19:27:42Z        │
│ b60b2b11-03df-4b94-b35d-6e5ac9c196c7                 ┆         stopped         ┆         2023-05-16T19:38:16Z        │
│ 4725ea77-3f5e-4f94-aeed-0f3aeff07d60                 ┆         stopped         ┆         2023-05-16T19:48:49Z        │
│ 913f1697-5860-4f33-93ce-6fb1444a7f4b                 ┆         stopped         ┆         2023-05-16T20:13:11Z        │
│ 7b86e1c8-ac62-45d9-b398-4838ec632915                 ┆         stopped         ┆         2023-05-16T20:22:49Z        │
│ 9d4cb67e-a7f3-4826-9376-b5ab82ddcbfe                 ┆         stopped         ┆         2023-05-16T20:37:58Z        │
│ c166c2a7-8e91-46eb-8d81-0295c9e9c750                 ┆         stopped         ┆         2023-05-16T20:39:16Z        │
│ 38888e14-c2c3-451c-916d-d955a50556c3                 ┆         stopped         ┆         2023-05-16T20:45:43Z        │
│ eee4cb56-6713-4ddc-9be2-60383ab90dcf                 ┆         stopped         ┆         2023-05-16T21:11:23Z        │
│ 509966e9-f9c1-42ca-8da5-fed0214cbe32                 ┆         crashed         ┆         2023-05-16T21:14:11Z        │
it's very strange, not it shows stopped or crashed, but yesterday was showing loading all the time
Copy code
2023-05-16T23:14:11.076206555Z ERROR {error="Custom error: cannot generate migrations"} shuttle_runtime::alpha: loading service failed
2023-05-16T23:14:11.077034930Z  INFO {success="false"} shuttle_deployer::deployment::run: loading response
2023-05-16T23:14:11.088616431Z ERROR {error="Custom error: cannot generate migrations"} shuttle_deployer::deployment::run: failed to load service

2023-05-16T23:14:11.093997745Z  INFO Entering crashed state
2023-05-16T23:14:11.099931008Z ERROR {error="Load error: Custom error: cannot generate migrations"} shuttle_deployer::deployment::run: service startup encountered an error
2023-05-16T23:14:11.074085858Z  WARN sqlx::query: SELECT pg_advisory_lock($1); rows affected: 1, rows returned: 1, elapsed: 670.038s
that advisory lock for 670s is what is causing the problem, looks like a problem with the postgres resource?
e
"cannot generate migrations" is reasonably clear... unsure why the lock is making it halt for that long though
do you have migrations / schema stuff in your 
./migrations
folder?
e
yes, one migration
strange, because when I run 
cargo shuttle run
works without issues
e
your migrations folder isn't `.gitignore`'d or anything, right?
e
it's not in .gitignore
e
I wonder if there are restrictions on what we can do in the postgres instance shuttle provides... just to try it out, maybes try removing the 
CREATE INDEX
on your migration? creating tables works, I know that much, but perhaps there's something missing to create an index for some reason?
e
this is what I see in the logs:
Copy code
2023-05-16T23:14:11.074715753Z  INFO sqlx::query: CREATE TABLE IF NOT …; rows affected: 0, rows returned: 0, elapsed: 458.742µs

CREATE TABLE IF NOT EXISTS _sqlx_migrations (
  version BIGINT PRIMARY KEY,
  description TEXT NOT NULL,
  installed_on TIMESTAMPTZ NOT NULL DEFAULT now(),
  success BOOLEAN NOT NULL,
  checksum BYTEA NOT NULL,
  execution_time BIGINT NOT NULL
);

2023-05-16T23:14:11.075703932Z  INFO sqlx::query: SELECT version FROM _sqlx_migrations …; rows affected: 0, rows returned: 0, elapsed: 928.954µs

SELECT
  version
FROM
  _sqlx_migrations
WHERE
  success = false
ORDER BY
  version
LIMIT
  1

2023-05-16T23:14:11.076141020Z  INFO sqlx::query: SELECT version, checksum FROM …; rows affected: 1, rows returned: 1, elapsed: 398.516µs

SELECT
  version,
  checksum
FROM
  _sqlx_migrations
ORDER BY
  version
the queries sent to the server are regarding the inner logic of _sqlx_migrations, it doesn't even reach the first statement that I have in the migrations of 
CREATE TABLE IF NOT EXISTS Users ...
s
The state not updating is an issue we had in old deployers. And I remember you mentioned upgrading from 0.8...
Hmmm, I wonder if sqlx changed the schema of their migrations table 🤔 . Are you still struggling to connect to the DB directly?
e
All this problems began with version 0.16, before it was deployed as of the methods in 0.8
Yes, basically I can't connect to the database due to invalid password
s
It seems we are falling to update the password when the deployment fails (bug), which means you end up connecting with the wrong password. Can you deploy without the migrations folder (maybe add it to 
.ignore
) to get the correct password. Then connect to it and fix the migrations table?
e
spot on, commenting out the migrations line make everything works: https://realworld-fullstack.shuttleapp.rs/ I understand that there is a bug, but I don't get why my database wasn't delete(I mean, I'm supa happy to keep it as it was) when I deleted my project.
s
Oh, we never wipe a database when a user deletes a project. We will have a command to delete resources soon though
e
That's cool
Thanks a lot for the help, and also, is there an issue to track this bug?
g
PreviousNext
Powered by