I've deployed my API with a db schema to Shuttle using the provisioned db and when I try to access a route that returns something from the database, it returns this:

error returned from database: password authentication failed for user "user-hello-rocket-e"

I've tried using the "log in" button on the website to get a new key and put it back into the CI but that doesn't seem to have fixed it, so presumably this isn't user error since there's no user input required for db provisioning

Yip, this is an error on our side. What is the full output of

cargo shuttle logs <id>

? (

cargo shuttle deployment list

to get the id)

Ahhh, okay so from the log I see the connection initially succeeded on the 3rd when the service started up. But then failed this morning (2 days later). This would mean the credentials changed and the only thing that can change the credentials will be a new deploy (we roll the credentials when a new deploy starts up). I see there was a new deploy (with id

ba7730c9-f60e-4ef9-af37-94f4ac6534b3

) yesterday (the 4th) but it never entered the started state. So this would not have rolled the credentials. But then there was another new deploy (

41ce7fd5-25c2-499b-970d-8b9b66be9975

) a bit later which did start up but then crashed on running the migrations. Since it started up [1], it did roll the credentials. Therefore the (older) running deploy has the wrong credentials. This is a catch 22 shuttle bug I would say. This is because we only stop a deployment if a never deployment enters the

running

state. But before a deployment is running we have to start it up (the

loading

state) and it's during this state that we provision resources or roll passwords. But technically an older running deployment will then still use the old credentials like it does here. Quick diagram of the states to help explain things

Queued -> Building -> Built -> Loading -> Running -> Stopped

[1]: This line in the logs is the same time the credentials are being rolled

Provisioning a shared::postgres on the shuttle servers. This can take a while...

Ah I see, yeah that makes sense - I did actually have some issues with my migrations crashing trying to run

cargo shuttle run

locally but I fixed it by just deleting my sqlx migrations table and running it again. I'm not sure if that's helpful information though since in a prod environment I don't think that would be feasible lol

I think for prod you will probably want to connect to the DB and fix the migrations manually 😅 The connection string from

cargo shuttle status

should be the one from the last credentials roll

Ok sounds good to me, I'll get on that and let you know the outcome