https://linen.dev logo
Powered by
Cannot get a deployment to succeed
# help
q
Still having some trouble after the 0.6.0 update. i try
Copy code
cargo shuttle project rm
then
Copy code
cargo shuttle project new
and then i check status
Copy code
cargo shuttle project status
and i see project is in "ready" state then i deploy
Copy code
cargo shuttle deploy
and it will do a build, and spit out the following message:
Copy code
Finished dev [unoptimized + debuginfo] target(s) in 8.70s
     Running unittests src/lib.rs (opt/shuttle/shuttle-builds/discord-fank/target/debug/deps/discord_fank-dfc22324455a74a5)
   Doc-tests discord-fank
Deployment has not entered the running state
and status shows this:
Copy code
╰─ cargo shuttle project status                                                                                         ─╯
project 'discord-fank' is ready
my last deployment id is
Copy code
deployment '7a24aea6-1f3e-4ff4-8282-a0e4a3bf7544' is queued
s
What is 
cargo shuttle deployment list
?
The following usually means that deployment built successfully but failed to start up
Copy code
Deployment has not entered the running state
Currently the actual error as to why doesn't make it from the server to the client and it's one of the things we did not get to fixing today Also, what is the shuttle version in 
Cargo.toml
?
q
shuttle version is 
0.6.0
s
Are you using mongo or postgres?
q
postgres
my 
cargo.toml
s
Hmm, I can see it errored with
Copy code
Run(Custom(status: Unknown, message: "transport error", details: [], metadata: MetadataMap { headers: {} }

Caused by:
    0: transport error
    1: connection error: connection reset
    2: connection reset))
The 
Run(Custom
is us catching an error during startup in the shuttle main function. This might be a bit far fetched, but does a local run work?
Copy code
bash
cargo shuttle run
q
yes
local runs fine
cargo shuttle run
is running fine, just wth some warnings i need to clean up, but otherwise connects to discord fine
s
I'll have to get back on this in the morning. Hopefully I can thing of a way to improve the output
q
so this is something that fails when it goes to deploy on your infra?
heres output of the 
cargo shuttle run
it starts locally fine
any other thoughts on htis
s
I've been planning how to expose the error to users. And I think I might have a solution that will take care of two other bugs as well. Just have not gotten to the coding part yet...
g
I suspect I'm having the same issue. 
run
works fine, but getting: 
Deployment has not entered the running state
Even on a completely clean project. My last deployment is 
3a7f2618-7c32-41f8-a761-1a012be995d7
q
i have a deployment id stuck in "building " state
55b2c372-2989-47c4-96d2-b5417be45e5a
anyway to kill that administratively on the shuttle side
can someone please look at my deployment queue, is there a way to clear these administratively
i have a slew of deployments that have not succeeded
all say crashed except for 1 that is stuck in 'building state'
ok created a new project
based on changing the name in cargo.tomlk
deployment immediately crashes
s
These are just historical records of the last state those deployments were in. We have a bit of polish to do here in the code since stop/removing a project can cause the states to get a bit lost
q
Ah to know good
What about the ‘building’ state that is stuck
I switched frameworks to rocket and stjll won’t deploy
g
Yeah, I'm using rocket and I can't get it to deploy. It builds successfully, but then won't start up as per the original post. There are no logs returned either. I just had a thought whether it was secrets related as none appear when I try to list secrets and these would stop the app from launching. Although I would still expect some logs
Hey @stocky-kangaroo-53672 I'm sure you guys are busy working away at this one. Let me know if there's any tests you want me to do or any extra information I can send you to help debug.
s
@gorgeous-refrigerator-23568 thanks! Sure, I'll do
@gorgeous-refrigerator-23568 and @quiet-australia-98981 I'm hoping you are both able to see the errors now? 🤞
q
let me check, im troubleshooting some SQL issues in my app rn
g
Just updating cargo-shuttle. Will let you know shortly.
I have logs!
Copy code
2022-10-17T22:28:51.219582426Z  INFO shuttle_deployer::deployment::queue: Moving built library

2022-10-17T22:28:51.221699406Z  INFO Entering built state

2022-10-17T22:28:51.293003848Z  INFO Entering loading state
2022-10-17T22:28:51.298648142Z  INFO shuttle_deployer::deployment::provisioner_factory: Provisioning a shared::postgres on the shuttle servers. This can take a while...
2022-10-17T22:28:51.302757505Z DEBUG {service.ready=true} tower::buffer::worker: processing request

2022-10-17T22:28:51.310635139Z  INFO Entering crashed state
2022-10-17T22:28:51.313907134Z ERROR {error="Run error: Custom error: status: Unknown, message: \"transport error\", details: [], metadata: MetadataMap { headers: {} }"} shuttle_deployer::deployment::run: service startup encountered an error
But I'm not any the wiser what the issue is. Does this mean something to you guys?
Deployment id 
0889303b-d243-4300-89e4-b9814d801a3b
s
I ran into a transport error with one of my release tests earlier. It is postgres failing to connect. I'm wondering, did you deploy your service for the first time during the v0.6.0 release?
g
No, it was deployed a few days before 0.6.0
s
Hmm, also with a shared postgres?
g
Yes
s
Mind switching over to rds for a debug test? If we know rds works, then I can focus my efforts on what could be wrong with the shared pg...
g
Sure! Let me give that a go.
OK, that seems to have worked better. The database migration script seems to have run successfully. However, I'm now getting an error about one of my secrets... Is there a special way to add secrets since 0.6.0? (I've updated to use the new secrets crate, and changed the code to match the 0.6.0 example) It looks like it has at least tried to do something with secrets:
Copy code
2022-10-17T22:46:54.357331204Z  INFO shuttle_deployer::deployment::provisioner_factory: Fetching secrets for deployment
2022-10-17T22:46:54.359889846Z  INFO shuttle_deployer::deployment::provisioner_factory: Done fetching secrets
But I eventually get a log from my application:
Copy code
2022-10-17T22:46:54.397799813Z  INFO Entering crashed state
2022-10-17T22:46:54.404311311Z ERROR {error="Run error: Custom error: failed to get PASETO_SECRET_KEY from secrets store"} shuttle_deployer::deployment::run: service startup encountered an error
Also, here's the results of the secrets command:
Copy code
$ cargo shuttle --working-directory api secrets
No secrets are linked to this service
s
Output of 
cargo shuttle secrets
?
Oh boy, the secrets did not ship with the deploy it seems. If they did ship, then you should see another line about setting secrets right before the build kicked off?
g
I can't see a line to that effect. Also, when I run the deploy there are a series of lines prefixed with "Archiving" but these seem to skip my Secrets.toml file.
s
Secrets.toml
should most definitely be in the "Archiving" lines
g
I'm just trying again without 
--working-directory
.
s
Are they excluded in 
Cargo.toml
?
g
Nope.
I'm also going to try removing it from my 
.gitignore
s
I think I can create a ticket so long 🤣
g
That did it.
s
--working-directory
?
g
It wasn't being included because it was in my 
.gitignore
s
🤔 well, that's unexpected. I'm creating a ticket for it. Thanks for the discovery!!
g
No problem! I agree it's unexpected, whilst I can see why you might want that behaviour in some cases, you do want secrets files to be ignored by git exactly as you describe in your documentation: https://docs.shuttle.rs/guide/secrets.html#usage
s
I'm hitting the sack for new. Will debug PG later...
g
Sounds like a plan! It has been a long day. 0.7.0 is looking great, though. Big QOL improvement.
For tomorrow, I'm now in a state where I'm not getting any errors, and the app is stuck in a loading state:
Copy code
$ cargo shuttle deploy --allow-dirty
...
Deployment has not entered the running state
$ cargo shuttle deployment list
...
3d01107f-6d88-45bd-8465-b7a5234c0d4a                                  ┆                  loading                  ┆                  2022-10-17T22:06:02Z
$ cargo shuttle logs 3d01107f-6d88-45bd-8465-b7a5234c0d4a
...
2022-10-17T23:07:19.760045100Z  INFO sqlx::query: SELECT pg_advisory_unlock($1); rows affected: 1, rows returned: 1, elapsed: 385.756µs
2022-10-17T23:07:19.777797802Z DEBUG globset: built glob set; 0 literals, 0 basenames, 0 extensions, 0 prefixes, 0 suffixes, 1 required extensions, 0 regexes
s
Rds tends to take ~5mins to start up... hopefully that is your last log message 🤞
q
hey @stocky-kangaroo-53672 , so 
0.7.0
looks good with the upgrade and the logging, i upgraded the crates and pushed, and ran it locally, however, i am still getting the crash state, almost immediately
Copy code
╰─ cargo shuttle logs a790f321-13db-4c68-af4e-438d4ecf831c                                                              ─╯

2022-10-17T23:26:27.460870378Z  INFO Entering queued state

2022-10-17T23:26:27.462074820Z  INFO Entering building state
2022-10-17T23:26:27.481596918Z  INFO shuttle_deployer::deployment::queue: Fetching POSTed data
2022-10-17T23:26:27.493744807Z DEBUG shuttle_deployer::deployment::queue: Received 7874 bytes
2022-10-17T23:26:27.500390097Z DEBUG shuttle_deployer::deployment::queue: Received 16384 bytes
2022-10-17T23:26:27.507751129Z DEBUG shuttle_deployer::deployment::queue: Received 8349 bytes

2022-10-17T23:26:27.516611126Z  INFO Entering crashed state
2022-10-17T23:26:27.525531019Z ERROR {error="Streaming error: error reading a body from connection: end of file before message length reached"} shuttle_deployer::deployment::queue: service build encountered an error
s
I remember you are using a hybrid service, can you try commenting out axum and then discord separately to isolate which one is causing the issue?
... this one is new to me, so might take a bit of time...
g
Morning @stocky-kangaroo-53672 , unfortunately it looks like my problem is not RDS startup as it's still not responding. I'll share the logs shortly
@stocky-kangaroo-53672 I've looked at the status and logs, and it is unchanged from last night. (https://discord.com/channels/803236282088161321/1030516396402151424/1031690837752549396)
Can you help give some idea as to what is going on with deployment 
3d01107f-6d88-45bd-8465-b7a5234c0d4a
, please?
s
I can see it entered the loading state, but did not progress after that. This would mean your 
shuttle_service::main
function isn't returning. Is it perhaps stuck at an 
.await
?
g
Aha, I'll add more logging. It is quite possible it's waiting for the SQL migration, though I do see most of the logs for that. Thanks for the hint. I'll let you know how I get on.
I've added more logs, but they haven't done any good. I'm now getting the same error I did with the shared PG database, but with Amazon RDS:
Copy code
2022-10-18T10:25:49.514851401Z  INFO Entering loading state
2022-10-18T10:25:49.556150959Z  INFO shuttle_deployer::deployment::provisioner_factory: Provisioning a aws_rds::postgres on the shuttle servers. This can take a while...
2022-10-18T10:25:49.558137980Z DEBUG {service.ready=true} tower::buffer::worker: processing request

2022-10-18T10:25:49.636459931Z  INFO Entering crashed state
2022-10-18T10:25:49.646095468Z ERROR {error="Run error: Custom error: status: Unknown, message: \"transport error\", details: [], metadata: MetadataMap { headers: {} }"} shuttle_deployer::deployment::run: service startup encountered an error
Maybe worth noting that I have added a log line in the entrypoint of my code which does not appear above. Perhpas this transport error may be occurring in the setup of the pool that is passed in?
I'm trying to connect to the connection string presented in 
cargo shuttle status
and it's just a hanging connection. Almost as though there's a firewall in the middle dropping my traffic (maybe this is expected?)
t
I did notice that 
cargo shuttle deploy
seems to be not uploading the newest code (continues to show build warnings of dead code i had in previous release) which is in the end results in non running state. I will upgrade to 0.7 tomorrow
g
Yoroshi, it's just a hunch but I have felt that I need to stage changes before they are picked up. I think the files that are sent to the server take into account git state. Maybe try that and see if it helps.
(Also remember the 
--allow-dirty
flag if the files aren't checked in)
t
I have tried both git committing and --allow-dirty with the same results. unfortunately
g
Ah, fair enough.
s
Hmm, I'm thinking the same. Will check this in a bit...
Okay, so a 
transport error
seems to mean the connection to our resource provisioner was lost... Strangely, it seems that another deploy seconds after this error succeeds
So for now @gorgeous-refrigerator-23568 I think moving back to the shared DB and possibly running deploy twice will get you by? We'll have to investigate why this connection goes stale and how to improve the error message
q
i commented our serenity, tried a deploy, same result, i commented out rocket, tried a deploy, same result
s
Same result as in the same error message? 😰
q
yes unfortunately
the first crash was serenity commented out, the second one was rocket, so must be something else in my package
ill try poking with it more later
s
Ohhh, now I see. Your archive failed to upload/download. A bit of a shot in the dark, but have your tried 
cargo clean
?
q
just did
same result
s
@quiet-australia-98981 did anything change since this deploy?
q
i changed from axum to rocket
oh and i created a new project
s
I wonder if the archive got too big? What is the output of 
ls -lh target/package
?
q
here is where im registering rocket in the 
tokio::select!
as well as serenity
perhaps something changed about how these two are registered
but produced the same result when i commented each out, and ran a 
cargo shuttle deploy --allow-dirty
s
From this, we failed before / during the unpacking of the archive... so it never even got to the build part
q
oh
you dont think it has anything to do with this old build still showing in 
building
state
│ 55b2c372-2989-47c4-96d2-b5417be45e5a
s
No, that's just us loosing track when a project was removed while a build was running... when a project starts up (is created) it will only look for deployments in the "running" state
q
its very odd, it seems to try to enter build state but then fails right after
Copy code
2022-10-18T11:37:56.537285252Z  INFO Entering building state
2022-10-18T11:37:56.554118114Z  INFO shuttle_deployer::deployment::queue: Fetching POSTed data
2022-10-18T11:37:56.567677820Z DEBUG shuttle_deployer::deployment::queue: Received 7873 bytes
2022-10-18T11:37:56.575128791Z DEBUG shuttle_deployer::deployment::queue: Received 16384 bytes
2022-10-18T11:37:56.582209303Z DEBUG shuttle_deployer::deployment::queue: Received 8349 bytes

2022-10-18T11:37:56.592929894Z  INFO Entering crashed state
2022-10-18T11:37:56.599574851Z ERROR {error="Streaming error: error reading a body from connection: end of file before message length reached"} shuttle_deployer::deployment::queue: service build encountered an error
s
@quiet-australia-98981 interestingly I can get past this specific error by adding the following to `Cargo.toml`:
Copy code
toml
exclude = [
    "web/.env.example",
    "web/.gitignore",
    "web/README.md",
    "web/app.vue",
    "web/assets/logo.png",
    "web/assets/ogimage.png",
    "web/components/AppHeader.vue",
    "web/components/Rank.vue",
    "web/layouts/default.vue",
    "web/layouts/global.css",
    "web/nuxt.config.ts",
    "web/package.json",
    "web/pages/Rank/index.vue",
    "web/pages/index.vue",
    "web/store/index.js",
    "web/yarn.lock"
]
q
can i do
Copy code
rust
exclude = [
  "web/*"
]
let me play with this
s
Yip, 
exclude = [ "web/"]
works too
For some reason the 
web/tsconfig.json
file is fine 🤷‍♂️
q
got past that error!
thank you
yea theres a lot going on in this repo, i just figured to put my nuxt frontend in the 
./web/
dir for simplicities sake
ok new error
@stocky-kangaroo-53672
Copy code
2022-10-18T12:29:10.275267974Z  INFO     Finished dev [unoptimized + debuginfo] target(s) in 53.23s
2022-10-18T12:29:10.566822725Z  INFO      Running unittests src/lib.rs (opt/shuttle/shuttle-builds/discord-fank/target/debug/deps/discord_fank-f337983abd1ad82e)
2022-10-18T12:29:10.602865987Z  INFO    Doc-tests discord-fank
2022-10-18T12:29:16.067005922Z  INFO shuttle_deployer::deployment::queue: Moving built library

2022-10-18T12:29:16.070623888Z  INFO Entering built state

2022-10-18T12:29:16.203615009Z  INFO Entering loading state
2022-10-18T12:29:16.315856031Z  INFO shuttle_deployer::deployment::provisioner_factory: Provisioning a shared::postgres on the shuttle servers. This can take a while...
2022-10-18T12:29:16.316512490Z DEBUG {service.ready=true} tower::buffer::worker: processing request

2022-10-18T12:29:16.354142972Z  INFO Entering crashed state
2022-10-18T12:29:16.356128524Z ERROR {error="Run error: Custom error: status: Unknown, message: \"transport error\", details: [], metadata: MetadataMap { headers: {} }"} shuttle_deployer::deployment::run: service startup encountered an error
oh
ok, got further running a second time
ok now im at this, very managable error:
Copy code
{error="Run error: Custom error: Failed to get discord token"}
did secrets handling change in 0.6.0 or 0.7.0
i have a secrets.toml defined locally but not checked in
nvm, the secrets.toml fild being added to gitignore was my issue
g
Ok, I'll try this.
I have found where in my code the app is freezing. I am using Tera templates, and I have encountered this bug: https://github.com/Keats/tera/issues/740 Basically globwalk gets stuck in a loop when running from the root of linux, as it so often does when running from a Docker container. I'm in the process of putting logs in to work out where my files are in the container so that I can put sensible paths in to avoid this issue.
In the process of debugging I also tried deleting the project and starting again, however, upon recreating of the project it turns out it didn't actually delete the underlying service. When I try deleting just the service, I get an error back that a database constraint is stopping it being deleted. As this is a RDS database, I don't seem to be able to contact it directly to remove the constraint to delete the service. Assistance here would be welcome.
Furthermore, and my final post for tonight: I was doing a lot of deploys, and realised I had an issue in my code, so to save time I Ctrl-C'd the deployment. The whole thing now seems to be crashed and I am not getting any responses to any of my cargo shuttle commands. 🤞 I can get this deployed tomorrow!
t
Deploy also seems to not take into consideration removal of a file (i removed main.rs, committed, yet it still remains to be built (and crash) when deploying)
even after the server reboot, seem to be getting the same issue as above. (even removed and recreated the project)
g
@stocky-kangaroo-53672 are there any examples of how I can include templates (I'm specifically using tera) to be programmatically loaded in my application? Hunting around using 
fs::read_dir
is proving a very inefficient way to find my templates:
Copy code
2022-10-19T10:36:28.234696852Z  INFO calandar_api: Binary is located at: /usr/local
2022-10-19T10:36:28.234730625Z  INFO calandar_api: Name: /usr/local/bin
2022-10-19T10:36:28.234733048Z  INFO calandar_api: Name: /usr/local/etc
2022-10-19T10:36:28.234734591Z  INFO calandar_api: Name: /usr/local/games
2022-10-19T10:36:28.234735840Z  INFO calandar_api: Name: /usr/local/include
2022-10-19T10:36:28.234737081Z  INFO calandar_api: Name: /usr/local/lib
2022-10-19T10:36:28.234738324Z  INFO calandar_api: Name: /usr/local/man
2022-10-19T10:36:28.234739539Z  INFO calandar_api: Name: /usr/local/sbin
2022-10-19T10:36:28.234740634Z  INFO calandar_api: Name: /usr/local/share
2022-10-19T10:36:28.234741743Z  INFO calandar_api: Name: /usr/local/src
2022-10-19T10:36:28.234742813Z  INFO calandar_api: Name: /usr/local/cargo
2022-10-19T10:36:28.234743966Z  INFO calandar_api: Name: /usr/local/rustup
s
Do you mean 
main.rs
is still being archived when deploying even though it's been removed?
We currently don't have static file support. A workaround users have used in the past is to upload the files to an S3 and then download it to a known location in the shuttle main function
g
Aha. Good to know, thank you!
t
Yep like it hasn't been removed and is now causing the deployment to not build
s
Oh, yes I see why this would happen 🤦‍♂️ . Okay, I'm opening a ticket for this
t
Possibly a git rm cache thing?
But good you have an idea 🙂
s
No, two things really: 1. We don't clear the build folder before extracting the archive 2. We don't remove the docker volume when the project is deleted
g
@stocky-kangaroo-53672 Thank you for all your help. It has been many days, but finally this!
s
Thanks for the patience @gorgeous-refrigerator-23568!
t
i guess fixing these two things will be good QoL
s
Yip, urgent tickets has been created for both of them
q
hey @stocky-kangaroo-53672 just wanted to report, after fixing a config issue with rocket, and after the server went back up, my project successfully deployed!
s
This should be fixed with the latest release, and this workaround should no longer be needed
q
oh awesome!
s
This should also be fixed
q
whats the latest release 0.7.1 or 0.8.0?
oh i see the annoucement
very nice
s
For others, it's 0.7.1 😄
This is likely to be part of our next focus
t
🤞
PreviousNext
Powered by