https://livekit.io logo
Join Slack
Powered by
Hey, our CD is broken using the <deploy-action> ; ...
# cloud
f
Hey, our CD is broken using the deploy-action ; it works well locally but inside the CI it remains stuck at the pushing layers steps. It's not linked to one project because we have the same issues on all our environnements that use different projects.
Copy code
#20 exporting to image
#20 exporting layers
#20 exporting layers 208.1s done
#20 exporting manifest sha256:81646c3e3f96d84ae81ba20b8d1ba5afcb6ce4cda7f372e763e442f7b7bc439e done
#20 exporting config sha256:8e1d2a78cfa641236c0875a8a5c01d8cc5f14abf9f90319aa497a8a67b320b5b done
#20 pushing layers
Any idea of the root cause ? Thanks 🙏
@refined-appointment-81829 Sorry guys but it's quite urgent 😕
r
@echoing-kitchen-90064 any idea on this one?
@few-oxygen-16360 I do not see the error in that log you shared. What does that demonstrate? I assume you have verified the env vars etc in the CI env? https://github.com/livekit/deploy-action/blob/main/README.md#inputs
👍 1
e
hey @few-oxygen-16360 in the past when we see a build timeout it's usually due to not having a valid .dockerignore or .gitignore, we use those to ignore files we shouldn't include like a venv or node_modules etc.
also if you send me your agent_id/project_id i can take a better look
f
Hey @echoing-kitchen-90064, thanks for your answer guys, here is the project id 
p_3ze2fouwaih
and here is the 
CA_NhjasNac2Kpk
agent id
@refined-appointment-81829 That's the whole point.. the build does not output any errors, it just wait for hours starting from 
#20 pushing layers
Here is my github action
action.yaml
We are on this setup for one month now and it's the first time we are facing this issue, this is the same issue on project 
p_ub9eilr1nxy
Maybe not linked but on my project 
p_3ze2fouwaih
I can't delete this agent, neither on the UI and the CLI
Copy code
Using default project [poc]
Using agent [CA_qxNMWvRjMZPW]
unable to delete agent: This action isn't available for Builder agents. Please use the Cloud dashboard instead.
@refined-appointment-81829 @echoing-kitchen-90064 It now fixed 😉 I don't know how and why but now it works again.. any insights ?
e
hmm seems like a timeout somewhere. we'll dig into it, thanks for the report. cc @loud-breakfast-60455
l
We do not allow deleting builder agents from CLI. We'll investigate the issue in the UI. cc @strong-dress-64642
d
Hi Team, I am facing the same issue when trying to deploy an agent via cli using lk agent create. It just gets stuck at Pushing layers. Any idea on what worked or how to resolve this ? thank you in advance for your time on this. 🙏
l
Are you downloading large files (typically models) inside your agent?
🤔 1
d
Nothing too crazy, RAG using HuggingFace model.
I am bumping this to broader audience. Team, my agent works fine locally but I am unable to deploy it to cloud via CLI as it gets stuck at "pushing layers" stage ( see the screenshot in thread). No errors or logs to check, it stays in that state for hours, until manually interrupted. Any help or guidance on this will be highly appreciated. Thank you.
e
please send us some details. we need your agent and project id. can you also show us your .gitignore and your .dockerignore? if you can please include your dockerfile.
d
Thank you and sure. agent_id: CA_83h9kGDvu3vq project_id: p_5pjd7kgz7sm
.dockerignoreDockerfile
e
what region of the world are you in? and which region are you attempting to deploy to?
d
in US and deploying in us-east region
👍 1
e
does this happen on every attempt or only intermittently? also what version of the livekit cli are you on? 
lk --version
d
lk version 2.12.9. Been trying since yesterday on and off and happens every time. ( on a free tier currently 🤫, so i delete old agent for every try)
e
hmm having troubles repro-ing this, will continue to dig for a bit
@dazzling-greece-3781 could you try to run this again? i'm assuming it will still never complete for you (fwiw here after you get to the pushing layers section, you can kill it after a few mins if it doesn't return). once it gets to that point i'd like to try and pin down what is timing out. so ping me once it gets to pushing layers? 🙏
nm, looks like this one is still stuck there 
CA_83h9kGDvu3vq
will take a look at it
i think we have found a solution for this issue, we'll get this change into our staging env today and ensure it is working as expected, we'll try to get it live early next week.
f
Thanks!
d
thank you, just tried again (agent ID: CA_4mUzebU8ziwV ) with same results. Please let me know if i need to do anything differently. Thanks
e
we have pushed a change that increases the amount of time available for this to complete and also should return a better error. you should be sure you're not including very large files in your build though. can you try again please?
✅ 1
🙏 1
🙌 1
d
Thanks, trying now. Already seeing more details on terminal, still going, will update here if the deployment after the fix was successful.
🙌 1
yay, that worked! Thank you very much @echoing-kitchen-90064
e
nice, glad it worked, would be interested in understanding why it took so long. did it end up retrying a few times? not sure why there are multiple sections of Building in your log output
d
no idea, here are raw build logs from the dashboard in case that helps:
Untitled
no idea, here are raw build logs from the dashboard in case that helps:
l
Would you be able to share your Dockerfile?
d
i did, but here
Dockerfile
🙏 1
Open in Slack
PreviousNext
Powered by