I'm having an issue where lk deploy seems to succe...
# clouda
agreeable-cricket-21439
12/10/2025, 8:53 PMI'm having an issue where lk deploy seems to succeed, build logs say deploy completed, but the code does not get updated.
How would I troubleshoot something like that?
agreeable-cricket-21439
12/10/2025, 8:53 PMI can tell from the logs that I'm running old code
e
echoing-kitchen-90064
12/10/2025, 9:11 PMhey there, can i get your agent_id/project_id?
a
agreeable-cricket-21439
12/10/2025, 9:15 PMproj:
p_a5qvb3muhe0CA_tRAdggkA3NQke
echoing-kitchen-90064
12/10/2025, 9:18 PMhmm i'm not seeing a current build, could you try a rebuild?
a
agreeable-cricket-21439
12/10/2025, 9:18 PMfrom when is the last build that you see?
agreeable-cricket-21439
12/10/2025, 9:19 PMbtw I've just triggered a deploy
e
echoing-kitchen-90064
12/10/2025, 9:20 PMfrom when is the last build that you see?26h ago
a
agreeable-cricket-21439
12/10/2025, 9:22 PMbut I do see newer versions on my UI
agreeable-cricket-21439
12/10/2025, 9:23 PMimage.png
agreeable-cricket-21439
12/10/2025, 9:23 PMIdk if this link is useful:
https://cloud.livekit.io/projects/p_a5qvb3muhe0/agents/CA_tRAdggkA3NQk
e
echoing-kitchen-90064
12/10/2025, 9:25 PMdid your new build complete?
a
agreeable-cricket-21439
12/10/2025, 9:25 PMWhere can I see that?
agreeable-cricket-21439
12/10/2025, 9:25 PMIs there any logs for the build process?
agreeable-cricket-21439
12/10/2025, 9:26 PMThe last thing I see is that is exporting this to a docker image:
e
echoing-kitchen-90064
12/10/2025, 9:27 PMah, are you using the builder through the UI?
echoing-kitchen-90064
12/10/2025, 9:27 PMand not the cli?
a
agreeable-cricket-21439
12/10/2025, 9:27 PMI'm running trough the cli
e
echoing-kitchen-90064
12/10/2025, 9:27 PMthe cli will output logs when you build
a
agreeable-cricket-21439
12/10/2025, 9:29 PM Copy code
echo "Deploying to LiveKit Cloud..."
lk agent deploy \
--url "${LIVEKIT_URL}" \
--api-key "${LIVEKIT_API_KEY}" \
--api-secret "${LIVEKIT_API_SECRET}" \
--secrets-file .env.production \
--config deploy/livekit.toml
Deploying to LiveKit Cloud...
Using url, api-key, api-secret from environment
Using agent [CA_tRAdggkA3NQk]
Using secrets file [.env.production]
[+] Building 138.4s (21/21) FINISHED
=> [internal] load remote build context 0.4s
=> copy /context / 8.4s
=> [internal] load metadata for <http://ghcr.io/astral-sh/uv:latest|ghcr.io/astral-sh/uv:latest> 0.3s
=> [internal] load metadata for <http://docker.io/library/python:3.13-slim-bookworm|docker.io/library/python:3.13-slim-bookworm> 0.1s
=> FROM <http://ghcr.io/astral-sh/uv:latest@sha256:5cb6b54d2bc3fe2eb9a8483db958a0b9eebf9edff68adedb369df8e7b98711a2|ghcr.io/astral-sh/uv:latest@sha256:5cb6b54d2bc3fe2eb9a8483db958a0b9eebf9edff68adedb369df8e7b98711a2> 0.0s
=> => resolve <http://ghcr.io/astral-sh/uv:latest@sha256:5cb6b54d2bc3fe2eb9a8483db958a0b9eebf9edff68adedb369df8e7b98711a2|ghcr.io/astral-sh/uv:latest@sha256:5cb6b54d2bc3fe2eb9a8483db958a0b9eebf9edff68adedb369df8e7b98711a2> 0.0s
=> [builder 1/8] FROM <http://docker.io/library/python:3.13-slim-bookworm@sha256:ac4274977cc5093aa7675b5f39e59403c7b692d0bf7549b49563177ddc02552d|docker.io/library/python:3.13-slim-bookworm@sha256:ac4274977cc5093aa7675b5f39e59403c7b692d0bf7549b49563177ddc02552d> 0.0s
=> => resolve <http://docker.io/library/python:3.13-slim-bookworm@sha256:ac4274977cc5093aa7675b5f39e59403c7b692d0bf7549b49563177ddc02552d|docker.io/library/python:3.13-slim-bookworm@sha256:ac4274977cc5093aa7675b5f39e59403c7b692d0bf7549b49563177ddc02552d> 0.0s
=> CACHED [builder 2/8] COPY --from=<http://ghcr.io/astral-sh/uv:latest|ghcr.io/astral-sh/uv:latest> /uv /uvx /bin/ 0.0s
=> CACHED [builder 3/8] WORKDIR /app 0.0s
=> CACHED [builder 4/8] COPY pyproject.toml uv.lock ./ 0.0s
=> CACHED [builder 5/8] RUN uv sync --no-dev 0.0s
=> [builder 6/8] COPY agents/ ./agents/ 2.8s
=> [builder 7/8] COPY pyproject.toml ./ 0.0s
=> [builder 8/8] RUN .venv/bin/python -m agents.entrypoints.prepare_livekit 8.6s
=> CACHED [stage-1 4/10] RUN useradd -m -u 1000 appuser 0.0s
=> CACHED [stage-1 5/10] COPY --from=builder /app/.venv /app/.venv 0.0s
=> [stage-1 6/10] COPY --from=builder /root/.cache/huggingface /home/appuser/.cache/huggingface 0.5s
=> [stage-1 7/10] COPY agents/ ./agents/ 0.1s
=> [stage-1 8/10] COPY pyproject.toml ./ 0.1s
=> [stage-1 9/10] COPY deploy/entrypoint.sh /app/entrypoint.sh 0.1s
=> [stage-1 10/10] RUN chown -R appuser:appuser /app /home/appuser/.cache && chmod +x /app/entrypoint.sh 74.0s
=> exporting to image 33.2s
=> => exporting layers 25.4s
=> => exporting manifest sha256:c9485ab67153fc0416c6f3bc2bae4c357d13d2c55486a33cc947f337364621b7 0.0s
=> => exporting config sha256:a8337e947545ee7c2235f64af9ed0e400f60a3c29aa27c390f90fbfaa878a2fc 0.0s
=> => pushing layers 7.6s
=> => pushing manifest for <http://iad.ocir.io/axyci3pr8vxm/production-cloud-agents:p-a5qvb3muhe0-ca-tradggka3nqk-v20251210212010@sha256:c9485ab67153fc0416c6f3bc2bae4c357d13d2c55486a33cc947f337364621b7|iad.ocir.io/axyci3pr8vxm/production-cloud-agents:p-a5qvb3muhe0-ca-tradggka3nqk-v20251210212010@sha256:c9485ab67153fc0416c6f3bc2bae4c357d13d2c55486a33cc947f337364621b7> 0.1s
Deployed agente
echoing-kitchen-90064
12/10/2025, 9:46 PMlk a update-secrets --overwriteechoing-kitchen-90064
12/10/2025, 9:48 PMah sorry, i am wrong, could you instead try a new
lk a deploya
agreeable-cricket-21439
12/10/2025, 9:49 PMTrying right now
š 1
agreeable-cricket-21439
12/10/2025, 9:50 PMThere was a lk agent deploy 6 mins ago
agreeable-cricket-21439
12/10/2025, 9:53 PM@echoing-kitchen-90064 deployment just finished
e
echoing-kitchen-90064
12/10/2025, 9:56 PMhmm very strange, this agent seems to be in a weird state in our backend.
to get you unblocked could you try to create a new agent? you can move/rename the livekit.toml to keep for later.
l
loud-breakfast-60455
12/10/2025, 9:56 PMAre you deploying the same image? Could you try adding a comment in your code and build again?
e
echoing-kitchen-90064
12/10/2025, 9:56 PMah that's also a good idea, try Steve's idea first
a
agreeable-cricket-21439
12/10/2025, 9:59 PMDeploying with a comment rn
h
high-airport-95594
12/10/2025, 10:00 PMHi Livekit! Name's Raphael, I'm working with Kalil.
Thanks for helping us on this issue!
š 1
high-airport-95594
12/10/2025, 10:01 PMWorth noting that yesterday at around 2PM (26h ago), we attempted a "Rollback" from the UI, which failed.
e
echoing-kitchen-90064
12/10/2025, 10:01 PMah interesting, thanks we'll include that in our bug hunt once we get you back on track
š 1
h
high-airport-95594
12/10/2025, 10:01 PMYou mentioning "Weird state" made me recall this
a
agreeable-cricket-21439
12/10/2025, 10:02 PMAgent just deployed with the comment
h
high-airport-95594
12/10/2025, 10:05 PMWe're deploying a new agent
numa-agents-teste
echoing-kitchen-90064
12/10/2025, 10:05 PMyeah i think a new agent is the correct approach now, seems like the old one is still stuck
h
high-airport-95594
12/10/2025, 10:13 PMhttps://cloud.livekit.io/projects/p_a5qvb3muhe0/agents/CA_mkk7945jUYtX
We're fixing the missing secrets and will update SIP and Trunking Config after that for a test call.
š 1
e
echoing-kitchen-90064
12/10/2025, 10:15 PM[3:01 PM]
Worth noting that yesterday at around 2PM (26h ago), we attempted a "Rollback" from the UI, which faileddo you remember if there was a specific error message when this happened?
h
high-airport-95594
12/10/2025, 10:16 PMNo, sorry, no idea of the timestamp.
It was from nick@numa.com, not me.
high-airport-95594
12/10/2025, 10:16 PMnick.marquis@numa.com, actually
e
echoing-kitchen-90064
12/10/2025, 10:17 PMah ok, no worries. gives us a place to start at least
h
high-airport-95594
12/10/2025, 10:19 PMCommit history of deploy that worked vs the one that didn't made it says it was between 1:43PM and 1:54PM (ET)
e
echoing-kitchen-90064
12/10/2025, 10:27 PMok we'll dig into this issue, thanks for the report, let me know if you run into any other issues with the new agent
h
high-airport-95594
12/10/2025, 11:06 PMok.
ā¢
numa-agents-v2high-airport-95594
12/10/2025, 11:07 PMKeep us posted for our other, stuck agent!
š 1
e
echoing-kitchen-90064
12/11/2025, 12:26 AMit looks like your stuck agent is functional again, we haven't touched anything. i think we are tracking down a bug in our version handling
h
high-airport-95594
12/11/2025, 6:49 PMThanks for the follow-up.