<@U07FRRZH2GN> getting 401 unauthorized while gene...
# server-apif
fierce-answer-55580
09/02/2025, 3:37 PM@icy-policeman-64898 getting 401 unauthorized while generating token. It was working just fine earlier
r
refined-appointment-81829
09/02/2025, 3:42 PMThis is nearly always LiveKit credentials not set correctly or the system clock is not correct.
b
bored-motorcycle-22550
09/02/2025, 5:20 PM@refined-appointment-81829 Highjacking this thread, but related to mine above it:
I tried debugging both these at the time. I verified the claims map on the issued token contained a valid expiration. It was done correctly, and showed the token was valid for 1h. (You can check this yourself comparing the Slack timestamp when I posed to the claim.) Additionally, I rotated and verified the LK credentials used were correct.
Expires At: Tuesday, September 2, 2025 85957 AM GMT-07:00 DST
Despite both those, I encountered the 401 issue on multiple projects. Then suddenly both started working again. Now seeing another user has this issue at the same time, I really encourage someone to check logs and see if there's an outlier on LK's side.
Any information to prevent this happening again is valuable to me and would be much appreciated! 🙏
r
refined-appointment-81829
09/02/2025, 5:21 PMdid you check
nbfrefined-appointment-81829
09/02/2025, 5:22 PMMost of the time when I have seen this the system clock is a little fast and try and use the token before it becomes valid.
b
bored-motorcycle-22550
09/02/2025, 5:23 PMNo, looks good there too, I'll convert that real quick just to make it convenient to debug.
NBF: Tuesday, September 2, 2025 75957 AM GMT-07:00 DST
bored-motorcycle-22550
09/02/2025, 5:24 PMToken was generated about 8:03 maybe 8:04am my time.
r
refined-appointment-81829
09/02/2025, 5:24 PMok, need to know down to the milli second to check its validity
b
bored-motorcycle-22550
09/02/2025, 5:25 PMI have logs and can get that info for you, one moment.
r
refined-appointment-81829
09/02/2025, 5:25 PMWhy was the nbf generated for past?
refined-appointment-81829
09/02/2025, 5:25 PMseems fishy
refined-appointment-81829
09/02/2025, 5:27 PMdoes it fail 100% or intermittent?
b
bored-motorcycle-22550
09/02/2025, 5:28 PM2025-09-02 075951.760 PT was the exact request timestamp in GCP
r
refined-appointment-81829
09/02/2025, 5:29 PMthat looks like the issue
refined-appointment-81829
09/02/2025, 5:29 PMGlad you found the issue.
b
bored-motorcycle-22550
09/02/2025, 5:31 PMJust so I understand correctly, the solution would be to set the NBF claim with a margin large enough to not cause issues with other systems? Given a clock desync of a couple seconds may be possible?
bored-motorcycle-22550
09/02/2025, 5:31 PM(I obviously don't control the internal clocks at Cloud Run lol)
r
refined-appointment-81829
09/02/2025, 5:31 PMno, the correct solution is to keep systems clock in sync with ntp
b
bored-motorcycle-22550
09/02/2025, 5:33 PMAlright, well that safely puts this outside the scope of LK then. I appreciate the direction @refined-appointment-81829 and should be able to figure out something to keep this from happening. I appreciate the quick turnaround!
👍🏼 1
bored-motorcycle-22550
09/02/2025, 6:01 PM@refined-appointment-81829 Off the phone here with my GCP account rep. They have had desynchronized clocks reported to their Cloud Run offering by multiple customers today. Cloud Run VMs are supposed to be automatically synchronized with Google's Public NTP. Looks like that's temporarily degraded for some customers today.
If you see this issue reported again by other LK customers today, check if they are on GCP.
r
refined-appointment-81829
09/02/2025, 6:10 PMThank you for letting me know.
🙌 1
2 Views