https://linen.dev logo
Powered by
So there is a scenario when it s
# general
p
So there is a scenario when it's possible to start a workspace (@ first time creation and @ restart) with an expired 
data.coder_workspace.me.owner_oidc_access_token
We use the OIDC access token to interact w/ Gitlab API on first-time workspace creation and startup. - So the workspace restarts (say by schedule) isn't affecting us. - But since Gitlab OIDC tokens expire after 2hours (since Gitlab v15), it is possible that the a user logged in to Coder more than 2 hours ago and then creates a workspace after the Gitlab OIDC access token has expired. Then our attempts to use that token from agent-startup script fails. Is it possible to do one of the following? - Make the Coder auth expiration time window (CLI and UI) configurable? - Then the admins can set it to their desired value (in our case the same as OIDC auth expiration length)matching - If the OIDC auth has expired, trigger a re-login?
k
Do you allow refresh for the OIDC token? We automatically refresh the token if it's permitted, so the 2hrs expiration might not apply.
p
Is that a config option on Coder side?
k
Humph, I assumed it would be an option in GitLab, but maybe something weird is going on here.
So we attempt to refresh if the token is expired... maybe that isn't working as expected though.
p
Gitlab API has the capability to refresh the OIDC token, I don't see an option to disable it on Gitlab side
Should I submit a bug ticket?
k
Well to confirm, if you perform a workspace restart within the two hour window everything works as expected. Only when you exceed the two hour window you need to re-authenticate?
p
yep, everything was working fine and we upgraded Gitlab to v15 (before that tokens never expired)... and then we noticed that workspaces that are scheduled to auto start on schedule were failing...
we don't care so much for solving this problem for restart scenario as we don't really need the OIDC token upon restarts. So we updated the agent-startup script to not run that code block after a successful first time workspace initialization.
The problem we can't handle on our end is, 1. user logs into coder ui 2. user waits more than 2hrs (and oidc token expires) 3. user creates a new workspace 4. coder provides expired token to coder agent-startup script
and to be explicit, - creating new workspaces - and restarting existing workspaces get a functioning oidc token within 2hours
k
@peterpathirana any shot you could enter your PSQL database so we can debug this a bit more? The condition for us refreshing is: The 
access_token
is expired, the expiry time is greater than zero, and the 
refresh_token
isn't an empty string. You can get this by doing the following query:
Copy code
sql
SELECT * FROM user_links WHERE login_type = 'oidc';
And paste me the output (with the access tokens scrubbed away).
p
Absolutely... I will get that to you on Monday! also the tokens would have been expired by then 😄
k
Awesome. Thank you!
p
Good morning Kyle... Ran the query you gave adjusted to hide tokens...
For example, this user is me:
Copy code
9488370d-0f91-4e42-a2e1-79ed53812dba | oidc       | 216        | 7a47  | 34d7  | 2023-04-21 23:59:05.939328+00
My workspace container started this morning on schedule (before I ever logged into UI) w/ environment variable populated by 
data.coder_workspace.me.owner_oidc_access_token
set to empty string. If I log out from UI, and re-login and restart, that environment variable will have a value set and it will work against Gitlab API.
Any update on what else we can try here?
Also see this error when workspaces are started by the scheduler:
Copy code
/var/log/syslog.1:Apr 27 08:22:56 d-coder-demo coder[2261413]: 2023-04-27 08:22:56.528 [INFO]#011(coderd)#011<./coderd/workspaceagents.go:194>#011(*API).postWorkspaceAgentStartup#011post workspace agent version#011{"request_id": "bc58cfd1-b004-44ae-9223-64152bd07b7f", "agent_id": "4dd315fa-2f21-4ca8-9706-2e36b068e878", "agent_version": "v0.20.1+ed9a3b9"}
/var/log/syslog.1:Apr 27 08:22:56 d-coder-demo coder[2261413]: 2023-04-27 08:22:56.768 [WARN]#011<./provisionerd/provisionerd.go:330>#011(*Server).acquireJob#011acquire job ...
/var/log/syslog.1:Apr 27 08:22:56 d-coder-demo coder[2261413]:   "error": request job was invalidated: obtain OIDC access token: update user link: unauthorized: forbidden
/var/log/syslog.1:Apr 27 08:22:56 d-coder-demo coder[2261413]:            #011storj.io/drpc/drpcwire.UnmarshalError:26
/var/log/syslog.1:Apr 27 08:22:56 d-coder-demo coder[2261413]:            #011storj.io/drpc/drpcstream.(*Stream).HandlePacket:198
/var/log/syslog.1:Apr 27 08:22:56 d-coder-demo coder[2261413]:            #011storj.io/drpc/drpcmanager.(*Manager).manageReader:216
k
Ahh, that's a bug on our end! I'll make sure this is fixed.
Those logs are helpful. On it!
p
Thank you!
k
I believe this is fixed in v0.23.0. Could you try updating?
p
Hi Kyle, we just updated yesterday and can confirm today that it is indeed fixed. Thank you so much.
k
Wonderful!
p
Actually, I have to take that back... What appears to have been fixed is that workspaces no longer fail to start up on schedule. But the OIDC credential injected into the workspace env does not appear to be current.
Copy code
curl -fsSL -H "Authorization: Bearer <oidc-token>" "https://<self-hosted-gitlab>/api/v4/user"
curl: (22) The requested URL returned error: 401
I am double checking this and will confirm in 2 hours...
Test plan: - log-out/log-in to UI to generate new oidc token - restart workspace w/ valid token - confirm oidc token works against gitlab rest api - then wait 2 hours for the token to expire - restart workspace (using CLI instead of UI) - check if the newly injected oidc token works against gitlab api
PreviousNext
Powered by