Hi, We’re experiencing weird behaviour on DataHub...
# troubleshoot
l
Hi, We’re experiencing weird behaviour on DataHub v. 0.10.0 with SSO functionality enabled. It manifests with
502 Bad Gateway
error message. Steps to reproduce (updated Chrome browser, MacOS/Win): 1. Login using SSO to DataHub - success 2. Play around for a while [optional] 3. Sign out from the UI 4. Once back on the login screen - re-login using SSO. Step #4 gets us the above mentioned 502 error. To us, the probable scenario looks like an internal error in DataHub. We’ve noticed
PLAY_SESSION
cookie holding the user currently logged in is not removed after the sign out. The browser still sends this cookie even though the session has already expired on the server. The latter refrains from responding to the request sent with an inactive cookie leading the load balancer to respond with an 502 error message. Any suggestions on how we could fix the problem? Thanks in advance!
a
Hi @lively-spring-5482 - could you share what SSO provider you’re using? CC: @big-carpet-38439
l
We’re connecting to OneLogin
c
hello, using Okta, I'm getting the same behavior on v0.10.0
s
Hello Team , we are facing same isssue with Google SSO . Any recommendations on this ?
a
Hi All, this is a known issue we’re working on getting a fix out for! https://github.com/datahub-project/datahub/issues/7144
For some providers it’s currently bugged
s
Thank you @astonishing-answer-96712, will track the issue .
For us the workaround is to open the session from an incognito window , This is working fine for us , but user experience is bad while adopting the workaround . Question : Is there any way we can control the session timeout or eliminating the sessions from backend of datahub , so it will ask for SSO ? cc: @astonishing-answer-96712
c
Hi, I pushed a temp fix for the 502 errors when hitting
/sso
. It looks to be working on my env now. The drawback is shared resource links will work for pre-authenticated users only, otherwise it lands on main page. But, it's better then getting the 502. https://datahubspace.slack.com/archives/C017W0NTZHR/p1678202530939229 hope that helps
s
Hello Team , Happy monday . This sso issue looks like still affecting us . It would be great if any recommendations or workaround is available . Thank you .
c
Hi Milan, for now I'm running the
datahub-frontend
with the pr change I posted, but Jjoyce pointed out it might be some Nginx issue when routing the
302
response with
location
header. But, I haven't had the chance to dig into it yet.
a
Please let us know if y’all trace it back to Nginx!
h
Hi all, any update on this issue?
b
Are you all using an additional reverse proxy in front of datahub?
h
I think it's also nginx. Deleting cookies resolved the problem. Could it be the datahub upgrade that creates the issue (cookie from old version)?
g
As someone who just went down this rabbit hole: it most likely is nginx! • check your nginx controller’s logs and you’ll likely see errors about large request headers from your datahub endpoint. Something like ◦
upstream sent too big header while reading response header from upstream
• check your nginx.conf and you’ll probably have a default
proxy_buffer_size
of
4k
. • increase this to
128k
or some reasonable size. ◦ if using helm to deploy nginx, update
values.controller.config.proxy-buffer-size
and
values.controller.config.proxy-buffers-number
accordingly ◦ if using argoCD + helm, update the source.helm section of your argo app spec:
Copy code
chart: ingress-nginx
    helm:
      parameters:
      - name: controller.extraArgs.enable-ssl-passthrough
        value: 'true'
      - name: controller.config.proxy-buffer-size
        value: '128k'
      - name: controller.config.proxy-buffers-number
        value: '4'
hopefully this helps anyone who runs into this!
plus1 1