https://datahubproject.io logo
Join Slack
Powered by
Hi all, I met a problem when log in datahub after ...
# troubleshoot
a
Hi all, I met a problem when log in datahub after I enabled SSO for datahub. here is the error msg in browser console: <h1>Bad Message 431</h1><pre>reason: Request Header Fields Too Large</pre> could anyone help me?
g
Hey @adorable-tomato-97942 - ive seen issues like this when someone has set up a custom deployment rather than following our helm charts. How have you deployed datahub?
a
Hi Gabe, after some investigation, we think it may caused by the header size limitation of Jetty used in GMS service. Is there any way to change this header size configuration?
@green-football-43791
g
Oh-- is your user in many groups by any chance? We have noticed that for users in many many groups this issue can happen.
Hmm 🤔 I am not aware of how that can be altered
a
yes, we enabled SSO for datahub, the cookies could be very large if a user logged in many sites through SSO.
do we have to change the source code if we want to change that header size?
g
I see- its possible there is an env variable that can address this. @big-carpet-38439 do you happen to know if there’s a flag we can set to help Ethan here?
Let me also look into this Ethan.
a
OK, thanks a lot!
g
Can you log in with incognito mode out of curiosity?
and does this happen for all users or just a subset?
a
I’ll try the incognito mode. currently, this problem happens for many users, I think it would be a common problem if we bring datahub to PROD for all users.
g
Understood- we’ll definitely make sure to get to the bottom of this for you Ethan.
Trying in incognito mode would at least give us some more diagnostic information
a
OK, thank you Gabe, let me try incognito mode now.
Hi Gabe, incognito mode works fine, I can log in with incognito mode.
b
@adorable-tomato-97942 Do you know why the cookie would be getting so large? It's interesting that it works in incognito mode
g
@adorable-tomato-97942 snooping around I also found this:
Copy code
play.server.akka.max-header-value-length
i can try to see if that would work later today
could you tell us how many kb your header is ethan?
a
2022-03-10 122425.185WARNoejh.HttpParserqtp1615780336 78 Header is too large 8193>8192
above is the WARN log from the GMS
I am not sure if the play.server.akka.max-header-value-length will work for us, because we’ve tried this configuration before but it did not work then. we noted that it is GMS service report this error rather than datahub-frontend.
g
Ah ok, thanks for that additional context Ethan.
Can you share the full stacktrace from GMS?
m
I have experienced something similar on my side with Azure SSO. Not exactly the same error, but my 
PLAY_SESSION
cookie was too large. As Gabe said, my user belonged to many groups and when ever the 
groups
claim was enabled in SSO, the cookie would be too large. But I had the same behavior with or without incognito. I ended up dropping the groups feature and used roles mapping in Azure to achieve what I needed. And in my case, the server was fine, the cookie was rejected by the browser...
a
Finally, we found a workaround that works for us quite well, @some-hydrogen-44606 Hi zhongjie, could you please update how you fix this problem here?
s
We solved this problem with the Jetty configuration in GMS.
jetty.xml
Copy code
<Set name="connectors">
   <Array type="org.eclipse.jetty.server.Connector">
     <Item>
       <New class="org.eclipse.jetty.server.ServerConnector">
         <Arg name="server"><Ref refid="Server" /></Arg>
  <Arg name="acceptors" type="int"><Property name="jetty.http.acceptors" deprecated="http.acceptors" default="-1"/></Arg>
         <Arg name="selectors" type="int"><Property name="jetty.http.selectors" deprecated="http.selectors" default="-1"/></Arg>
  <Arg name="factories">
    <Array type="org.eclipse.jetty.server.ConnectionFactory">
      <Item>
   <New class="org.eclipse.jetty.server.HttpConnectionFactory">
     <Arg name="config">
       <New class="org.eclipse.jetty.server.HttpConfiguration">
         <Set name="requestHeaderSize">100000</Set>
       </New>
     </Arg>
   </New>
      </Item>
    </Array>
  </Arg>
</New>
     </Item>
   </Array>
 </Set>
GMS start.sh
Copy code
COMMON="
    $WAIT_FOR_EBEAN \
    $WAIT_FOR_KAFKA \
    $WAIT_FOR_NEO4J \
    -timeout 240s \
    java $JAVA_OPTS $JMX_OPTS \
    $OTEL_AGENT \
    $PROMETHEUS_AGENT \
    -jar /jetty-runner.jar \
    --jar jetty-util.jar \
    --jar jetty-jmx.jar \
    --config /datahub/datahub-gms/scripts/jetty.xml \
    /datahub/datahub-gms/bin/war.war"
b
This is awesome! Thank you
I think we should consider making this the default
To avoid this pitfall later on
g
Definitely 👍 @some-hydrogen-44606 would you be able to contribute this fix back to Datahub?
s
Sure
g
Amazing!!
b
This would be greatly appreciated- thank you!
s
My pleasure. Here is the pull request: https://github.com/datahub-project/datahub/pull/4435
🙌 1
b
Thank you! Will have a close look soon
b
Just found this thread - I have the same issue and was wondering whether this will be merged?
b
Its merged in 🙂
🎉 1
b
Do you have an idea when this will be included in a release?
b
@bumpy-activity-74405 Should be today!
🎉 1
Open in Slack
PreviousNext
Powered by