This message was deleted.
# opals
Slackbot
08/18/2023, 6:08 PMThis message was deleted.
d
David Hamilton
08/18/2023, 6:09 PMMy previous solution to this was to just run OPA non-embedded. For simplicity for all the teams on my project that will be starting to use OPA(L), though, I would love to be able to use embedded and simplify the charts for them
o
Or Weis
08/18/2023, 6:42 PMHi @David Hamilton, you might be able find your previous thread in the archives:
https://www.linen.dev/s/permit/c/general
Or Weis
08/18/2023, 6:45 PMI definitely think we can investigate this and come up with a potential solution.
@Asaf Cohen do you have any suspects here?
David Hamilton
08/18/2023, 6:48 PM"Ro'e Katz
05/16/2023, 4:01 AM
There’s a known issue of long loading times when having a lot of rego files - OPA is currently recompiling everything every time you load a new policy file."
David Hamilton
08/18/2023, 6:50 PMNew info: this ONLY occurs if OPA (not OPAL) log level is debug or info
David Hamilton
08/18/2023, 6:51 PMif I set to error everything appears fine
o
Or Weis
08/18/2023, 6:52 PMOh - yeah - makes total sense.
Seems you should be able to circumnavigate the issue by simply waiting longer on the health-check
d
David Hamilton
08/18/2023, 6:53 PMIt's just strange that this only occurs with embedded OPA. I have a separate OPA container running with debug and console decision logs and it behaves just fine
David Hamilton
08/18/2023, 6:54 PMsame OPA version and everything
o
Or Weis
08/18/2023, 7:02 PMThis might be a combination of issue coming together;
Maybe there’s an additional slow down when the OPA logs are been piped into the OPAL-client process; which ends up creating a race-condition for the healthcheck.
We can potentially add an option to use the opa-runner but without piping the OPA logs
Or Weis
08/18/2023, 7:05 PMIt might be best to wait for @Asaf Cohen/ @Shaul Kremer / @Ro'e Katz to chime in (Which should be on Sunday)
Or Weis
08/18/2023, 7:10 PM@David Hamilton, Maybe in the meanwhile you can test the theory by running inline OPA but with logs sent to file instead of STDERR
d
David Hamilton
08/18/2023, 7:28 PMThanks, @Or Weis
o
Or Weis
08/22/2023, 8:52 AMSeems the rest of the team is stumped about this issue as well.
Currently I think the option of running OPA with either a different log level, or with with logs piped to a file are is our best course
r
Ro'e Katz
08/22/2023, 9:09 AM@David Hamilton Just went over the original thread - My quote regarding long loading times seems irrelevant as you mentioned logs indicating all policies were already loaded.
I’ll try to have another look at this, see if I can find something wrong
Ro'e Katz
08/22/2023, 10:00 AMI might have found something. We use
stream.readline()\nd
David Hamilton
08/22/2023, 11:21 AMI think you're on the right track for sure. If opa logging is set to debug, it logs the entire policy file with newlines replaced by "\n" so very long lines. When it is in info log mode, it only logs that the request was made
r
Ro'e Katz
08/22/2023, 11:41 AMGonna release an RC for you to test soon…
Ro'e Katz
08/22/2023, 12:23 PM@David Hamilton Pushed image tag
0.7.4-rc🙏 1
🦜 1
d
David Hamilton
08/22/2023, 1:10 PMsure, I'll get my configs back to the failing scenario and confirm that, then switch the image tag
David Hamilton
08/22/2023, 3:23 PM@Ro'e Katz Thank you! That appears to have fixed it! I have to figure out if there is a way I can get OPA to not log large requests like that in the first place. I want console logs, but not the entire content of the policies OPAL sends to be logged, or the data that OPAL sends!
But that's a matter for OPA. Looks like this issue is resolved 😄
r
Ro'e Katz
08/22/2023, 6:01 PMGreat! I would work to merge this and soon release a new OPAL version