This message was deleted.

Hi @Avo Sepp are you talking about passing an inline JSON to one of the variables (like OPAL_CONFIG_DATA_SOURCES) or about something else?

Yes. So, the Helm chart takes YAML and then performs

toJSON

, and then encapsulates it in single quotes. Then it creates the Environment variable OPAL_CONFIG_DATA_SOURCES on my behalf. So there’s a few levels of transformations happening here. I was hoping that

\

escaping would work, but it seems like I need to do something differently for the Helm tempalte to work. Was hoping someone had the answer on hand for the correct escape.

i don't know the answer here, i will defer to @Raz Co who might. we are actually working on moving the entire opal config to YAML in one of the next versions (as well as keeping backwards compatibility to env vars) so that it will be easier to configure from kubernetes.

That would be a massive improvement for Kubernetes use.

We hope it will 🙂

I think I need to try using

toRawJson

in the Helm chart instead of

toJson

. Because it seems like my >= <= signs are being escaped.

Hey @Avo Sepp , Unfortunately the ci is broken. We are planning to fix and improve it on our next major OPAL release like I’ve mentioned before.

I found the source of my issue and I”m not certain how to solve it yet. When I supply double quotes

"

they are being escaped by

toJson

and

toRawJson

as

\"

. But this escape makes its way all the way to the OPAL environment variable. So the query is failing. When I supply single quotes, it breaks the

squote

mechanism applied to the template.

Maybe @Ro'e Katz has some idea ? maybe we can add another temporary variable that accepts base64 encoding until we move to full YAML

@Avo Sepp There’s no need to pass it as Json, pass it as yaml!

dataConfigSources:
    config:
       entries:
       - url: <http://opal-server:7002/policy-data>
         topics: ["policy_data"]
         dst_path: "/static"

I will try this today! The problem is the Helm YAML to JSON parser adding escape characters, which it needs to do to prevent formatting errors, but in this case breaks the query. I'll get back to you if it works.

@Ro'e Katz @Raz Co can you please assist here?

@Avo Sepp I’m sorry but I still don’t understand what you’re trying to do. Through what value are you trying to pass that query, and why?

dataConfigSources

is used to configure data sources for opal to pull data from, which it than loads into OPA.

I have a high confidence that it is not possible to configure dataConfigSources which include queries with quotes in them, if you are using Helm to set those values. At least not without changes to OPAL. If OPAL can now accept raw YAML instead of JSON, then the JSON formatter should be removed from the Helm chart and we should try and submit YAML to OPAL. I may have to pop on a call to explain why I believe this is impossible to do with the Helm template tool.

My confusion is not about the possibility of escaping the query, but that I don’t see what this query have to do with dataConfigSources in the first place (and under what sub-key you try to pass it? As inline data?). Lmk when you’re available, we can definitely hop on a call :)

https://github.com/permitio/opal-fetcher-postgres The fetcher paradigm includes the query in data config sources. Both the postgres one released by your team, and the CosmosDB one I wrote, include the query in dataConfigSources.

Thanks @Avo Sepp, now I’m on the same page 🙂 I don’t have a full setup with Cosmos, but looking at client’s logs it seems like everything passed correctly when my

myvalues.yaml

file is:

server:
  dataConfigSources:
    config:
      entries:
        - dst_path: "/static"
          topics: ["policy_data"]
          url: "..."
          config:
            fetcher: CosmosFetchProvider
            query: '{"Id":demo.value.Id,"Abbr":demo.value.Abbr} AS Name'

Have you tried something like that? (That works with either

toJson

or

toRawJson

)

If I try and wrap the query in single quotes, I will get a key error using the official Helm chart.

My example uses the official helm chart! Sorry missed your message, but I’m available for the next hour if you’re still up for a call

So what I’m thinking of doing, is using Kustomize to post-render the Helm chart. See if I can overwrite this after Helm creates the original deployment.