https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# opal
s
This message was deleted.
d
Copy code
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.312553+0000 | opal_client.data.updater                | INFO  | Triggering data update with id: 853bdc25d8d1434a94d64f39e31d1515
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.313032+0000 | opal_client.data.updater                | INFO  | Fetching policy data
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.313605+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: <postgresql://postgres@example_db:5432/postgres>
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.318516+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server <ws://opal_server:7002/ws>
bip-baaz-opal-opal_server-1        | 2023-05-17T18:31:08.326523+0000 | opal_common.git.bundle_maker            | INFO  | Using root manifest dir path (new-fashioned): '.'
bip-baaz-opal-opal_server-1        | 2023-05-17T18:31:08.326899+0000 | opal_common.git.bundle_maker            | INFO  | Compiling manifest file .manifest
bip-baaz-opal-opal_server-1        | 2023-05-17T18:31:08.332779+0000 | opal_common.git.bundle_maker            | INFO  | Compiling manifest file .manifest -> utils/.manifest
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.359165+0000 | opal_fetcher_postgres.provider          |DEBUG  | PostgresFetchProvider fetching from <postgresql://postgres@example_db:5432/postgres>
bip-baaz-opal-opal_server-1        | 2023-05-17T18:31:08.362099+0000 | opal_common.git.bundle_maker            | INFO  | Compiling manifest file .manifest -> rest/.manifest
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.374788+0000 | opal_client.data.updater                | INFO  | Saving fetched data to policy-store: source url='<postgresql://postgres@example_db:5432/postgres>', destination path='/participants'
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.389636+0000 | charset_normalizer.api                  |DEBUG  | Encoding detection on empty bytes, assuming utf_8 intention.
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.391185+0000 | opal_client.opa.logger                  | INFO  | Received request.    PUT /v1/data/participants
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.392441+0000 | opal_client.opa.logger                  | INFO  | Sent response.       PUT /v1/data/participants -> 204
bip-baaz-opal-opal_server-1        | 2023-05-17T18:31:08.392843+0000 | opal_common.git.bundle_maker            |WARNING |   Path '.' is outside current .manifest directory
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.394794+0000 | opal_client.policy_store.opa_client     |DEBUG  | processing store transaction: {'id': '853bdc25d8d1434a94d64f39e31d1515', 'actions': ['set_policy_data'], 'transaction_type': <TransactionType.data: 'data'>, 'success': True, 'error': '', 'creation_time': '2023-05-17T18:31:08.370894', 'end_time': '2023-05-17T18:31:08.394555', 'remotes_status': [{'remote_url': '<postgresql://postgres@example_db:5432/postgres>', 'succeed': True, 'error': 'None'}]}
bip-baaz-opal-opal_server-1        | 2023-05-17T18:31:08.410901+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 172.25.0.5:56066 - "GET /policy?path=. HTTP/1.1" 200
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.415339+0000 | opal_client.policy.fetcher              | INFO  | Fetched valid bundle, id: 43dd17c961b50da8c2c7d1940e68f6106263b6ed
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.416298+0000 | opal_client.policy.updater              | INFO  | Got policy bundle with 49 rego files, 1 data files, commit hash: '43dd17c961b50da8c2c7d1940e68f6106263b6ed'
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.420465+0000 | opal_client.opa.logger                  | INFO  | Received request.    GET /v1/policies
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.421071+0000 | opal_client.opa.logger                  | INFO  | Sent response.       GET /v1/policies -> 200
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.431782+0000 | opal_client.opa.logger                  | INFO  | Received request.    PUT /v1/data
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.432902+0000 | charset_normalizer.api                  |DEBUG  | Encoding detection on empty bytes, assuming utf_8 intention.
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.433725+0000 | opal_client.opa.logger                  | INFO  | Sent response.       PUT /v1/data -> 204
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.441981+0000 | opal_client.opa.logger                  | INFO  | Received request.    PUT /v1/policies/utils/string.rego
bip-baaz-opal-opal_client-1        | 2023-05-17T18:31:08.444275+0000 | opal_client.opa.logger                  | INFO  | Sent response.       PUT /v1/policies/utils/string.rego -> 200
you can see it fetches the postgres data, does PUT /v1/data/participants, then it gets the policies and does /put/data
so it seems that having a data.json in my repository is causing the postgres data to be overwritten
Aha, seems to be a timing thing... if the policies take longer to retrieve, the call to PUT /v1/data might be after PUT /v1/data/participants. Below is the log when using the sample policy repo:
Copy code
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.042843+0000 | opal_client.data.updater                | INFO  | Fetching policy data
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.043765+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: <postgresql://postgres@example_db:5432/postgres>
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.048093+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server <ws://opal_server:7002/ws>
bip-baaz-opal-opal_server-1        | 2023-05-17T18:52:54.055048+0000 | opal_common.git.bundle_maker            | INFO  | Using root manifest dir path (new-fashioned): '.'
bip-baaz-opal-opal_server-1        | 2023-05-17T18:52:54.055241+0000 | opal_common.git.bundle_maker            | INFO  | Compiling manifest file .manifest
bip-baaz-opal-opal_server-1        | 2023-05-17T18:52:54.058554+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 172.25.0.5:56002 - "GET /policy?path=. HTTP/1.1" 200
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.084930+0000 | opal_client.policy.fetcher              | INFO  | Fetched valid bundle, id: f10608f2d759f1982c1e0d9eb7048d771cea4f2f
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.086008+0000 | opal_client.policy.updater              | INFO  | Got policy bundle with 2 rego files, 1 data files, commit hash: 'f10608f2d759f1982c1e0d9eb7048d771cea4f2f'
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.090322+0000 | opal_client.opa.logger                  | INFO  | Received request.    GET /v1/policies
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.091526+0000 | opal_client.opa.logger                  | INFO  | Sent response.       GET /v1/policies -> 200
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.094447+0000 | opal_fetcher_postgres.provider          |DEBUG  | PostgresFetchProvider fetching from <postgresql://postgres@example_db:5432/postgres>
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.095777+0000 | opal_client.opa.logger                  | INFO  | Received request.    PUT /v1/data
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.096312+0000 | opal_client.opa.logger                  | INFO  | Sent response.       PUT /v1/data -> 204
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.097916+0000 | charset_normalizer.api                  |DEBUG  | Encoding detection on empty bytes, assuming utf_8 intention.
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.108535+0000 | opal_client.opa.logger                  | INFO  | Received request.    PUT /v1/policies/utils.rego
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.113937+0000 | opal_client.data.updater                | INFO  | Saving fetched data to policy-store: source url='<postgresql://postgres@example_db:5432/postgres>', destination path='/participants'
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.118608+0000 | opal_client.opa.logger                  | INFO  | Sent response.       PUT /v1/policies/utils.rego -> 200
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.124224+0000 | opal_client.opa.logger                  | INFO  | Received request.    PUT /v1/data/participants
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.129850+0000 | charset_normalizer.api                  |DEBUG  | Encoding detection on empty bytes, assuming utf_8 intention.
bip-baaz-opal-opal_client-1        | 2023-05-17T18:52:54.131198+0000 | opal_client.opa.logger                  | INFO  | Sent response.       PUT /v1/data/participants -> 204
In this case it called PUT /v1/data before PUT /v1/data/participants (then followed by more policy files) and I can see my participant data properly.
So is there a way to force it to load the policies (and static policy data) first, then the dynamic sources of data?
my other option I guess would just to move all my static data to a directory like /static then update all of my policies to reference data.static.*
o
Hi David, as a quick guess here- are you triggering the update event right with the starting the OPAL-client; if you wait for it’s readiness check to stabilize first, you’d avoid this issue.
Let me know if I got it wrong / if you have different expectations
d
Hi Or, I'm not triggering a data update in this case, this is the initial load. For the OPAL server I have
Copy code
OPAL_DATA_CONFIG_SOURCES={"config":{"entries":[{"url":"<postgresql://postgres@example_db:5432/postgres>","config":{"fetcher":"PostgresFetchProvider","query":"SELECT * from participant_data;","connection_params":{"password":"postgres"}},"topics":["participant_data"],"dst_path":"/participants"}]}}
and in the client
Copy code
-OPAL_FETCH_PROVIDER_MODULES=opal_common.fetcher.providers,opal_fetcher_postgres.provider
      - OPAL_DATA_TOPICS=participant_data
basically just a slight modification of the postgres example in that I have changed the topic name and the data in the db. I would expect this needs to occur before the readiness as I cannot make decisions without this data. The problem is that the policy and data appear to be retrieved asynchronously and the policy writes to PUT /v1/data after the postgres was written to PUT /v1/data/participants
Seems like it would be best to make sure the PUT /v1/data happens prior to any of the data in OPAL_DATA_CONFIG_SOURCES
o
Hi @Asaf Cohen , @Shaul Kremer can you chime in here ?
s
@David Hamilton This sounds like the right way to solve that, but it'll take some time (though if you want to fix it yourself we'd love a PR). In the mean time, I suggest you do as you previously suggested and put your static data under data.static.
d
well i was thinking about a commit, but kind of looks like @Ori Shavit already did this? https://github.com/permitio/opal/commit/e7ee37863afcee54b77f4e86bace6202fdf78783 Looks like if I set OPAL_SPLIT_ROOT_DATA=true it may resolve my issue! Going to try that now
s
That's not out yet in the most recent release (which came out last week), but you can build your own Docker image with the current development code with:
Copy code
git clone <https://github.com/permitio/opal.git>
cd opal
docker build -t custom_opal_server --target server -f docker/Dockerfile .
d
Ah thanks. I am not working today so I'll give that a try tomorrow. I appreciate the help!
s
No problem! Let me know if it works.
d
looking into it now, but getting this failure during build
Copy code
Step 11/75 : RUN cd /tmp/cedar-agent && 	cargo build ${cargo_flags} && 	cp /tmp/cedar-agent/target/*/cedar-agent /
 ---> Running in 6de99a012d55
error: could not find `Cargo.toml` in `/tmp/cedar-agent` or any parent directory
The command '/bin/sh -c cd /tmp/cedar-agent && 	cargo build ${cargo_flags} && 	cp /tmp/cedar-agent/target/*/cedar-agent /' returned a non-zero code: 101
do i also need to clone and build cedar-agent to work with this newer version?
ok added these steps prior to the 
docker build
Copy code
git clone <https://github.com/permitio/cedar-agent.git>
cd cedar-agent
git checkout 1838635f16ba6db60d16c2ca28cb257e970bdff0 
cd ..
but now getting an ssl error
Copy code
Step 11/75 : RUN cd /tmp/cedar-agent && 	cargo build ${cargo_flags} && 	cp /tmp/cedar-agent/target/*/cedar-agent /
 ---> Running in c66ac1acea36
    Updating <http://crates.io|crates.io> index
 Downloading crates ...
error: failed to download from `<https://crates.io/api/v1/crates/aead/0.5.2/download>`

Caused by:
  [60] SSL peer certificate or SSH remote key was not OK (SSL certificate problem: unable to get local issuer certificate)
The command '/bin/sh -c cd /tmp/cedar-agent && 	cargo build ${cargo_flags} && 	cp /tmp/cedar-agent/target/*/cedar-agent /' returned a non-zero code: 101
a
Hi @David Hamilton, try to run in your local clone:
Copy code
git submodule init
git submodule update
this will initialize the git submodule of the cedar-agent and clone that code, and you would be able to build the docker image
i am not familiar with the SSL certificate issue, maybe it's related to your workaround? please try the submodule commands and see if the docker manages to build
d
thanks i figured that wasn't the best way to get the cedar-agent
still getting the ssl error, but it's probably something specific to my machine setup so i'm looking into it
2 Views
Open in Slack
PreviousNext
Powered by