https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# opal
s
This message was deleted.
o
HI @raghanag Yes you can. The easiest way is to override the higher key, replacing everything under it, you can also create (or fork) a custom data fetcher to give you more control of how you shape the data being loaded There’s also a PR coming for adding more flexibility around data loading and deleting https://github.com/permitio/opal/pull/447 CC: @Ro'e Katz
r
thank you @Or Weis, can we also do the same for rego policies too (create/edit/delete)
o
Yes, OPAL would reflect the changes you have in Git (or in the API bundle server)
r
what i meant is instead of Git, can we just use OPAL server API to push data and policies instead of relying on Git
a
For policy sources, we currently only support: 1. git repository 2. http server that serves policy bundles in tar.gz
if you want to use the second option, this is the tutorial you need: https://docs.opal.ac/tutorials/track_an_api_bundle_server
r
hi @Asaf Cohen thanks for sharing, i changed the OPAL server env vars to a different repo than the example one that comes by default, and restarted the pods but the query to OPA still gets me old data. May i ask if I need to do anything on the client side to get the new policies and data to replace the old one https://docs.opal.ac/getting-started/running-opal/run-opal-server/policy-repo-location
Copy code
curl --request GET '<http://localhost:8181/v1/data/users>' --header 'Content-Type: application/json'
{"result":{"alice":{"location":{"country":"US","ip":"8.8.8.8"},"roles":["admin"]},"bob":{"location":{"country":"US","ip":"8.8.8.8"},"roles":["employee","billing"]},"eve":{"location":{"country":"US","ip":"8.8.8.8"},"roles":["customer"]},"sunil":{"location":{"country":"US","ip":"8.8.8.8"},"roles":["guest"]}}}
o
Hi @raghanag can you share the server and client logs. Need to see that the client successfuly connected and subscribed to the server
r
client logs:
Copy code
|ERROR  | OPA transaction failed, transaction id=5b716c71cbd7439bbc8e92e046182fdb, actions=[], error=ValueError('unexpected response code while fetching bundle: 503')
2023-05-17T06:19:47.876104+0000 | opal_client.policy_store.opa_client     | INFO  | processing store transaction: {'id': '5b716c71cbd7439bbc8e92e046182fdb', 'actions': [], 'transaction_type': <TransactionType.policy: 'policy'>, 'success': False, 'error': "ValueError('unexpected response code while fetching bundle: 503')", 'creation_time': '2023-05-17T06:19:47.875365', 'end_time': '2023-05-17T06:19:47.875411', 'remotes_status': [{'remote_url': '<http://titanopal-server:7002/policy>', 'succeed': False, 'error': "ValueError('unexpected response code while fetching bundle: 503')"}]}
2023-05-17T06:19:47.876433+0000 | opal_client.policy_store.opa_client     | INFO  | persisting health check policy: ready=true, healthy=true
2023-05-17T06:19:47.876547+0000 | opal_client.policy_store.opa_client     | INFO  | Policy and data statistics: policy: (successful 830, failed 427);	data: (successful 1259, failed 0)
2023-05-17T06:19:47.888489+0000 | opal_client.policy.fetcher              | INFO  | Fetching policy bundle from <http://titanopal-server:7002/policy>
2023-05-17T06:19:47.956369+0000 | opal_common.utils                       |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
2023-05-17T06:19:47.957126+0000 | opal_client.policy.fetcher              |WARNING | Failed all attempts to fetch bundle, got error: ValueError('unexpected response code while fetching bundle: 503')
2023-05-17T06:19:47.957441+0000 | opal_client...base_policy_store_client  |ERROR  | OPA transaction failed, transaction id=71db3f45a10b4cc48523c40850125535, actions=[], error=ValueError('unexpected response code while fetching bundle: 503')
2023-05-17T06:19:47.957653+0000 | opal_client.policy_store.opa_client     | INFO  | processing store transaction: {'id': '71db3f45a10b4cc48523c40850125535', 'actions': [], 'transaction_type': <TransactionType.policy: 'policy'>, 'success': False, 'error': "ValueError('unexpected response code while fetching bundle: 503')", 'creation_time': '2023-05-17T06:19:47.957289', 'end_time': '2023-05-17T06:19:47.957321', 'remotes_status': [{'remote_url': '<http://titanopal-server:7002/policy>', 'succeed': False, 'error': "ValueError('unexpected response code while fetching bundle: 503')"}]}
2023-05-17T06:19:47.957783+0000 | opal_client.policy_store.opa_client     | INFO  | persisting health check policy: ready=true, healthy=true
2023-05-17T06:19:47.957854+0000 | opal_client.policy_store.opa_client     | INFO  | Policy and data statistics: policy: (successful 830, failed 428);	data: (successful 1259, failed 0)
2023-05-17T06:19:48.165346+0000 | opal_client.policy.fetcher              | INFO  | Fetching policy bundle from <http://titanopal-server:7002/policy>
2023-05-17T06:19:48.179707+0000 | opal_common.utils                       |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
2023-05-17T06:19:48.917980+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.244.1.1:46450 - "GET /healthcheck HTTP/1.1" 200
2023-05-17T06:19:49.180634+0000 | opal_client.policy.fetcher              | INFO  | Fetching policy bundle from <http://titanopal-server:7002/policy>
2023-05-17T06:19:49.192546+0000 | opal_common.utils                       |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
2023-05-17T06:19:50.698069+0000 | opal_client.policy.fetcher              | INFO  | Fetching policy bundle from <http://titanopal-server:7002/policy>
2023-05-17T06:19:50.714264+0000 | opal_common.utils                       |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
server logs:
Copy code
|WARNING | Serving default all-data route, meaning DATA_CONFIG_SOURCES was not configured!
2023-05-17T06:20:05.388558+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.244.1.27:60762 - "GET /policy-data HTTP/1.1" 200
2023-05-17T06:20:05.880238+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.244.1.27:60768 - "GET /policy?path=. HTTP/1.1" 503
2023-05-17T06:20:07.035139+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.244.1.27:60772 - "GET /policy?path=. HTTP/1.1" 503
2023-05-17T06:20:07.232784+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.244.1.27:60782 - "GET /policy?path=. HTTP/1.1" 503
2023-05-17T06:20:07.507926+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.244.1.27:60786 - "GET /policy?path=. HTTP/1.1" 503
2023-05-17T06:20:08.169315+0000 | opal_server.data.api                    | INFO  | Serving source configuration
2023-05-17T06:20:08.170475+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.244.1.27:60798 - "GET /data/config HTTP/1.1" 200
2023-05-17T06:20:08.228180+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.244.1.27:60804 - "GET /policy?path=. HTTP/1.1" 503
2023-05-17T06:20:08.291717+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.244.1.27:60808 - "GET /policy?path=. HTTP/1.1" 503
2023-05-17T06:20:08.638703+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.244.1.27:60822 - "GET /policy?path=. HTTP/1.1" 503
2023-05-17T06:20:08.687869+0000 | opal_server.data.api                    |WARNING | Serving default all-data route, meaning DATA_CONFIG_SOURCES was not configured!
2023-05-17T06:20:08.689151+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.244.1.27:60816 - "GET /policy-data HTTP/1.1" 200
2023-05-17T06:20:09.633583+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.244.1.27:60826 - "GET /policy?path=. HTTP/1.1" 503
2023-05-17T06:20:11.654324+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.244.1.27:51490 - "GET /policy?path=. HTTP/1.1" 503
2023-05-17T06:20:12.773889+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 10.244.1.27:51492 - "GET /policy?path=. HTTP/1.1" 503
a
hi @raghanag it looks like the server could not clone the repo
probably a credentials issue
r
and also in server
Copy code
cmdline: git clone -v --branch=master -- <mailto:git@gitlab-odx.com|git@gitlab-odx.com>:main/code/opa-policies-and-data.git /opal/regoclone/opal_repo_clone
  stderr: '06:20:33.214788 git.c:444               trace: built-in: git clone -v --branch=master -- <mailto:git@gitlab-odx.com|git@gitlab-odx.com>:main/code/opa-policies-and-data.git /opal/regoclone/opal_repo_clone
Cloning into '/opal/regoclone/opal_repo_clone'...
06:20:33.233708 run-command.c:664       trace: run_command: unset GIT_DIR; GIT_PROTOCOL=version=2 'ssh -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -i //opal/.ssh/opal_repo_ssh_key' -o SendEnv=GIT_PROTOCOL <mailto:git@gitlab-odx.oracledx.com|git@gitlab-odx.oracledx.com> 'git-upload-pack '\''cx-sales/code/titan/titan-opa-policies-and-data.git'\'''
fatal: Remote branch master not found in upstream origin
fatal: the remote end hung up unexpectedly
remote: 
remote: ========================================================================
remote: 
remote: rpc error: code = Canceled desc = user canceled the fetch
remote: 
remote: ========================================================================
remote: 
'
a
that's the issue:
Copy code
Remote branch master not found in upstream origin
how did you configure the git repo
?
r
i gave the ssh private key as part of OPAL_POLICY_REPO_SSH_KEY
a
sometimes there's an encoding issue of the private ssh key
r
OPAL_POLICY_REPO_URL = git@gitlab-odx.com:main/code/opa-policies-and-data.git
a
did you encode it in one line with underscores?
r
you meant base64 encode?
a
nope, just a sec i'll give you an example
r
Copy code
The passed value must be the contents of the SSH key in one line (replace new-line with underscore, i.e: \n with _)
a
yes this
also - it might be that opal cannot decode this ssh key due to unsupported algorithm
try creating a key like this:
this is for github but the important part is how to generate the ssh key
r
i generated using the same like above
and my private key is in 
id_ed25519
a
i see, there's no immediate thing that jumps to mind
maybe we can schedule a zoom call with one of our engineers that will help you look into it?
cc @Ro'e Katz @Ori Shavit @Shaul Kremer
r
sure
a
i'll ask one of my engineers to follow up with you and schedule a time
r
thank you
In the meantime can you let me know how to update existing policies and data with new ones without git repo or bundle server
s
Hi @raghanag, as Asaf said, OPAL currently supports reading policies from git and bundle servers. However, you can easily create a local git repo that works with OPAL:
Copy code
mkdir /tmp/policy
cp your_rego_files.rego /tmp/policy/
cd /tmp/policy
git init
git add *.rego
git commit
docker run -p 7002:7002 -v /tmp/policy:/policy -e OPAL_POLICY_REPO_URL=/policy permitio/opal-server:latest
Also, as this is an actual git repository mapped as a volume, any changes you commit outside of the container will be reflected in OPAL.
r
so this to start the opal-server which is mapped to a volume and then pass the env vars with repo url as that mapped directory right?
@Asaf Cohen ssh key git clone is working as expected in opal-server
Open in Slack
PreviousNext
Powered by