Thilak Reddy
03/05/2024, 10:35 AMUziel Sulkies
03/06/2024, 1:37 PMKhandoker Tahmid Sami
03/06/2024, 11:18 PMStefan Schneider
03/07/2024, 9:53 AMC
03/11/2024, 12:21 PMBenny O'Neill
03/11/2024, 4:22 PMSteven Daniels
03/11/2024, 5:57 PMLOG_FORMAT
ENV configuration. what is the default format and what options can be used in creating the format?
I'm also using OPAL_LOG_SERIALIZE="true"
, so the logs are in json format and the text
attribute appears to have color-related unicode (which I don't want to log)Sankar
03/13/2024, 3:43 PM- name: OPAL_DATA_CONFIG_SOURCES
value: '{"config":{"entries":[{"url":"<http://rbs-2-dev.n1028250356.scus-dev-a3.cluster.k8s.us.walmart.net/data>","topics":["policy_data"],"dst_path":"ds","config":{"headers":{"Authorization":"Bearer FAKE-POLICY-SECRET"}}}]}}'
However when hitting /policy-data endpoint I am getting empty json.
In the logs I noticed this line
| [34mopal_server.data.api [0m|[33m[1mWARNING | Serving default all-data route, meaning DATA_CONFIG_SOURCES was not configured![0m
Am I missing something here ? When I configure the same data config source in the opal-fetcher-postgres project and run the docker in local, I am able to get the response. Same way I have added the env variable for this container. However the policy repo added along with it is picked.
I am attaching full logs and the deployment-server.yaml file here.
Please help. Have been working on this for a while nowGil Dagan
03/13/2024, 5:02 PMconst opalUpdateConfig = {
id: updateId,
entries: [
{
url: "",
config: {},
topics: ['policy_data'],
dst_path: `/${path}${id ? `/${id}` : ``}`,
save_method: "PATCH",
data: [updateData]
}
],
callback: {
callbacks: [
["<http://opal-sync-hub:7003/data/callback_report>",
{'headers': {'X-update-start-time': 'time'}}]
]
}
}
But Iβm getting a TypeError from opal_client/callbacks/register.py", line 45
These are the opal-client logs:
environment-opal_client-1 | elif isinstance(callback, CallbackConfig):
environment-opal_client-1 | β β typing.Tuple[str, opal_common.fetcher.providers.http_fetch_provider.HttpFetcherConfig]
environment-opal_client-1 | β ('<http://opal-sync-hub:7003/data/callback_report>', HttpFetcherConfig(fetcher=None, headers={'X-update-start-time': 'token'}, ...
environment-opal_client-1 | File "/usr/local/lib/python3.12/typing.py", line 1176, in __instancecheck__
environment-opal_client-1 | return self.__subclasscheck__(type(obj))
environment-opal_client-1 | β β β ('<http://opal-sync-hub:7003/data/callback_report>', HttpFetcherConfig(fetcher=None, headers={'X-update-start-time': 'token'}, ...
environment-opal_client-1 | β β <function _BaseGenericAlias.__subclasscheck__ at 0x7ffffe943880>
environment-opal_client-1 | β typing.Tuple[str, opal_common.fetcher.providers.http_fetch_provider.HttpFetcherConfig]
environment-opal_client-1 | File "/usr/local/lib/python3.12/typing.py", line 1179, in __subclasscheck__
environment-opal_client-1 | raise TypeError("Subscripted generics cannot be used with"
environment-opal_client-1 |
environment-opal_client-1 | TypeError: Subscripted generics cannot be used with class and instance checks
When Iβm not using a tuple in the callbacks array it works fine, but for some reason the isinstance(callback, tuple)
throws this error.
Have you faced this problem previously?
Thanks in advanceπRambabu
03/14/2024, 8:55 AMRambabu
03/14/2024, 8:55 AMRo'e Katz
03/15/2024, 10:06 AMfastapi_websocket_pubsub
& permit-broadcaster
version upgrades for faster, more stable PubSub.
β’ Bug fixes and optimizations in Scopes
β’ And more...
As usual, big thanks to all our amazing community contributors! π
Slack ConversationYemane Yohannes
03/15/2024, 10:20 PMclient:
OPAL_OPA_HEALTH_CHECK_POLICY_ENABLED=True
server:
OPAL_POLICY_BUNDLE_SERVER_TYPE=AWS-S3
OPAL_POLICY_BUNDLE_SERVER_TOKEN_ID=AK...
OPAL_POLICY_BUNDLE_SERVER_TOKEN=afd...
OPAL_POLICY_BUNDLE_URL=<https://s3.amazonaws.com/bucket/folder/subfolder>
OPAL_POLICY_REPO_POLLING_INTERVAL=10
Sankar
03/18/2024, 8:29 AMenv:
- name: OPAL_DATA_CONFIG_SOURCES
value: '{"config":{"entries":[{"url":"<http://rbs-2-dev.n1028250356.scus-dev-a3.cluster.k8s.us.walmart.net/data>","topics":["policy_data"],"dst_path":"ds","config":{"headers":{"Authorization":"Bearer FAKE-POLICY-SECRET"}}}]}}'
After deployment of client, I was able to get the data when I call /v1/data.
However after I redeployed the service that returns the JSON, /v1/data is not having this data.
1. Even If the returning service goes down and come up, shouldn't the opal client be having the data in the memory ?
2. Since I am using external endpoint for data source, how would OPAL know if more data has been added to the database from which the endpoint queries data
3. Since external endpoint is configured to run (select * from table) query and return json, each time opal queries, will it be fetching the whole data in one go ?Yemane Yohannes
03/20/2024, 7:04 PM|WARNING | base_hash f10608f2d759f1982c1e0d9eb7048d771cea4f2f not exist in the repo
C
03/27/2024, 1:12 PMSteve Mastrorocco
03/27/2024, 4:08 PMSteve Fleetwood
03/27/2024, 5:04 PMopal server
to sync policy changes from a git repo (polling), I can see that the subscribed opal clients
make PUT requests to OPA
, for new files, but it doesn't seem to delete removed ones, is that correct?Sankar
03/28/2024, 2:27 PMcurl --location --request POST '<http://opal-server-test.dev.walmart.com/data/config>' \
--header 'accept: application/json' \
--header 'Content-Type: application/json' \
--data-raw '{
"id": "randomid",
"entries": [
{
"url": "",
"config": {},
"topics": [
"policy_data"
],
"dst_path": "/users",
"save_method": "PATCH",
"data": [
{
"op": "remove",
"path": "/agradip"
}
]
}
],
"reason": "user bob is deleted from the system",
"callback": {
"callbacks": []
}
}'
https://permit-io.slack.com/archives/C01RUUYV3TP/p1710752535861009?thread_ts=1710750548.510389&cid=C01RUUYV3TPSteve Mastrorocco
03/28/2024, 3:41 PMopal_scope
, like the example for /countries
. do I need to restart the pdp to see the data in the v1/data
endpoint? or is there a trigger to get it to pull in new data sources? I'm also not clear on the above question from Sankar, of how to exactly trigger data updates once it is available.C
03/29/2024, 7:06 PMC
03/29/2024, 7:18 PMCai Walkowiak
04/02/2024, 8:11 PMYemane Yohannes
04/04/2024, 6:35 PM/policies
in plain text. Is there a setting to make it so, that anyone that requests that endpoint isn't able to get the plain text policy output?Elena Bernardini
04/15/2024, 6:04 AMRambabu
04/16/2024, 12:07 PMPiyush Sardana
04/22/2024, 2:29 PM2024-04-22T14:20:25.584554+0000 | opal_common.git.repo_cloner |ERROR | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
24
cmdline: git clone -v --branch=master -- <https://gitlab.company.com/company/Server/iam/authorization-policies-test> /opal/regoclone/opal_repo_clone
23
stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
22
fatal: could not read Username for '<https://gitlab.company.com>': No such device or address
Thilak Reddy
04/23/2024, 7:38 AMWhen running in production, you should run with multiple workers per server instance (i.e: container/node), if not multiple containers, and thus deploying the backbone service becomes mandatory for production environments.Is it for reliability only(for high load/frequent updates happening) or does it also help in improving latencies as well(i am guessing not)?
Piyush Sardana
04/23/2024, 9:34 AMYemane Yohannes
04/24/2024, 6:13 PM