I set it "*periodic_update_interval":*1.0 now and i was expecting the fetcher to fetch info from the database each second, while in logs i see only periodic opal-fetcher-sqlite-opal_server-1 | 2023-08-23T145745.962905+0000 | opal_common.sources.git_policy_source | INFO | Pulling changes from remote: 'origin' opal-fetcher-sqlite-opal_server-1 | 2023-08-23T145746.664412+0000 | opal_common.sources.git_policy_source | INFO | No new commits: HEAD is at 'f10608f2d759f1982c1e0d9eb7048 d771cea4f2f' opal-fetcher-sqlite-opal_server-1 | 2023-08-23T145816.665559+0000 | opal_common.sources.git_policy_source | INFO | Pulling changes from remote: 'origin' opal-fetcher-sqlite-opal_server-1 | 2023-08-23T145817.165103+0000 | opal_common.sources.git_policy_source | INFO | No new commits: HEAD is at 'f10608f2d759f1982c1e0d9eb7048 d771cea4f2f' opal-fetcher-sqlite-opal_server-1 | 2023-08-23T145847.166221+0000 | opal_common.sources.git_policy_source | INFO | Pulling changes from remote: 'origin' opal-fetcher-sqlite-opal_server-1 | 2023-08-23T145847.729575+0000 | opal_common.sources.git_policy_source | INFO | No new commits: HEAD is at 'f10608f2d759f1982c1e0d9eb7048 d771cea4f2f'

as far as i understand this is more about - OPAL_POLICY_REPO_POLLING_INTERVAL=30 , but not periodic_update_interval

while periodic_update_interval seems not applied: see config below environment: - OPAL_BROADCAST_URI=redis://broadcast_channel:6379 - UVICORN_NUM_WORKERS=4 - OPAL_POLICY_REPO_URL=https://github.com/permitio/opal-example-policy-repo - OPAL_POLICY_REPO_POLLING_INTERVAL=30 - OPAL_DATA_CONFIG_SOURCES={"config":{"entries":[{"*periodic_update_interval*":1.0,"topics":["policy_data"],"dst_path":"cities","url":"file:/db/opal.db","config":{"fetcher":"SQLiteFetchProvider","query":"SELECT * FROM city;"}}]}}

below is log when i do a POST manually and all works: opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.475486+0000 | opal_server.data.data_update_publisher | INFO | [11] Publishing data update to topics: ['policy_data'], r eason: None, entries: [('file:/db/opal.db', 'PUT', 'cities')] opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.475741+0000 | uvicorn.protocols.http.httptools_impl | INFO | 172.28.0.1:36960 - "POST /data/config?subject=d2b3aa1a-1b 5c-54ad-b088-51a5459105b2 HTTP/1.1" 200 opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.476312+0000 | fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': 'fc017d6127e543659363580615b74c5c ', 'subscriber_id': 'f12a7e8467a54e89acf9fb573a519d2c', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_webs ocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7f51acdff370>>, 'notifier_id': None} opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.476577+0000 | fastapi_websocket_pubsub.event_notifier | INFO | calling subscription callbacks: topic=policy_data (ALL_TO PICS), subscription_id=fc017d6127e543659363580615b74c5c, subscriber_id=f12a7e8467a54e89acf9fb573a519d2c opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.476734+0000 | fastapi_websocket_pubsub.event_broadc...| INFO | Broadcasting incoming event: {'topic': 'policy_data', 'no tifier_id': 'f12a7e8467a54e89acf9fb573a519d2c'} opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.477000+0000 | asyncio_redis.connection | INFO | Connecting to redis opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.477736+0000 | asyncio_redis.protocol | INFO | Redis connection made opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.478048+0000 | asyncio_redis.connection | INFO | Connecting to redis opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.478737+0000 | asyncio_redis.protocol | INFO | Redis connection made opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.479706+0000 | asyncio_redis.protocol | INFO | Redis connection lost opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.480003+0000 | asyncio_redis.protocol | INFO | Redis connection lost opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.479979+0000 | fastapi_websocket_pubsub.event_broadc...| INFO | Handling incoming broadcast event: {'topics': ['policy_da ta'], 'src': 'f12a7e8467a54e89acf9fb573a519d2c'} opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.480274+0000 | fastapi_websocket_pubsub.event_notifier | INFO | calling subscription callbacks: topic=policy_data, subscr iption_id=2e3f9d5e1a1e422897e9534fc596596e, subscriber_id=f865fbbe1a2a4651829deb92662b42fc opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.480344+0000 | fastapi_websocket_pubsub.event_notifier | INFO | Removing Subscription of topic='__EventNotifier_ALL_TOPIC S__' for subscriber=f12a7e8467a54e89acf9fb573a519d2c opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.480536+0000 | fastapi_websocket_pubsub.rpc_event_me...| INFO | Notifying other side: {'subscriber_id': 'f865fbbe1a2a4651 829deb92662b42fc', 'subscription_id': '2e3f9d5e1a1e422897e9534fc596596e', 'topic': 'policy_data'} opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.482060+0000 | opal_client.data.rpc | INFO | Received notification of event: policy_data opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.482325+0000 | opal_client.data.updater | INFO | Updating policy data, reason: None opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.482557+0000 | opal_client.data.updater | INFO | Triggering data update with id: f8a73155f06b44b0b542c4b54 f996552 opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.482697+0000 | opal_client.data.updater | INFO | Fetching policy data opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.482863+0000 | opal_client.data.fetcher | INFO | Fetching data from url: file:/db/opal.db opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.492225+0000 | opal_client.data.updater | INFO | Saving fetched data to policy-store: source url='file:/db /opal.db', destination path='/cities' opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.494287+0000 | opal_client.engine.logger | INFO | Received request. PUT /v1/data/cities opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.495305+0000 | opal_client.engine.logger | INFO | Sent response. PUT /v1/data/cities -> 204 opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.496136+0000 | opal_client.policy_store.opa_client | INFO | OPA client health: True (policy: True, data: True) opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.496301+0000 | opal_client.policy_store.opa_client | INFO | persisting health check policy: ready=True, healthy=True opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.496418+0000 | opal_client.policy_store.opa_client | INFO | Policy and data statistics: policy: (successful 1, failed 0); data: (successful 7, failed 0) opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.496520+0000 | opal_client.policy_store.opa_client | INFO | OPA client health: True (policy: True, data: True) opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.498301+0000 | opal_client.engine.logger | INFO | Received request. PUT /v1/policies/engine/healthcheck/ opal.rego opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.503210+0000 | opal_client.engine.logger | INFO | Sent response. PUT /v1/policies/engine/healthcheck/ opal.rego -> 200 opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.504726+0000 | opal_client.callbacks.reporter | INFO | Reporting the update to requested callbacks opal-fetcher-sqlite-opal_client-1 | 2023-08-23T150916.504969+0000 | opal_client.data.fetcher | INFO | Fetching data from url: http://opal_server:7002/data/call back_report opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.507817+0000 | opal_server.data.api | INFO | Received update report: {'update_id': 'f8a73155f06b44b0b5 42c4b54f996552', 'reports': [{'entry': {'url': 'file:/db/opal.db', 'topics': ['policy_data'], 'dst_path': 'cities', 'save_method': 'PUT'}, 'fetched': True, 'saved': True, 'hash': 'ce41 ca9696e4753de4b91794ad38fd522cdf8bf6822615daed8a277958ba104e'}], 'policy_hash': None} opal-fetcher-sqlite-opal_server-1 | 2023-08-23T150916.508065+0000 | uvicorn.protocols.http.httptools_impl | INFO | 172.28.0.4:55314 - "POST /data/callback_report HTTP/1.1" 200

2. no errors

Are you seeing a log line with “Establishing Period Updates for the following source” in the middle

?

Also are you using OPAL scopes (they don’t support periodic updates at the moment) ?

no log line containing "Establishing Period Updates for the following source" is present

not using OPAL scores

Relevant code: https://github.com/permitio/opal/blob/18dbe29dbf520cc2f4e6c721d2ddb1067b992e40/packages/opal-server/opal_server/data/data_update_publisher.py#L[…]65

see full log of docker compose up

D:\Workspace\opal-fetcher-sqlite>docker compose up [+] Running 3/0 ✔️ Container opal-fetcher-sqlite-broadcast_channel-1 Created 0.0s ✔️ Container opal-fetcher-sqlite-opal_server-1 Created 0.0s ✔️ Container opal-fetcher-sqlite-opal_client-1 Created 0.0s Attaching to opal-fetcher-sqlite-broadcast_channel-1, opal-fetcher-sqlite-opal_client-1, opal-fetcher-sqlite-opal_server-1 opal-fetcher-sqlite-broadcast_channel-1 | 1:C 24 Aug 2023 075602.749 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo opal-fetcher-sqlite-broadcast_channel-1 | 1:C 24 Aug 2023 075602.749 # Redis version=5.0.14, bits=64, commit=00000000, modified=0, pid=1, just started opal-fetcher-sqlite-broadcast_channel-1 | 1:C 24 Aug 2023 075602.749 # Warning: no config file specified, using the default config. In order to specify a config file use redis-serve r /path/to/redis.conf opal-fetcher-sqlite-broadcast_channel-1 | 1:M 24 Aug 2023 075602.750 * Running mode=standalone, port=6379. opal-fetcher-sqlite-broadcast_channel-1 | 1:M 24 Aug 2023 075602.750 # Server initialized opal-fetcher-sqlite-broadcast_channel-1 | 1:M 24 Aug 2023 075602.750 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. opal-fetcher-sqlite-broadcast_channel-1 | 1:M 24 Aug 2023 075602.750 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and mem ory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain t he setting after a reboot. Redis must be restarted after THP is disabled. opal-fetcher-sqlite-broadcast_channel-1 | 1:M 24 Aug 2023 075602.750 * DB loaded from disk: 0.000 seconds opal-fetcher-sqlite-broadcast_channel-1 | 1:M 24 Aug 2023 075602.750 * Ready to accept connections opal-fetcher-sqlite-opal_server-1 | [2023-08-24 075603 +0000] [1] [INFO] Starting gunicorn 20.1.0 opal-fetcher-sqlite-opal_server-1 | [2023-08-24 075603 +0000] [1] [INFO] Listening at: http://0.0.0.0:7002 (1) opal-fetcher-sqlite-opal_server-1 | [2023-08-24 075603 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker opal-fetcher-sqlite-opal_server-1 | [2023-08-24 075603 +0000] [8] [INFO] Booting worker with pid: 8 opal-fetcher-sqlite-opal_server-1 | [2023-08-24 075603 +0000] [9] [INFO] Booting worker with pid: 9 opal-fetcher-sqlite-opal_server-1 | [2023-08-24 075603 +0000] [10] [INFO] Booting worker with pid: 10 opal-fetcher-sqlite-opal_server-1 | [2023-08-24 075603 +0000] [11] [INFO] Booting worker with pid: 11 opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075603.978946+0000 | opal_server.server | INFO | OPAL was not provided with JWT encryption keys, cannot ve rify api requests! opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.035890+0000 | opal_server.server | INFO | triggered startup event opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.036115+0000 | opal_common.topics.publisher | INFO | started topic publisher opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.036319+0000 | opal_server.server | INFO | leadership lock acquired, leader pid: 8 opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.036413+0000 | opal_server.server | INFO | listening on webhook topic: 'webhook' opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.036812+0000 | opal_server.server |WARNING | Found previous policy repo clone: /regoclone/opal_repo_c lone-2c7bf01bf50746928c172b325008736f, removing it to avoid conflicts. opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.037669+0000 | opal_server.server | INFO | Policy repo will be cloned to: /regoclone/opal_repo_clone -aea3a4a28edc476da8480500254a05da opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.039058+0000 | fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': 'b1d7d49217214f1abc497976bcbed3e8 ', 'subscriber_id': 'e17424f3402b4909a67617bd2192fa7b', 'topic': 'webhook', 'callback': functools.partial(<function trigger_repo_watcher_pull at 0x7f9bf284e820>, <opal_server.policy.wa tcher.task.PolicyWatcherTask object at 0x7f9bf26cdf10>), 'notifier_id': None} opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.039335+0000 | opal_server.policy.watcher.task | INFO | Launching policy watcher opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.039473+0000 | opal_common.topics.publisher | INFO | started broadcaster keepalive task: topic is '__broadcast _session_keepalive__', interval is 3600 seconds opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.040269+0000 | opal_common.git.repo_cloner | INFO | Cloning repo from 'https://github.com/permitio/opal-examp le-policy-repo' to '/regoclone/opal_repo_clone-aea3a4a28edc476da8480500254a05da' (branch: 'master') opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.070194+0000 | opal_server.server | INFO | OPAL was not provided with JWT encryption keys, cannot ve rify api requests! opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.123366+0000 | opal_server.server | INFO | triggered startup event opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.123528+0000 | opal_common.topics.publisher | INFO | started topic publisher opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.158056+0000 | opal_server.server | INFO | OPAL was not provided with JWT encryption keys, cannot ve rify api requests! opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.163016+0000 | opal_server.server | INFO | OPAL was not provided with JWT encryption keys, cannot ve rify api requests! opal-fetcher-sqlite-opal_client-1 | [2023-08-24 075604 +0000] [7] [INFO] Starting gunicorn 20.1.0 opal-fetcher-sqlite-opal_client-1 | [2023-08-24 075604 +0000] [7] [INFO] Listening at: http://0.0.0.0:7000 (7) opal-fetcher-sqlite-opal_client-1 | [2023-08-24 075604 +0000] [7] [INFO] Using worker: uvicorn.workers.UvicornWorker opal-fetcher-sqlite-opal_client-1 | [2023-08-24 075604 +0000] [8] [INFO] Booting worker with pid: 8 opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.213705+0000 | opal_server.server | INFO | triggered startup event opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.214690+0000 | opal_common.topics.publisher | INFO | started topic publisher opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.222927+0000 | opal_server.server | INFO | triggered startup event opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075604.223116+0000 | opal_common.topics.publisher | INFO | started topic publisher opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075604.455990+0000 | opal_client.policy_store.opa_client | INFO | Authentication mode for policy store: PolicyStoreAuth.NON E opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075604.458310+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'FastApiRpcFetchProvider' found a t: <class 'opal_common.fetcher.providers.fastapi_rpc_fetch_provider.FastApiRpcFetchProvider'> opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075604.458505+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'HttpFetchProvider' found at: <cl ass 'opal_common.fetcher.providers.http_fetch_provider.HttpFetchProvider'> opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075604.466134+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'SQLiteFetchProvider' found at: < class 'opal_fetcher_sqlite.provider.SQLiteFetchProvider'> opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075604.466337+0000 | opal_common.fetcher.fetcher_register | INFO | Fetcher Register loaded opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075604.466641+0000 | opal_client.callbacks.register | INFO | Callbacks register loaded opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075604.467096+0000 | opal_client.client | INFO | API authentication disabled (public encryption key was no t provided) opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075604.491954+0000 | opal_client.engine.runner | INFO | Launching engine runner opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075604.492806+0000 | opal_client.engine.runner | INFO | Running policy engine inline: opa run --server --addr=:81 81 --authentication=off --authorization=off --log-level=info opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075604.506700+0000 | opal_client.engine.logger | INFO | Initializing server. {"addrs": [":8181"], "diagnostic-add rs": [], "time": "2023-08-24T075604Z"} opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.152815+0000 | opal_common.git.repo_cloner | INFO | Clone succeeded opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.157652+0000 | opal_common.sources.base_policy_source | INFO | Launching polling task, interval: 30 seconds opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.158437+0000 | opal_common.sources.git_policy_source | INFO | Pulling changes from remote: 'origin' opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.496015+0000 | opal_client.engine.runner | INFO | Running policy engine initial start callbacks opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.496896+0000 | opal_client.policy_store.opa_client | INFO | OPA client health: False (policy: False, data: False) opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.497121+0000 | opal_client.policy_store.opa_client | INFO | persisting health check policy: ready=False, healthy=Fals e opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.497262+0000 | opal_client.policy_store.opa_client | INFO | Policy and data statistics: policy: (successful 0, failed 0); data: (successful 0, failed 0) opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.497343+0000 | opal_client.policy_store.opa_client | INFO | OPA client health: False (policy: False, data: False) opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.502068+0000 | opal_client.engine.logger | INFO | Received request. PUT /v1/policies/engine/healthcheck/ opal.rego opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.505424+0000 | opal_client.engine.logger | INFO | Sent response. PUT /v1/policies/engine/healthcheck/ opal.rego -> 200

opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.506558+0000 | opal_client.data.updater | INFO | Launching data updater opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.506823+0000 | opal_client.policy.updater | INFO | Launching policy updater opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.506994+0000 | opal_client.data.updater | INFO | Subscribing to topics: ['policy_data'] opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.507199+0000 | opal_client.policy.updater | INFO | Subscribing to topics: ['policy:.'] opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.507570+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal_server:70 02/ws opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.508350+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal_server:7002/ws opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.510391+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal_server:70 02/ws opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.510707+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal_server:7002/ws opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.515758+0000 | fastapi_websocket_pubsub.event_broadc...| INFO | Listening for incoming events from broadcast channel (fir st listener started) opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.516078+0000 | fastapi_websocket_pubsub.event_broadc...| INFO | Listening for incoming events from broadcast channel (fir st listener started) opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.544837+0000 | fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': 'bdd700796fa0418297de5cb2abed1347 ', 'subscriber_id': 'effb2c74a5b74dd1970b852dbbbbc445', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_webs ocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7f9bf2862070>>, 'notifier_id': None} opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.545525+0000 | fastapi_websocket_rpc.websocket_rpc_e...| INFO | Client connected opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.545820+0000 | fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': '4f435ccaa18d4e09978f7cefb3b4d90d ', 'subscriber_id': 'b76636ab31614da68fddff472ebd985c', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_webs ocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7f9bf2862130>>, 'notifier_id': None} opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.546027+0000 | fastapi_websocket_pubsub.event_broadc...| INFO | Starting broadcaster listener opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.546932+0000 | asyncio_redis.connection | INFO | Connecting to redis opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.547022+0000 | fastapi_websocket_rpc.websocket_rpc_e...| INFO | Client connected opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.547558+0000 | fastapi_websocket_pubsub.event_broadc...| INFO | Starting broadcaster listener opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.548256+0000 | asyncio_redis.connection | INFO | Connecting to redis opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.549657+0000 | asyncio_redis.protocol | INFO | Redis connection made opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.551058+0000 | asyncio_redis.connection | INFO | Connecting to redis opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.551663+0000 | asyncio_redis.protocol | INFO | Redis connection made opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.552091+0000 | asyncio_redis.connection | INFO | Connecting to redis opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.552359+0000 | asyncio_redis.protocol | INFO | Redis connection made opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.553036+0000 | asyncio_redis.protocol | INFO | Redis connection made opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.560020+0000 | fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': 'a1ccc0123abc4fb9b7ac3c0dc3b2b7b1 ', 'subscriber_id': '9b824bff816f4940841c16a00481c5c7', 'topic': 'policy_data', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7f9bf24c0dc0>, 'notifier_id ': None} opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.561434+0000 | fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': 'e793e540dfce40339090c3cd378a07e7 ', 'subscriber_id': 'fcbb9eec1c2349e28a4a1b9cb4027e7c', 'topic': 'policy:.', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7f9bf24c0dc0>, 'notifier_id': None} opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.563430+0000 | opal_client.data.updater | INFO | Connected to server opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.563663+0000 | opal_client.data.updater | INFO | Performing data configuration, reason: Initial load opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.563779+0000 | opal_client.data.updater | INFO | Getting data-sources configuration from 'http://opal_serv er:7002/data/config' opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.565926+0000 | opal_client.policy.updater | INFO | Connected to server opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.566254+0000 | opal_client.policy.updater | INFO | Refetching policy code (full bundle) opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.566820+0000 | opal_client.policy.fetcher | INFO | Fetching policy bundle from http://opal_server:7002/polic y opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.569242+0000 | opal_server.data.api | INFO | Serving source configuration opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.570539+0000 | uvicorn.protocols.http.httptools_impl | INFO | 172.29.0.4:33348 - "GET /data/config HTTP/1.1" 200 opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.572773+0000 | opal_client.data.updater | INFO | Triggering data update with id: 20132215e3994b42b38cefaa2 2340e26 opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.573117+0000 | opal_client.data.updater | INFO | Fetching policy data opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.573325+0000 | opal_client.data.fetcher | INFO | Fetching data from url: http://host.docker.internal:7002/ policy-data opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.576001+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal_server:7002/ws opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.584283+0000 | opal_server.data.api |WARNING | Serving default all-data route, meaning DATA_CONFIG_SOUR CES was not configured! opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.584897+0000 | uvicorn.protocols.http.httptools_impl | INFO | 172.29.0.1:51460 - "GET /policy-data HTTP/1.1" 200 opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.588091+0000 | opal_client.data.updater | INFO | Saving fetched data to policy-store: source url='<http://h> ost.docker.internal:7002/policy-data', destination path='/' opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.590780+0000 | opal_client.engine.logger | INFO | Received request. PUT /v1/data opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.591750+0000 | opal_client.engine.logger | INFO | Sent response. PUT /v1/data -> 204 opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.592274+0000 | uvicorn.protocols.http.httptools_impl | INFO | 172.29.0.4:33354 - "GET /policy?path=. HTTP/1.1" 200 opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.592589+0000 | opal_client.policy_store.opa_client | INFO | OPA client health: False (policy: False, data: True) opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.593625+0000 | opal_client.policy_store.opa_client | INFO | persisting health check policy: ready=False, healthy=Fals e opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.593976+0000 | opal_client.policy_store.opa_client | INFO | Policy and data statistics: policy: (successful 0, failed 0); data: (successful 1, failed 0) opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.594431+0000 | opal_client.policy_store.opa_client | INFO | OPA client health: False (policy: False, data: True) opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.597044+0000 | opal_client.policy.fetcher | INFO | Fetched valid bundle, id: f10608f2d759f1982c1e0d9eb7048d7 71cea4f2f opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.598102+0000 | opal_client.policy.updater | INFO | Got policy bundle with 2 rego files, 1 data files, commit hash: 'f10608f2d759f1982c1e0d9eb7048d771cea4f2f' opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.600582+0000 | opal_client.engine.logger | INFO | Received request. PUT /v1/policies/engine/healthcheck/ opal.rego opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.601330+0000 | opal_client.engine.logger | INFO | Received request. GET /v1/policies opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.605764+0000 | opal_client.engine.logger | INFO | Sent response. GET /v1/policies -> 200 opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.606630+0000 | opal_client.engine.logger | INFO | Sent response. PUT /v1/policies/engine/healthcheck/ opal.rego -> 200 opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.608130+0000 | opal_client.callbacks.reporter | INFO | Reporting the update to requested callbacks opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.608473+0000 | opal_client.data.fetcher | INFO | Fetching data from url: http://opal_server:7002/data/call back_report opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.610774+0000 | opal_client.engine.logger | INFO | Received request. PUT /v1/data opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.611641+0000 | opal_client.engine.logger | INFO | Sent response. PUT /v1/data -> 204 opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.615103+0000 | opal_client.engine.logger | INFO | Received request. PUT /v1/policies/utils.rego opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.616491+0000 | opal_server.data.api | INFO | Received update report: {'update_id': '20132215e3994b42b3

8cefaa22340e26', 'reports': [{'entry': {'url': 'http://host.docker.internal:7002/policy-data', 'topics': ['policy_data'], 'dst_path': '', 'save_method': 'PUT'}, 'fetched': True, 'saved ': True, 'hash': '44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a'}], 'policy_hash': None} opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.617236+0000 | uvicorn.protocols.http.httptools_impl | INFO | 172.29.0.4:33364 - "POST /data/callback_report HTTP/1.1" 200 opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.619273+0000 | opal_client.engine.logger | INFO | Sent response. PUT /v1/policies/utils.rego -> 200 opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.622593+0000 | opal_client.engine.logger | INFO | Received request. PUT /v1/policies/rbac.rego opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.627143+0000 | opal_client.engine.logger | INFO | Sent response. PUT /v1/policies/rbac.rego -> 200 opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.628341+0000 | opal_client.policy_store.opa_client | INFO | OPA client health: True (policy: True, data: True) opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.628479+0000 | opal_client.policy_store.opa_client | INFO | persisting health check policy: ready=True, healthy=True opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.628617+0000 | opal_client.policy_store.opa_client | INFO | Policy and data statistics: policy: (successful 1, failed 0); data: (successful 1, failed 0) opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.628717+0000 | opal_client.policy_store.opa_client | INFO | OPA client health: True (policy: True, data: True) opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.629700+0000 | opal_client.callbacks.reporter | INFO | Reporting the update to requested callbacks opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.629983+0000 | opal_client.data.fetcher | INFO | Fetching data from url: http://opal_server:7002/data/call back_report opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.631897+0000 | opal_client.engine.logger | INFO | Received request. PUT /v1/policies/engine/healthcheck/ opal.rego opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.633362+0000 | opal_server.data.api | INFO | Received update report: {'update_id': None, 'reports': [] , 'policy_hash': 'f10608f2d759f1982c1e0d9eb7048d771cea4f2f'} opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.633665+0000 | uvicorn.protocols.http.httptools_impl | INFO | 172.29.0.4:33370 - "POST /data/callback_report HTTP/1.1" 200 opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.638937+0000 | opal_client.engine.logger | INFO | Sent response. PUT /v1/policies/engine/healthcheck/ opal.rego -> 200 opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075605.639826+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal_server:7002/ws opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075605.682576+0000 | opal_common.sources.git_policy_source | INFO | No new commits: HEAD is at 'f10608f2d759f1982c1e0d9eb7048 d771cea4f2f' opal-fetcher-sqlite-opal_client-1 | 2023-08-24T075606.243135+0000 | opal_client.engine.logger | INFO | OPA is out of date. {"current_version": "0.53.1", "downl oad_opa": "https://openpolicyagent.org/downloads/v0.55.0/opa_linux_amd64", "latest_version": "0.55.0", "release_notes": "https://github.com/open-policy-agent/opa/releases/tag/v0.55.0", "time": "2023-08-24T075606Z"} opal-fetcher-sqlite-opal_server-1 | 2023-08-24T075635.683505+0000 | opal_common.sources.git_policy_source | INFO | Pulling changes from remote: 'origin'

any ideas? :(

so i found out one thing:

here is my config - OPAL_DATA_CONFIG_SOURCES={"config":{"entries":[{"topics":["policy_data"],"dst_path":"cities","url":"file:/db/opal.db","periodic_update_interval":1.0,"config":{"fetcher":"SQLiteFetchProvider","query":"SELECT * FROM city;"}}]}}

But when i do GET /data/config Get Data Sources Config i am getting

{
  "entries": [
    {
      "url": "file:/db/opal.db",
      "config": {
        "fetcher": "SQLiteFetchProvider",
        "query": "SELECT * FROM city;"
      },
      "topics": [
        "policy_data"
      ],
      "dst_path": "cities",
      "save_method": "PUT"
    }
  ]
}

So, you see, for some reason even if i do have periodic_update_interval property in config, it's for some reason is missing if i get config from server after it launched

may be there is some problems with config parser or something?

/data/config

is meant for clients and they aren’t aware of the periodic variable it’s just something the server handles (And removes for the clients)

You can try and run the server as a command line and see how its parsed This works for me (shows the periodic value) - notice I did wrap the variable with

''

OPAL_DATA_CONFIG_SOURCES='{"config":{"entries":[{"topics":["policy_data"],"dst_path":"cities","url":"file:/db/opal.db","periodic_update_interval":1.0,"config":{"fetcher":"SQLiteFetchProvider","query":"SELECT * FROM city;"}}]}}' opal-server print-config

i do not see it

OPAL_DATA_CONFIG_SOURCES='{"config":{"entries":[{"topics":["policy_data"],"dst_path":"cities","url":"file:/db/opal.db","periodic_up date_interval":1.0,"config":{"fetcher":"SQLiteFetchProvider","query":"SELECT * FROM city;"}}]}}' opal-server print-config Printing configuration values { "ALLOWED_ORIGINS": "['*']", "ALL_DATA_ROUTE": "/policy-data", "ALL_DATA_TOPIC": "policy_data", "ALL_DATA_URL": "http://host.docker.internal:7002/policy-data", "AUTH_JWKS_STATIC_DIR": "/jwks_dir", "AUTH_JWKS_URL": "/.well-known/jwks.json", "AUTH_MASTER_TOKEN": "None", "AUTH_PRIVATE_KEY": "None", "AUTH_PRIVATE_KEY_FORMAT": "EncryptionKeyFormat.pem", "AUTH_PRIVATE_KEY_PASSPHRASE": "None", "BROADCAST_KEEPALIVE_INTERVAL": "3600", "BROADCAST_KEEPALIVE_TOPIC": "__broadcast_session_keepalive__", "BROADCAST_URI": "redis://broadcast_channel:6379", "DATA_CALLBACK_DEFAULT_ROUTE": "/data/callback_report", "DATA_CONFIG_ROUTE": "/data/config", *"DATA_CONFIG_SOURCES": "config=DataSourceConfig(entries=[DataSourceEntry(url='file:/db/opal.db', config={'fetcher': 'SQLiteFetchProvider', 'query': 'SELECT * FROM city;'}, topics=['policy_data'], dst_path='cities', save_method='PUT')]) external_source_url=None",* "DATA_UPDATE_TRIGGER_ROUTE": "/data/update", "ENABLE_DATADOG_APM": "False", "LEADER_LOCK_FILE_PATH": "/tmp/opal_server_leader.lock", "MAX_CHANNELS_PER_CLIENT": "15", "NO_RPC_LOGS": "True", "OPAL_WS_LOCAL_URL": "ws://localhost:7002/ws", "OPAL_WS_TOKEN": "THIS_IS_A_DEV_SECRET", "POLICY_BUNDLE_GIT_ADD_PATTERN": "*", "POLICY_BUNDLE_SERVER_TOKEN": "None", "POLICY_BUNDLE_TMP_PATH": "/tmp/bundle.tar.gz", "POLICY_BUNDLE_URL": "None", "POLICY_REPO_CLONE_FOLDER_PREFIX": "opal_repo_clone", "POLICY_REPO_CLONE_PATH": "/regoclone", "POLICY_REPO_CLONE_TIMEOUT": "0", "POLICY_REPO_MAIN_BRANCH": "main", "POLICY_REPO_MANIFEST_PATH": "policies", "POLICY_REPO_POLLING_INTERVAL": "30", "POLICY_REPO_SSH_KEY": "-----BEGIN RSA PRIVATE KEY-----*-----END RSA PRIVATE KEY-----", "POLICY_REPO_URL": "***", "POLICY_REPO_WEBHOOK_SECRET": "None", "POLICY_SOURCE_TYPE": "PolicySourceTypes.Git", "PUBLISHER_ENABLED": "True", "REPO_WATCHER_ENABLED": "True", "SERVER_HOST": "127.0.0.1", "SERVER_PORT": "7002", "SERVER_WORKER_COUNT": "None", "STATISTICS_STATE_SYNC_CHANNEL": "__opal_stats_state_sync", "STATISTICS_WAKEUP_CHANNEL": "__opal_stats_wakeup" } { "ALLOWED_ORIGINS": "['*']", "AUTH_JWT_ALGORITHM": "JWTAlgorithm.RS256", "AUTH_JWT_AUDIENCE": "https://api.opal.ac/v1/", "AUTH_JWT_ISSUER": "https://opal.ac/", "AUTH_PUBLIC_KEY": "None", "AUTH_PUBLIC_KEY_FORMAT": "EncryptionKeyFormat.ssh", "CLIENT_SELF_SIGNED_CERTIFICATES_ALLOWED": "False", "CLIENT_SSL_CONTEXT_TRUSTED_CA_FILE": "None", "FETCHING_CALLBACK_TIMEOUT": "10", "FETCHING_ENQUEUE_TIMEOUT": "10", "FETCHING_WORKER_COUNT": "5", "FETCH_PROVIDER_MODULES": "['opal_common.fetcher.providers']", "GIT_SSH_KEY_FILE": "/root/.ssh/opal_repo_ssh_key", "LOG_COLORIZE": "True", "LOG_DIAGNOSE": "True", "LOG_FILE_COMPRESSION": "None", "LOG_FILE_LEVEL": "INFO", "LOG_FILE_PATH": "opal_{time}.log", "LOG_FILE_RETENTION": "10 days", "LOG_FILE_ROTATION": "250 MB", "LOG_FILE_SERIALIZE": "True", "LOG_FORMAT": "<green>{time}</green> | <blue>{name: <40}</blue>|<level>{level:^6} | {message}</level>\n{exception}", "LOG_FORMAT_INCLUDE_PID": "False", "LOG_LEVEL": "INFO", "LOG_MODULE_EXCLUDE_LIST": "['uvicorn']", "LOG_MODULE_INCLUDE_LIST": "['uvicorn.protocols.http']", "LOG_PATCH_UVICORN_LOGS": "True", "LOG_SERIALIZE": "False", "LOG_SHOW_CODE_LINE": "True", "LOG_TO_FILE": "False", "LOG_TRACEBACK": "True", "STATISTICS_ADD_CLIENT_CHANNEL": "__opal_stats_add", "STATISTICS_ENABLED": "False", "STATISTICS_REMOVE_CLIENT_CHANNEL": "__opal_stats_rm" }

Dude

docker pull permitio/opal-server

You’re using a two year old image

before the rename

Compare https://hub.docker.com/r/permitio/opal-server and https://hub.docker.com/r/authorizon/opal-server