https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# opal
s
This message was deleted.
z
Here is the docker compose I am using.
Copy code
version: '3.8'

networks:
  company_name:
    name: company_name

services:
  company_name-authorization-webapi:
    image: company_name-authorization-webapi:latest
    container_name: company_name-authorization-webapi
    build:
      context: ./
      dockerfile: src/company_name.Authorization.WebApi/docker-authorization
    networks:
      - company_name
    ports:
      - 5040:8080

  opal_redis:
    image: redis:6.2-alpine
    restart: always
    ports:
      - 6378:6379
    command: redis-server --loglevel warning
    networks:
      - company_name

  opal_server:
    image: permitio/opal-server:0.7.3
    container_name: company_name-opal-server
    environment:
      - OPAL_BROADCAST_URI=<redis://opal_redis:6379>
      - UVICORN_NUM_WORKERS=4
      - OPAL_POLICY_BUNDLE_URL=<http://api_policy_source_server>
      - OPAL_POLICY_SOURCE_TYPE=API
      - OPAL_POLICY_REPO_CLONE_PATH=~/opal
      - OPAL_POLICY_REPO_POLLING_INTERVAL=30
      - OPAL_DATA_CONFIG_SOURCES={"config":{"entries":[{"url":"<http://host.docker.internal:7002/policy-data>","topics":["policy_data"],"dst_path":"/static"}]}}
      - OPAL_LOG_FORMAT_INCLUDE_PID=true
    ports:
      - "7002:7002"
    depends_on:
      - opal_redis
    restart: unless-stopped
    networks:
     - company_name

  opal_client:
    image: permitio/opal-client:0.7.3
    container_name: company_name-opal-client
    environment:
      - OPAL_SERVER_URL=<http://opal_server:7002>
      - OPAL_LOG_FORMAT_INCLUDE_PID=true
      - OPAL_INLINE_OPA_LOG_FORMAT=http
    ports:
      - "7766:7000"
      - "8181:8181"
    depends_on:
      - opal_server
    command: sh -c "exec ./wait-for.sh opal_server:7002 --timeout=20 -- ./start.sh"
    restart: unless-stopped
    networks:
     - company_name

  api_policy_source_server:
    image: nginx
    networks:
      - company_name
    ports:
      - 8000:80
    volumes:
      - ./src/company_name.Authorization.WebApi/docker_files/bundle_files:/usr/share/nginx/html
      - ./src/company_name.Authorization.WebApi/docker_files/nginx.conf:/etc/nginx/nginx.conf
o
Hi @Zach Vosburgh, Using the API bundle server should affect what the OPAL server is publishing for policy, not for data (though bundles and repos can of course contain data as well in the form of data.json files) Did the opal-client not get the content you expected? If so have you looked through your opal-server and client logs? Did the oapl-server successfuly connect to the bundle server, did the client successfully connect to the opal server and downloaded the policy from it?
z
These are the logs from the opal-server. Nothing here made me think it was failing to reach the bundle server
Copy code
2023-09-08 11:53:39 [2023-09-08 16:53:39 +0000] [1] [INFO] Starting gunicorn 20.1.0
2023-09-08 11:53:39 [2023-09-08 16:53:39 +0000] [1] [INFO] Listening at: <http://0.0.0.0:7002> (1)
2023-09-08 11:53:39 [2023-09-08 16:53:39 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
2023-09-08 11:53:39 2023-09-08 16:53:39.634 | WARNING  | __config__:when_ready:34 - Finished pre loading scopes...
2023-09-08 11:53:39 [2023-09-08 16:53:39 +0000] [9] [INFO] Booting worker with pid: 9
2023-09-08 11:53:39 [2023-09-08 16:53:39 +0000] [10] [INFO] Booting worker with pid: 10
2023-09-08 11:53:39 [2023-09-08 16:53:39 +0000] [11] [INFO] Booting worker with pid: 11
2023-09-08 11:53:39 2023-09-08T16:53:39.742993+0000 | 9 | opal_server.server                      | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2023-09-08 11:53:39 2023-09-08T16:53:39.743463+0000 | 9 | opal_server.pubsub                      | INFO  | Initializing broadcaster for server<->server communication
2023-09-08 11:53:39 2023-09-08T16:53:39.784415+0000 | 10 | opal_server.server                      | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2023-09-08 11:53:39 2023-09-08T16:53:39.785301+0000 | 10 | opal_server.pubsub                      | INFO  | Initializing broadcaster for server<->server communication
2023-09-08 11:53:39 2023-09-08T16:53:39.807707+0000 | 11 | opal_server.server                      | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2023-09-08 11:53:39 2023-09-08T16:53:39.808493+0000 | 11 | opal_server.pubsub                      | INFO  | Initializing broadcaster for server<->server communication
2023-09-08 11:53:39 [2023-09-08 16:53:39 +0000] [14] [INFO] Booting worker with pid: 14
2023-09-08 11:53:39 2023-09-08T16:53:39.858398+0000 | 9 | opal_server.server                      | INFO  | *** OPAL Server Startup ***
2023-09-08 11:53:39 2023-09-08T16:53:39.858689+0000 | 9 | opal_server.server                      | INFO  | leadership lock acquired, leader pid: 9
2023-09-08 11:53:39 2023-09-08T16:53:39.858819+0000 | 9 | opal_common.topics.publisher            | INFO  | started broadcaster keepalive task: topic is '__broadcast_session_keepalive__', interval is 3600 seconds
2023-09-08 11:53:39 2023-09-08T16:53:39.859628+0000 | 9 | opal_server.policy.watcher.factory      | INFO  | Policy repo will be cloned to: //opal/opal/opal_repo_clone
2023-09-08 11:53:39 2023-09-08T16:53:39.859989+0000 | 9 | opal_server.policy.watcher.task         | INFO  | Launching policy watcher
2023-09-08 11:53:39 2023-09-08T16:53:39.861024+0000 | 9 | opal_server.data.data_update_publisher  | INFO  | [9] Starting Polling Updates
2023-09-08 11:53:39 2023-09-08T16:53:39.861416+0000 | 9 | fastapi_websocket_pubsub.event_broadc...| INFO  | Listening for incoming events from broadcast channel (first listener started)
2023-09-08 11:53:39 2023-09-08T16:53:39.861811+0000 | 9 | opal_server.policy.watcher.task         | INFO  | listening on webhook topic: 'webhook'
2023-09-08 11:53:39 2023-09-08T16:53:39.862296+0000 | 9 | fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': 'ef6489c438e448d78b127fc9ab3c2a6d', 'subscriber_id': 'f9f0573c7538442e8fbf594252a3a63c', 'topic': 'webhook', 'callback': <bound method BasePolicyWatcherTask._on_webhook of <opal_server.policy.watcher.task.PolicyWatcherTask object at 0x7f32aebabd00>>, 'notifier_id': None}
2023-09-08 11:53:39 2023-09-08T16:53:39.863983+0000 | 9 | fastapi_websocket_pubsub.event_broadc...| INFO  | Starting broadcaster listener
2023-09-08 11:53:39 2023-09-08T16:53:39.887555+0000 | 10 | opal_server.server                      | INFO  | *** OPAL Server Startup ***
2023-09-08 11:53:39 2023-09-08T16:53:39.908406+0000 | 11 | opal_server.server                      | INFO  | *** OPAL Server Startup ***
2023-09-08 11:53:39 2023-09-08T16:53:39.909293+0000 | 14 | opal_server.server                      | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2023-09-08 11:53:39 2023-09-08T16:53:39.909586+0000 | 9 | asyncio_redis.connection                | INFO  | Connecting to redis
2023-09-08 11:53:39 2023-09-08T16:53:39.909772+0000 | 14 | opal_server.pubsub                      | INFO  | Initializing broadcaster for server<->server communication
2023-09-08 11:53:39 2023-09-08T16:53:39.914347+0000 | 9 | asyncio_redis.protocol                  | INFO  | Redis connection made
2023-09-08 11:53:39 2023-09-08T16:53:39.914938+0000 | 9 | asyncio_redis.connection                | INFO  | Connecting to redis
2023-09-08 11:53:39 2023-09-08T16:53:39.916128+0000 | 9 | asyncio_redis.protocol                  | INFO  | Redis connection made
2023-09-08 11:53:39 2023-09-08T16:53:39.992450+0000 | 14 | opal_server.server                      | INFO  | *** OPAL Server Startup ***
2023-09-08 11:53:41 2023-09-08T16:53:41.263386+0000 | 14 | fastapi_websocket_pubsub.event_broadc...| INFO  | Listening for incoming events from broadcast channel (first listener started)
2023-09-08 11:53:41 2023-09-08T16:53:41.291623+0000 | 14 | fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': '2a211f49beef4eb995015aa7d742c316', 'subscriber_id': 'dc9f58ef9ec849278b6bc4192ba5139e', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7f32aed1c2b0>>, 'notifier_id': None}
2023-09-08 11:53:41 2023-09-08T16:53:41.291949+0000 | 14 | fastapi_websocket_pubsub.event_broadc...| INFO  | Starting broadcaster listener
2023-09-08 11:53:41 2023-09-08T16:53:41.292133+0000 | 14 | asyncio_redis.connection                | INFO  | Connecting to redis
2023-09-08 11:53:41 2023-09-08T16:53:41.294077+0000 | 14 | fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client connected
2023-09-08 11:53:41 2023-09-08T16:53:41.295291+0000 | 14 | websockets.legacy.server                | INFO  | connection open
2023-09-08 11:53:41 2023-09-08T16:53:41.295738+0000 | 14 | fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client connected
2023-09-08 11:53:41 2023-09-08T16:53:41.296538+0000 | 14 | websockets.legacy.server                | INFO  | connection open
2023-09-08 11:53:41 2023-09-08T16:53:41.296955+0000 | 14 | asyncio_redis.protocol                  | INFO  | Redis connection made
2023-09-08 11:53:41 2023-09-08T16:53:41.297137+0000 | 14 | asyncio_redis.connection                | INFO  | Connecting to redis
2023-09-08 11:53:41 2023-09-08T16:53:41.300182+0000 | 14 | asyncio_redis.protocol                  | INFO  | Redis connection made
2023-09-08 11:53:41 2023-09-08T16:53:41.300867+0000 | 14 | fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': '71fe6c58d78b4768b9c321ad227cbfa5', 'subscriber_id': 'db6e8e5e04844be0a54c65e2b84bba72', 'topic': 'policy:.', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7f32adae6440>, 'notifier_id': None}
2023-09-08 11:53:41 2023-09-08T16:53:41.302736+0000 | 14 | fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': 'a9a103e3f3744f85979d5204f963d8e2', 'subscriber_id': 'c5b1ee485d384cc59b889ae2d2ebcb8b', 'topic': 'policy_data', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7f32adae5240>, 'notifier_id': None}
2023-09-08 11:53:41 2023-09-08T16:53:41.311535+0000 | 11 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.128.6:36110 - "GET /policy?path=. HTTP/1.1" 503
2023-09-08 11:53:41 2023-09-08T16:53:41.312730+0000 | 10 | opal_server.data.api                    | INFO  | Serving source configuration
2023-09-08 11:53:41 2023-09-08T16:53:41.313337+0000 | 10 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.128.6:36116 - "GET /data/config HTTP/1.1" 200
2023-09-08 11:53:41 2023-09-08T16:53:41.324410+0000 | 14 | opal_server.data.api                    |WARNING | Serving default all-data route, meaning DATA_CONFIG_SOURCES was not configured!
2023-09-08 11:53:41 2023-09-08T16:53:41.325195+0000 | 14 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.128.1:40876 - "GET /policy-data HTTP/1.1" 200
2023-09-08 11:53:42 2023-09-08T16:53:42.266084+0000 | 11 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.128.6:36122 - "GET /policy?path=. HTTP/1.1" 503
2023-09-08 11:53:42 2023-09-08T16:53:42.605807+0000 | 14 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.128.6:36130 - "GET /policy?path=. HTTP/1.1" 503
2023-09-08 11:53:44 2023-09-08T16:53:44.934431+0000 | 9 | opal_common.sources.api_policy_source   | INFO  | Not modified at: 2023-09-08 16:53:44
2023-09-08 11:53:45 2023-09-08T16:53:45.935548+0000 | 14 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.128.6:58348 - "GET /policy?path=. HTTP/1.1" 503
2023-09-08 11:53:47 2023-09-08T16:53:47.290950+0000 | 14 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.128.6:58358 - "GET /policy?path=. HTTP/1.1" 503
2023-09-08 11:53:49 2023-09-08T16:53:49.952662+0000 | 9 | opal_common.sources.api_policy_source   | INFO  | Not modified at: 2023-0
I have tried manually uploading the bundle file to OPA and it works correctly. All of the data and rego files get stord correctly when I do it manually. But when trying to use Opal's bundle server I cannot get either the .rego or data.json files to load.
o
Hi @Zach Vosburgh, Your log and config seem to be at odds, as the log says “Serving default all-data route, meaning DATA_CONFIG_SOURCES was not configured!” while you do have it set in your docker-compose. For the bundle server - I’d expect to see the following log message - when it fetches the bundle - “New bundle found, hash is: {hash}“; or at least some line coming from the 
api_policy_source
module. Did you check the logs of the bundle server - did it get any requests for it’s bundle ?
z
I had not but it is looking like it did.
Copy code
2023-09-08 11:53:38 /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
2023-09-08 11:53:38 /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
2023-09-08 11:53:38 /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
2023-09-08 11:53:38 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
2023-09-08 11:53:38 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
2023-09-08 11:53:38 /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: using the "epoll" event method
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: nginx/1.25.2
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14) 
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: OS: Linux 5.15.90.1-microsoft-standard-WSL2
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: start worker processes
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: start worker process 29
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: start worker process 30
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: start worker process 31
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: start worker process 32
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: start worker process 33
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: start worker process 34
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: start worker process 35
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: start worker process 36
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: start worker process 37
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: start worker process 38
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: start worker process 39
2023-09-08 11:53:38 2023/09/08 16:53:38 [notice] 1#1: start worker process 40
2023-09-08 11:53:38 /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
2023-09-08 11:53:38 /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
2023-09-08 11:53:38 /docker-entrypoint.sh: Configuration complete; ready for start up
2023-09-08 11:53:39 192.168.128.5 - - [08/Sep/2023:16:53:39 +0000] "GET /bundle.tar.gz HTTP/1.1" 200 8728 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:53:44 192.168.128.5 - - [08/Sep/2023:16:53:44 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:53:49 192.168.128.5 - - [08/Sep/2023:16:53:49 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:53:54 192.168.128.5 - - [08/Sep/2023:16:53:54 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:53:59 192.168.128.5 - - [08/Sep/2023:16:53:59 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:54:04 192.168.128.5 - - [08/Sep/2023:16:54:04 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:54:10 192.168.128.5 - - [08/Sep/2023:16:54:10 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:54:15 192.168.128.5 - - [08/Sep/2023:16:54:15 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:54:20 192.168.128.5 - - [08/Sep/2023:16:54:20 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:54:25 192.168.128.5 - - [08/Sep/2023:16:54:25 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:54:30 192.168.128.5 - - [08/Sep/2023:16:54:30 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:54:35 192.168.128.5 - - [08/Sep/2023:16:54:35 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:54:40 192.168.128.5 - - [08/Sep/2023:16:54:40 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:54:45 192.168.128.5 - - [08/Sep/2023:16:54:45 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:54:50 192.168.128.5 - - [08/Sep/2023:16:54:50 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:54:55 192.168.128.5 - - [08/Sep/2023:16:54:55 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
2023-09-08 11:55:00 192.168.128.5 - - [08/Sep/2023:16:55:00 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
o
We are seeing the 304 (not modified) in the log you shared for the opal-server, but not the 200 - are these from the same run ?
Also what are you getting when you reach-out for the OPAL-server at 
/policy
CC: @Asaf Cohen, @Ro'e Katz (if you’re available by chance and you have an idea to share here)
a
Only idea is that you won’t see new bundle fetched when you get 304 from the http server (which means the bundle hash is the same and therefore opal won’t even download the bundle)
o
I think there’s actually no log for the first 
200
so - it seems (by the ngnix log) that it does download the bundle
So @Zach Vosburgh - question is what is being served at 
/policy
and what the opal-client is getting. A quick guess that comes to mind is that there could also be a topic mismatch as in policy - directories translate into topics
So if you wrapped your files into another folder inside the tar-gz it will mismatch the subscribed policy topic
z
This is our folder layout
o
Well you have your answer then --- https://permit-io.slack.com/archives/C01RUUYV3TP/p1694202585489919?thread_ts=1694193316.920779&amp;cid=C01RUUYV3TP You wrapped things into folders - which are mapped into different topics
I suggest you try and run the example provided in the docs, as is - see that it works for you, and then add your changes gradually. You’d notice that in the provided example tar-gz there are no folders
z
Ok that looks to have fixed the problem. Thank you so much. So when using bundles is there a way to have multiple data.json's? While using a github repo were were able to do this by having sub folder's with a data.json in each folder. Is there a way to achieve this same functionality with bundles?
o
Wonderful! Happy to help.
is there a way to achieve this same functionality with bundles?
It should work the same way for api-bundles (OPAL simply downloads the bundles and unpacks them into a local git- so from that point it’s exactly the same code) - I’m guessing you just added one or more wrapper levels on top of what you have in the git option. Alternatively you can use 
OPAL_POLICY_SUBSCRIPTION_DIRS
to adjust what each client subscribes to
z
To create the bundle file I just ran 
.\opa_windows_amd64.exe build ./
from the root directory of the github repo. Is there a better way to do this that will not create more wrapper levels?
a
To create the bundle file I just ran 
.\opa_windows_amd64.exe build ./
from the root directory of the github repo. Is there a better way to do this that will not create more wrapper levels?
I am not sure how OPA behaves if you pass 
./
as the bundle root, but this will work
Copy code
.\opa_windows_amd64.exe build bundle-dir/
it will take all the files and directories (and sub directories) from bundle-dir and will bundle them together. it's typically best practice to run OPA at least once with the bundle and check that the actual package names and data are at the right locations:
Copy code
opa run --server -b bundle.tar.gz
if you want to run at a port other than 8181 run this instead
Copy code
opa run --server -b bundle.tar.gz -a :8185
You would then GET /v1/policies and /v1/data to check out everything makes sense.
z
Let me try creating the bundle that way. I have ran the current bundle I have with
Copy code
opa run --server -b bundle.tar.gz
And all the policy/data files loaded correctly (verified with the endpoints you mentioned above). But then when I tried to load that into Opal via an nginx bundle server it would not load.
Same effect when I try to bundle using
Copy code
.\opa_windows_amd64.exe build bundle-dir/
It runs correctly when I run OPA on it's own using the bundle. But then when I try to run with OPAL those same endpoints are empty.
Are there any changes I should be making to the 
nging.conf
file in the example for different bundle files? Or maybe some other setting in the docker-compose that should change based on the bundle file contents?
a
there might be some settings in your opal config that causes this. could you please provide your opal config and your opal client and server logs so i can try to reproduce this?
z
Copy code
opal_server:
    image: permitio/opal-server:0.7.3
    container_name: company_name-opal-server
    environment:
      - OPAL_BROADCAST_URI=<redis://redis:6379>
      - UVICORN_NUM_WORKERS=4
      - OPAL_POLICY_BUNDLE_URL=<http://api_policy_source_server>
      - OPAL_POLICY_SOURCE_TYPE=API
      - OPAL_POLICY_REPO_CLONE_PATH=~/opal
      - OPAL_POLICY_REPO_POLLING_INTERVAL=30
      - OPAL_DATA_CONFIG_SOURCES={"config":{"entries":[{"url":"<http://host.docker.internal:7002/policy-data>","topics":["policy_data"],"dst_path":"/static"}]}}
      - OPAL_LOG_FORMAT_INCLUDE_PID=true
    ports:
      - "7002:7002"
    depends_on:
      - redis
    restart: unless-stopped
    networks:
     - company_name

  opal_client:
    image: permitio/opal-client:0.7.3
    container_name: company_name-opal-client
    environment:
      - OPAL_SERVER_URL=<http://opal_server:7002>
      - OPAL_LOG_FORMAT_INCLUDE_PID=true
      - OPAL_INLINE_OPA_LOG_FORMAT=http
    ports:
      - "7766:7000"
      - "8181:8181"
    depends_on:
      - opal_server
    command: sh -c "exec ./wait-for.sh opal_server:7002 --timeout=20 -- ./start.sh"
    restart: unless-stopped
    networks:
     - company_name

  api_policy_source_server:
    image: nginx
    networks:
      - company_name
    ports:
      - 8000:80
    volumes:
      - ./src/company_name.Authorization.WebApi/docker_files/bundle_files:/usr/share/nginx/html
      - ./src/company_name.Authorization.WebApi/docker_files/nginx.conf:/etc/nginx/nginx.conf
bundle.tar.gz
Opal server logs
Copy code
2023-09-11 11:36:03 [2023-09-11 16:36:03 +0000] [1] [INFO] Starting gunicorn 20.1.0
2023-09-11 11:36:03 [2023-09-11 16:36:03 +0000] [1] [INFO] Listening at: <http://0.0.0.0:7002> (1)
2023-09-11 11:36:03 [2023-09-11 16:36:03 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
2023-09-11 11:36:04 2023-09-11 16:36:04.943 | WARNING  | __config__:when_ready:34 - Finished pre loading scopes...
2023-09-11 11:36:04 [2023-09-11 16:36:04 +0000] [10] [INFO] Booting worker with pid: 10
2023-09-11 11:36:05 [2023-09-11 16:36:05 +0000] [11] [INFO] Booting worker with pid: 11
2023-09-11 11:36:05 [2023-09-11 16:36:05 +0000] [12] [INFO] Booting worker with pid: 12
2023-09-11 11:36:05 [2023-09-11 16:36:05 +0000] [13] [INFO] Booting worker with pid: 13
2023-09-11 11:36:05 2023-09-11T16:36:05.117791+0000 | 10 | opal_server.server                      | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2023-09-11 11:36:05 2023-09-11T16:36:05.119634+0000 | 10 | opal_server.pubsub                      | INFO  | Initializing broadcaster for server<->server communication
2023-09-11 11:36:05 2023-09-11T16:36:05.141331+0000 | 11 | opal_server.server                      | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2023-09-11 11:36:05 2023-09-11T16:36:05.142459+0000 | 11 | opal_server.pubsub                      | INFO  | Initializing broadcaster for server<->server communication
2023-09-11 11:36:05 2023-09-11T16:36:05.230623+0000 | 12 | opal_server.server                      | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2023-09-11 11:36:05 2023-09-11T16:36:05.231681+0000 | 12 | opal_server.pubsub                      | INFO  | Initializing broadcaster for server<->server communication
2023-09-11 11:36:05 2023-09-11T16:36:05.237128+0000 | 13 | opal_server.server                      | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2023-09-11 11:36:05 2023-09-11T16:36:05.238066+0000 | 13 | opal_server.pubsub                      | INFO  | Initializing broadcaster for server<->server communication
2023-09-11 11:36:05 2023-09-11T16:36:05.281558+0000 | 10 | opal_server.server                      | INFO  | *** OPAL Server Startup ***
2023-09-11 11:36:05 2023-09-11T16:36:05.281935+0000 | 10 | opal_server.server                      | INFO  | leadership lock acquired, leader pid: 10
2023-09-11 11:36:05 2023-09-11T16:36:05.282303+0000 | 10 | opal_common.topics.publisher            | INFO  | started broadcaster keepalive task: topic is '__broadcast_session_keepalive__', interval is 3600 seconds
2023-09-11 11:36:05 2023-09-11T16:36:05.283690+0000 | 10 | opal_server.policy.watcher.factory      | INFO  | Policy repo will be cloned to: //opal/opal/opal_repo_clone
2023-09-11 11:36:05 2023-09-11T16:36:05.284081+0000 | 10 | opal_server.policy.watcher.task         | INFO  | Launching policy watcher
2023-09-11 11:36:05 2023-09-11T16:36:05.285173+0000 | 10 | opal_server.data.data_update_publisher  | INFO  | [10] Starting Polling Updates
2023-09-11 11:36:05 2023-09-11T16:36:05.285647+0000 | 10 | fastapi_websocket_pubsub.event_broadc...| INFO  | Listening for incoming events from broadcast channel (first listener started)
2023-09-11 11:36:05 2023-09-11T16:36:05.286037+0000 | 10 | opal_server.policy.watcher.task         | INFO  | listening on webhook topic: 'webhook'
2023-09-11 11:36:05 2023-09-11T16:36:05.286663+0000 | 10 | fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': '8f25d9f94f99444da6251963ea19c520', 'subscriber_id': '7ef393f2e0314ad080a8c9442e7ef702', 'topic': 'webhook', 'callback': <bound method BasePolicyWatcherTask._on_webhook of <opal_server.policy.watcher.task.PolicyWatcherTask object at 0x7f7d9a27fc70>>, 'notifier_id': None}
2023-09-11 11:36:05 2023-09-11T16:36:05.288654+0000 | 11 | opal_server.server                      | INFO  | *** OPAL Server Startup ***
2023-09-11 11:36:05 2023-09-11T16:36:05.289663+0000 | 10 | fastapi_websocket_pubsub.event_broadc...| INFO  | Starting broadcaster listener
2023-09-11 11:36:05 2023-09-11T16:36:05.340086+0000 | 10 | asyncio_redis.connection                | INFO  | Connecting to redis
2023-09-11 11:36:05 2023-09-11T16:36:05.345214+0000 | 10 | asyncio_redis.protocol                  | INFO  | Redis connection made
2023-09-11 11:36:05 2023-09-11T16:36:05.345661+0000 | 10 | asyncio_redis.connection                | INFO  | Connecting to redis
2023-09-11 11:36:05 2023-09-11T16:36:05.348097+0000 | 10 | asyncio_redis.protocol                  | INFO  | Redis connection made
2023-09-11 11:36:05 2023-09-11T16:36:05.369065+0000 | 13 | opal_server.server                      | INFO  | *** OPAL Server Startup ***
2023-09-11 11:36:05 2023-09-11T16:36:05.374634+0000 | 12 | opal_server.server                      | INFO  | *** OPAL Server Startup ***
2023-09-11 11:36:07 2023-09-11T16:36:07.054923+0000 | 13 | fastapi_websocket_pubsub.event_broadc...| INFO  | Listening for incoming events from broadcast channel (first listener started)
2023-09-11 11:36:07 2023-09-11T16:36:07.054989+0000 | 12 | fastapi_websocket_pubsub.event_broadc...| INFO  | Listening for incoming events from broadcast channel (first listener started)
2023-09-11 11:36:07 2023-09-11T16:36:07.083304+0000 | 13 | fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': 'e59d1431e7d545c09dd61f6dbaf55c65', 'subscriber_id': '8fb496dfad5041e9963f55f808eeaa35', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7f7d9a3f0220>>, 'notifier_id': None}
2023-09-11 11:36:07 2023-09-11T16:36:07.083703+0000 | 13 | fastapi_websocket_pubsub.event_broadc...| INFO  | Starting broadcaster listener
2023-09-11 11:36:07 2023-09-11T16:36:07.083922+0000 | 13 | asyncio_redis.connection                | INFO  | Connecting to redis
2023-09-11 11:36:07 2023-09-11T16:36:07.084161+0000 | 12 | fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': '6081761e871b425986b7ab621fe6298f', 'subscriber_id': '31d84766cd5947c5a7ee9197906f8eb9', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7f7d9a3f0160>>, 'notifier_id': None}
2023-09-11 11:36:07 2023-09-11T16:36:07.084630+0000 | 12 | fastapi_websocket_pubsub.event_broadc...| INFO  | Starting broadcaster listener
2023-09-11 11:36:07 2023-09-11T16:36:07.084928+0000 | 12 | asyncio_redis.connection                | INFO  | Connecting to redis
2023-09-11 11:36:07 2023-09-11T16:36:07.086670+0000 | 13 | fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client connected
2023-09-11 11:36:07 2023-09-11T16:36:07.088370+0000 | 12 | fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client connected
2023-09-11 11:36:07 2023-09-11T16:36:07.088422+0000 | 13 | websockets.legacy.server                | INFO  | connection open
2023-09-11 11:36:07 2023-09-11T16:36:07.089336+0000 | 13 | asyncio_redis.protocol                  | INFO  | Redis connection made
2023-09-11 11:36:07 2023-09-11T16:36:07.089668+0000 | 13 | asyncio_redis.connection                | INFO  | Connecting to redis
2023-09-11 11:36:07 2023-09-11T16:36:07.089592+0000 | 12 | websockets.legacy.server                | INFO  | connection open
2023-09-11 11:36:07 2023-09-11T16:36:07.090634+0000 | 12 | asyncio_redis.protocol                  | INFO  | Redis connection made
2023-09-11 11:36:07 2023-09-11T16:36:07.091049+0000 | 12 | asyncio_redis.connection                | INFO  | Connecting to redis
2023-09-11 11:36:07 2023-09-11T16:36:07.093566+0000 | 13 | asyncio_redis.protocol                  | INFO  | Redis connection made
2023-09-11 11:36:07 2023-09-11T16:36:07.094429+0000 | 12 | asyncio_redis.protocol                  | INFO  | Redis connection made
2023-09-11 11:36:07 2023-09-11T16:36:07.096013+0000 | 13 | fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': '03e09adb98f54b2b877985b870d8413d', 'subscriber_id': '1366758b130a4f75843b79d179fdce4f', 'topic': 'policy_data', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7f7d999aba30>, 'notifier_id': None}
2023-09-11 11:36:07 2023-09-11T16:36:07.096397+0000 | 12 | fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': '7557ff9f1680484db35acb5115cdd322', 'subscriber_id': '968be98f87354fb1bef7d08927e38b7e', 'topic': 'policy:.', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7f7d999aba30>, 'notifier_id': None}
2023-09-11 11:36:07 2023-09-11T16:36:07.104369+0000 | 12 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.192.27:35018 - "GET /policy?path=. HTTP/1.1" 503
2023-09-11 11:36:07 2023-09-11T16:36:07.104405+0000 | 13 | opal_server.data.api                    | INFO  | Serving source configuration
2023-09-11 11:36:07 2023-09-11T16:36:07.105030+0000 | 13 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.192.27:35032 - "GET /data/config HTTP/1.1" 200
2023-09-11 11:36:07 2023-09-11T16:36:07.116588+0000 | 12 | opal_server.data.api                    |WARNING | Serving default all-data route, meaning DATA_CONFIG_SOURCES was not configured!
2023-09-11 11:36:07 2023-09-11T16:36:07.117065+0000 | 12 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.192.1:37230 - "GET /policy-data HTTP/1.1" 200
2023-09-11 11:36:07 2023-09-11T16:36:07.191442+0000 | 13 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.192.27:35048 - "GET /policy?path=. HTTP/1.1" 503
2023-09-11 11:36:07 2023-09-11T16:36:07.227118+0000 | 12 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.192.27:35054 - "GET /policy?path=. HTTP/1.1" 503
2023-09-11 11:36:10 2023-09-11T16:36:10.227455+0000 | 12 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.192.27:35070 -
Opal Client Logs
Copy code
2023-09-11 11:36:05 [2023-09-11 16:36:05 +0000] [1] [INFO] Starting gunicorn 20.1.0
2023-09-11 11:36:05 [2023-09-11 16:36:05 +0000] [1] [INFO] Listening at: <http://0.0.0.0:7000> (1)
2023-09-11 11:36:05 [2023-09-11 16:36:05 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
2023-09-11 11:36:05 [2023-09-11 16:36:05 +0000] [17] [INFO] Booting worker with pid: 17
2023-09-11 11:36:06 2023-09-11T16:36:06.000827+0000 | 17 | opal_client.policy_store.opa_client     | INFO  | Authentication mode for policy store: PolicyStoreAuth.NONE
2023-09-11 11:36:06 2023-09-11T16:36:06.003584+0000 | 17 | opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'FastApiRpcFetchProvider' found at: <class 'opal_common.fetcher.providers.fastapi_rpc_fetch_provider.FastApiRpcFetchProvider'>
2023-09-11 11:36:06 2023-09-11T16:36:06.003815+0000 | 17 | opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'HttpFetchProvider' found at: <class 'opal_common.fetcher.providers.http_fetch_provider.HttpFetchProvider'>
2023-09-11 11:36:06 2023-09-11T16:36:06.003914+0000 | 17 | opal_common.fetcher.fetcher_register    | INFO  | Fetcher Register loaded
2023-09-11 11:36:06 2023-09-11T16:36:06.004425+0000 | 17 | opal_client.callbacks.register          | INFO  | Callbacks register loaded
2023-09-11 11:36:06 2023-09-11T16:36:06.004716+0000 | 17 | opal_client.client                      | INFO  | API authentication disabled (public encryption key was not provided)
2023-09-11 11:36:06 2023-09-11T16:36:06.033334+0000 | 17 | opal_client.engine.runner               | INFO  | Launching engine runner
2023-09-11 11:36:06 2023-09-11T16:36:06.035064+0000 | 17 | opal_client.engine.runner               | INFO  | Running policy engine inline: opa run --server --addr=:8181 --authentication=off --authorization=off --log-level=info
2023-09-11 11:36:06 2023-09-11T16:36:06.055523+0000 | 17 | opal_client.engine.logger               | INFO  | Initializing server. {"addrs": [":8181"], "diagnostic-addrs": [], "time": "2023-09-11T16:36:06Z"}
2023-09-11 11:36:07 2023-09-11T16:36:07.040740+0000 | 17 | opal_client.engine.runner               | INFO  | Running policy engine initial start callbacks
2023-09-11 11:36:07 2023-09-11T16:36:07.041310+0000 | 17 | opal_client.policy.updater              | INFO  | Launching policy updater
2023-09-11 11:36:07 2023-09-11T16:36:07.041518+0000 | 17 | opal_client.data.updater                | INFO  | Launching data updater
2023-09-11 11:36:07 2023-09-11T16:36:07.041622+0000 | 17 | opal_client.policy.updater              | INFO  | Subscribing to topics: ['policy:.']
2023-09-11 11:36:07 2023-09-11T16:36:07.041820+0000 | 17 | opal_client.data.updater                | INFO  | Subscribing to topics: ['policy_data']
2023-09-11 11:36:07 2023-09-11T16:36:07.042048+0000 | 17 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Trying to connect to Pub/Sub server - <ws://opal_server:7002/ws>
2023-09-11 11:36:07 2023-09-11T16:36:07.042360+0000 | 17 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | Trying server - <ws://opal_server:7002/ws>
2023-09-11 11:36:07 2023-09-11T16:36:07.044193+0000 | 17 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Trying to connect to Pub/Sub server - <ws://opal_server:7002/ws>
2023-09-11 11:36:07 2023-09-11T16:36:07.044425+0000 | 17 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | Trying server - <ws://opal_server:7002/ws>
2023-09-11 11:36:07 2023-09-11T16:36:07.099197+0000 | 17 | opal_client.policy.updater              | INFO  | Connected to server
2023-09-11 11:36:07 2023-09-11T16:36:07.099455+0000 | 17 | opal_client.policy.updater              | INFO  | Refetching policy code (full bundle)
2023-09-11 11:36:07 2023-09-11T16:36:07.099844+0000 | 17 | opal_client.policy.fetcher              | INFO  | Fetching policy bundle from <http://opal_server:7002/policy>
2023-09-11 11:36:07 2023-09-11T16:36:07.100752+0000 | 17 | opal_client.data.updater                | INFO  | Connected to server
2023-09-11 11:36:07 2023-09-11T16:36:07.100883+0000 | 17 | opal_client.data.updater                | INFO  | Performing data configuration, reason: Initial load
2023-09-11 11:36:07 2023-09-11T16:36:07.101135+0000 | 17 | opal_client.data.updater                | INFO  | Getting data-sources configuration from '<http://opal_server:7002/data/config>'
2023-09-11 11:36:07 2023-09-11T16:36:07.105158+0000 | 17 | opal_common.utils                       |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
2023-09-11 11:36:07 2023-09-11T16:36:07.106516+0000 | 17 | opal_client.data.updater                | INFO  | Triggering data update with id: ef67787f92bf4b0bab5a380720c08217
2023-09-11 11:36:07 2023-09-11T16:36:07.106712+0000 | 17 | opal_client.data.updater                | INFO  | Fetching policy data
2023-09-11 11:36:07 2023-09-11T16:36:07.106852+0000 | 17 | opal_client.data.fetcher                | INFO  | Fetching data from url: <http://host.docker.internal:7002/policy-data>
2023-09-11 11:36:07 2023-09-11T16:36:07.107911+0000 | 17 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server <ws://opal_server:7002/ws>
2023-09-11 11:36:07 2023-09-11T16:36:07.119704+0000 | 17 | opal_client.data.updater                | INFO  | Saving fetched data to policy-store: source url='<http://host.docker.internal:7002/policy-data>', destination path='/static'
2023-09-11 11:36:07 2023-09-11T16:36:07.122155+0000 | 17 | opal_client.engine.logger               | INFO  | Received request.    PUT /v1/data/static
2023-09-11 11:36:07 2023-09-11T16:36:07.123039+0000 | 17 | opal_client.engine.logger               | INFO  | Sent response.       PUT /v1/data/static -> 204
2023-09-11 11:36:07 2023-09-11T16:36:07.188335+0000 | 17 | opal_client.policy.fetcher              | INFO  | Fetching policy bundle from <http://opal_server:7002/policy>
2023-09-11 11:36:07 2023-09-11T16:36:07.192115+0000 | 17 | opal_common.utils                       |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
2023-09-11 11:36:07 2023-09-11T16:36:07.224046+0000 | 17 | opal_client.policy.fetcher              | INFO  | Fetching policy bundle from <http://opal_server:7002/policy>
2023-09-11 11:36:07 2023-09-11T16:36:07.227780+0000 | 17 | opal_common.utils                       |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
2023-09-11 11:36:07 2023-09-11T16:36:07.489224+0000 | 17 | opal_client.engine.logger               | INFO  | OPA is out of date.  {"current_version": "0.53.1", "download_opa": "<https://openpolicyagent.org/downloads/v0.56.0/opa_linux_amd64>", "latest_version": "0.56.0", "release_notes": "<https://github.com/open-policy-agent/opa/releases/tag/v0.56.0>", "time": "2023-09-11T16:36:07Z"}
2023-09-11 11:36:10 2023-09-11T16:36:10.224186+0000 | 17 | opal_client.policy.fetcher              | INFO  | Fetching policy bundle from <http://opal_server:7002/policy>
2023-09-11 11:36:10 2023-09-11T16:36:10.228052+0000 | 17 | opal_common.utils                       |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
2023-09-11 11:36:13 2023-09-11T16:36:13.015201+0000 | 17 | opal_client.policy.fetcher              | INFO  | Fetching policy bundle from <http://opal_server:7002/policy>
2023-09-11 11:36:13 2023-09-11T16:36:13.020516+0000 | 17 | opal_common.utils                       |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
2023-09-11 11:36:13 2023-09-11T16:36:13.021049+0000 | 17 | opal_client.policy.fetcher              |WARNING | Failed all attempts to fetch bundle, got error: ValueError('unexpected response code while fetching bundle: 503')
2023-09-11 11:36:13 2023-09-11T16:36:13.021413+0000 | 17 | opal_client...base_policy_store_client  |ERROR  | OPA transaction failed, transaction id=0c5422601a9544379188b88a24a292ee, actions=[], error=ValueError('unexpected response code while fetching bundle: 503')
2023-09-11 11:36:13 2023-09-11T16:36:13.022186+0000 | 17 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server <ws://opal_server:7002/ws>
2023-09-11 11:45:43 2023-09-11T16:45:43.959542+0000 | 17 | opal_client.engine.logger               | INFO  | Received request.    GET /v1/policies
2023-09-11 11:45:43 2023-09-11T16:45:43.960403+0000 | 17 | opal_client.engine.logger               | INFO  | Sent response.       GET /v1/policies -> 200
2023-09-11 11:45:48 2023-09-11T16:45:48.995609+0000 | 17 | opal_client.engine.logger               | INFO  | Received request.    GET /v1/data
a
Hi @Zach Vosburgh according to your OPAL server logs, OPAL server does not even look at the policy bundle to get the policy. It's definitely a configuration issue. somewhere along the way, your docker-compose.yml and the example docker compose for api bundle server are different. to quickly find the, i suggest: 1. clone the OPAL repo 2. run the example compose:
Copy code
docker compose -f docker/docker-compose-api-policy-source-example.yml up --force-recreate
3. you should see this in the logs:
Copy code
docker-opal_server-1               | 2023-09-11T17:38:19.853062+0000 | 9 | opal_common.sources.api_policy_source   | INFO  | Fetching changes from remote: '<http://api_policy_source_server>'
...

docker-api_policy_source_server-1  | 172.19.0.4 - - [11/Sep/2023:17:38:19 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
docker-opal_server-1               | 2023-09-11T17:38:19.860149+0000 | 9 | opal_common.sources.api_policy_source   | INFO  | Not modified at: 2023-09-11 17:38:19
docker-opal_server-1               | 2023-09-11T17:38:19.860346+0000 | 9 | opal_common.sources.api_policy_source   | INFO  | No new version: current hash is: "6338ad75-b76"
4. now replace the 
bundle.tar.gz
under 
/docker/docker_files/bundle_files
with your bundle file everything should work with your bundle. Hope that's helpful, let me know if you need more assistance.
z
Hi @Asaf Cohen, My current configuration works fine when I use the bundle file from the example but breaks when I use the bundle file created from our current github repo with policy and data files. I tried what you mentioned above and ran with the OPAL repo. It worked correctly before any changes but broke in the same way my own configuration is when I introduced the different bundle file. I think that means there is an issue with the bundle file. Below are the error logs from the failed attempt using the OPAL repo.
Copy code
docker-opal_client-1               | 2023-09-11T18:03:36.151069+0000 | 15 | opal_client.policy.updater            
  | INFO  | Refetching policy code (full bundle)
docker-opal_client-1               | 2023-09-11T18:03:36.151408+0000 | 15 | opal_client.policy.fetcher            
  | INFO  | Fetching policy bundle from <http://opal_server:7002/policy>
docker-opal_client-1               | 2023-09-11T18:03:36.152332+0000 | 15 | opal_client.data.updater              
  | INFO  | Connected to server
docker-opal_client-1               | 2023-09-11T18:03:36.152465+0000 | 15 | opal_client.data.updater              
  | INFO  | Performing data configuration, reason: Initial load
docker-opal_server-1               | 2023-09-11T18:03:36.155465+0000 | 10 | opal_server.data.api                  
  | INFO  | Serving source configuration
docker-opal_client-1               | 2023-09-11T18:03:36.152524+0000 | 15 | opal_client.data.updater              
  | INFO  | Getting data-sources configuration from '<http://opal_server:7002/data/config>'
docker-opal_server-1               | 2023-09-11T18:03:36.156127+0000 | 10 | uvicorn.protocols.http.httptools_impl 
  | INFO  | 172.23.0.5:59986 - "GET /data/config HTTP/1.1" 200
docker-opal_client-1               | 2023-09-11T18:03:36.157603+0000 | 15 | opal_client.data.updater              
  | INFO  | Triggering data update with id: c1d48772887c4b4ea9ef8fcab1e18fd7
docker-opal_server-1               | 2023-09-11T18:03:36.162260+0000 | 14 | uvicorn.protocols.http.httptools_impl 
  | INFO  | 172.23.0.5:59982 - "GET /policy?path=. HTTP/1.1" 503
docker-opal_client-1               | 2023-09-11T18:03:36.157922+0000 | 15 | opal_client.data.updater              
  | INFO  | Fetching policy data
docker-opal_client-1               | 2023-09-11T18:03:36.158149+0000 | 15 | opal_client.data.fetcher              
  | INFO  | Fetching data from url: <http://host.docker.internal:7002/policy-data>
docker-opal_client-1               | 2023-09-11T18:03:36.159780+0000 | 15 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server <ws://opal_server:7002/ws>
docker-opal_client-1               | 2023-09-11T18:03:36.163264+0000 | 15 | opal_common.utils                     
  |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
docker-opal_server-1               | 2023-09-11T18:03:36.174908+0000 | 14 | opal_server.data.api                  
  |WARNING | Serving default all-data route, meaning DATA_CONFIG_SOURCES was not configured!
docker-opal_server-1               | 2023-09-11T18:03:36.175551+0000 | 14 | uvicorn.protocols.http.httptools_impl 
  | INFO  | 172.23.0.1:47088 - "GET /policy-data HTTP/1.1" 200
docker-opal_client-1               | 2023-09-11T18:03:36.178477+0000 | 15 | opal_client.data.updater              
  | INFO  | Saving fetched data to policy-store: source url='<http://host.docker.internal:7002/policy-data>', destination path='/static'
docker-opal_client-1               | 2023-09-11T18:03:36.180773+0000 | 15 | opal_client.engine.logger             
  | INFO  | Received request.    PUT /v1/data/static
docker-opal_client-1               | 2023-09-11T18:03:36.182011+0000 | 15 | opal_client.engine.logger             
  | INFO  | Sent response.       PUT /v1/data/static -> 204
docker-opal_client-1               | 2023-09-11T18:03:36.996084+0000 | 15 | opal_client.policy.fetcher            
  | INFO  | Fetching policy bundle from <http://opal_server:7002/policy>
docker-opal_client-1               | 2023-09-11T18:03:37.000018+0000 | 15 | opal_common.utils                     
  |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
docker-opal_server-1               | 2023-09-11T18:03:36.999498+0000 | 14 | uvicorn.protocols.http.httptools_impl 
  | INFO  | 172.23.0.5:59994 - "GET /policy?path=. HTTP/1.1" 503
docker-opal_client-1               | 2023-09-11T18:03:37.354239+0000 | 15 | opal_client.policy.fetcher            
  | INFO  | Fetching policy bundle from <http://opal_server:7002/policy>
docker-opal_server-1               | 2023-09-11T18:03:37.361245+0000 | 14 | uvicorn.protocols.http.httptools_impl 
  | INFO  | 172.23.0.5:60006 - "GET /policy?path=. HTTP/1.1" 503
docker-opal_client-1               | 2023-09-11T18:03:37.362317+0000 | 15 | opal_common.utils                     
  |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
docker-opal_client-1               | 2023-09-11T18:03:37.539890+0000 | 15 | opal_client.engine.logger             
  | INFO  | OPA is out of date.  {"current_version": "0.53.1", "download_opa": "<https://openpolicyagent.org/downloads/v0.56.0/opa_linux_amd64>", "latest_version": "0.56.0", "release_notes": "<https://github.com/open-policy-agent/opa/releases/tag/v0.56.0>", "time": "2023-09-11T18:03:37Z"}
docker-api_policy_source_server-1  | 172.23.0.4 - - [11/Sep/2023:18:03:39 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
docker-opal_server-1               | 2023-09-11T18:03:39.706484+0000 | 9 | opal_common.sources.api_policy_source  
 | INFO  | Not modified at: 2023-09-11 18:03:39
docker-opal_client-1               | 2023-09-11T18:03:41.130959+0000 | 15 | opal_client.policy.fetcher            
  | INFO  | Fetching policy bundle from <http://opal_server:7002/policy>
docker-opal_server-1               | 2023-09-11T18:03:41.136450+0000 | 14 | uvicorn.protocols.http.httptools_impl 
  | INFO  | 172.23.0.5:60016 - "GET /policy?path=. HTTP/1.1" 503
docker-opal_client-1               | 2023-09-11T18:03:41.137827+0000 | 15 | opal_common.utils                     
  |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
docker-api_policy_source_server-1  | 172.23.0.4 - - [11/Sep/2023:18:03:44 +0000] "GET /bundle.tar.gz HTTP/1.1" 304 0 "-" "Python/3.10 aiohttp/3.8.4" "-"
docker-opal_server-1               | 2023-09-11T18:03:44.716477+0000 | 9 | opal_common.sources.api_policy_source  
 | INFO  | Not modified at: 2023-09-11 18:03:44
docker-opal_client-1               | 2023-09-11T18:03:48.379450+0000 | 15 | opal_client.policy.fetcher            
  | INFO  | Fetching policy bundle from <http://opal_server:7002/policy>
docker-opal_server-1               | 2023-09-11T18:03:48.387243+0000 | 12 | uvicorn.protocols.http.httptools_impl 
  | INFO  | 172.23.0.5:48256 - "GET /policy?path=. HTTP/1.1" 503
docker-opal_client-1               | 2023-09-11T18:03:48.388192+0000 | 15 | opal_common.utils                     
  |WARNING | Unexpected response code 503: {'detail': 'policy repo was not found'}
docker-opal_client-1               | 2023-09-11T18:03:48.388645+0000 | 15 | opal_client.policy.fetcher            
  |WARNING | Failed all attempts to fetch bundle, got error: ValueError('unexpected response code while fetching bundle: 503')
docker-opal_client-1               | 2023-09-11T18:03:48.388921+0000 | 15 | opal_client...base_policy_store_client  |ERROR  | OPA transaction failed, transaction id=5c131edbaa6941c289bb338ef7428774, actions=[], error=ValueError('unexpected response code while fetching bundle: 503')
docker-opal_client-1               | 2023-09-11T18:03:48.389354+0000 | 15 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server <ws://opal_server:7002/ws>
a
Hi @Zach Vosburgh is the bundle file you attached previously the same one that you used now with the example compose file? If so I can try to run it myself and debug
z
@Asaf Cohen Yes it is.
This is the file structure that I have created a bundle from.
a
Hi @Zach Vosburgh, this is definitely the bundle. I changed only the bundle and getting different results as well
still investigating
👍 1
I tried to compare the contents of these two bundles: my bundle:
Copy code
❯ tree mybundle
mybundle
├── README.md
├── data.json
├── rbac.rego
└── utils.rego
your (faulty?) bundle:
Copy code
❯ tree yourbundle
yourbundle
├── MHC.Authorization
│   └── policy_files
│       ├── dashbaord_tests.rego
│       ├── default_allow_tests.rego
│       ├── forms_tests.rego
│       ├── northstar.rego
│       ├── scheduler_tests.rego
│       ├── tenant_tests.rego
│       └── workflow_tests.rego
└── data.json
i am suspecting one of the two: 1. OPAL cannot handle certain directory names (specifically 
MHC.Authorization
, not sure about the dot) 2. all your rego files have the same package name, which creates an illegal bundle - each rego file must have a unique package name. can you try to modify the bundle a bit to fix these two issues and try again?
z
This one is updated with those recommendations but I am still getting the same error.
bundle.tar.gz
a
Hi @Zach Vosburgh you have a file named 
opal_files\policy_files\northstar
which is probably not ideal. let me try to build it myself and see if i can make it work
Hi @Zach Vosburgh, i am not sure why it doesn't work with your bundle. I think this must be a bug somewhere in the api policy source. i'll open a ticket and let @Ro'e Katz investigate this.
👍 2
r
Gonna get to it very soon :)
👍 1
Hi @Zach Vosburgh, how do you create your tar.gz archive? When I try to extract is the way opal does:
Copy code
tarfile.TarFile.open(name="bundle.usernew.tar.gz").extractall()
I get:
Copy code
OSError: [Errno 30] Read-only file system: '/data.json'
Seems like all the file entries in your bundle has a path that begins with 
/
which fails the extraction (while in the example bundle there are no path prefixes at all). Opal should log any errors raised from extraction (instead it just keeps retrying and fail silently) - I will create a PR to fix that.
z
@Ro'e Katz I have been creating the bundle.tar.gz file using the
Copy code
.\opa_windows_amd64.exe build opal_files\
command.
r
Oh I see… I think OPAL bundles weren’t supposed to be the same as OPA bundles (@Or Weis am I right?) OPAL doesn’t load the bundle as is to OPA but rather just uses the files from the downloaded archive the same way it would use the files fetched from a tracked git repository. @Zach Vosburgh You should simply compress+archive the contents of 
opal_files\
into a tar.gz file (something like 
cd opal_files\ ; tar -czf bundle.tar.gz *
)
👍 1
21 Views
Open in Slack
PreviousNext
Powered by