Hi, I'm trying to follow the OPAL tutorial for con...
# opal
p
Hi, I'm trying to follow the OPAL tutorial for container playground. The command to start the containers seems to work ok, but when I send the example query (curl --request GET 'http://localhost:8181/v1/data/users' --header 'Content-Type: application/json' | python -m json.tool), I just get {} as the response? There is also a period websocket closure error in the container console... not sure if that's related?
a
Hi @phil.lewis, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
Based on the available sources, an empty
{}
response when querying
<http://localhost:8181/v1/data/users>
typically indicates that the data hasn't been successfully loaded into OPA yet (1) . The websocket closure errors you're seeing are likely related. OPAL uses websocket Pub/Sub connections to subscribe to both data and policy updates (2) . If these connections are failing, the OPAL client won't receive the initial data configuration needed to populate OPA (1) . Here are some troubleshooting steps: 1. ​*Check if all containers are running properly*​ - Ensure the OPAL server, OPAL client, broadcast channel, and OPA are all up and healthy . 2. ​*Review the OPAL client logs*​ - Look for messages indicating successful data fetching. You should see logs like "Triggering data update" and "Saving fetched data to policy-store" . 3. ​*Verify the websocket connection*​ - The OPAL client should show "Connected to server" in its logs. If you're seeing repeated connection failures or closures, this would prevent data synchronization . 4. ​*Check the data source configuration*​ - The OPAL server needs to be properly configured with
OPAL_DATA_CONFIG_SOURCES
pointing to your data source . A similar timing issue was reported where OPA wasn't ready when OPAL tried to push data, resulting in connection errors (3) . The logs showed "Opa connection error: ClientConnectorError" when trying to save data to the policy store . Could you share: • The complete logs from your OPAL client container • The docker-compose configuration you're using • Any error messages around the websocket closures This will help identify whet
To learn more, see the sources I considered: [[1] Policy syncing from API](https://docs.opal.ac/tutorials/track_an_api_bundle_server) [[2] Design](https://docs.opal.ac/overview/design) [[3] Unable to push the policy data since OPA is not ready yet in sidecar pattern](https://github.com/permitio/PDP/issues/266)
p
When the data update is triggered, there is this warning: DATA_CONFIG_SOURCES was not configured! The tutorial mentions nothing about configuring this? @Ask Inkeep
a
Thanks for tagging me @phil.lewis. I'll look through relevant sources and get back to you shortly.
The warning "DATA_CONFIG_SOURCES was not configured!" explains why you're getting an empty
{}
response. This configuration tells OPAL clients where to fetch dynamic data from (1) . OPAL has two distinct types of data sources : • *Policies (and static data) * - Comes from your Git repository • ​*Dynamic Data*​ - Requires
OPAL_DATA_CONFIG_SOURCES
configuration If you didn't set
OPAL_DATA_CONFIG_SOURCES
properly, OPAL clients will hit a default route that returns an empty dataset (empty dict) (2) . To fix this, you need to configure the
OPAL_DATA_CONFIG_SOURCES
environment variable on your OPAL server. Here's the format (1) :
Copy code
{
  "config": {
    "entries": [
      {
        "url": "<https://api.permit.io/v1/policy-config>",
        "topics": [
          "policy_data"
        ],
        "config": {
          "headers": {
            "Authorization": "Bearer FAKE-SECRET"
          }
        }
      }
    ]
  }
}
The configuration specifies : • ​*url*​ - Where to fetch the data from • ​*topics*​ - Which topics to subscribe to • ​*config*​ - Additional settings like headers for authentication • ​*dst_path*​ - Where to put the data in OPA's document tree (optional) • ​*save_method*​ - HTTP method to use when saving (optional) Without this configuration, your OPAL client doesn't know where to fetch the user data from, which is why you're seeing an empty response when querying
/v1/data/users
. Which tutorial are you following? Can you share the link or the docker-compose file you're using? This will help identify what data source configuration should be added.
To learn more, see the sources I considered: [[1] How to configure external data sources](https://docs.opal.ac/tutorials/configure_external_data_sources) [[2] GET /policy_data](https://opal-v2.permit.io/redoc#tag/Data-Updates/operation/default_all_data_policy_data_get)
p
a
Thanks for tagging me @phil.lewis. I'll look through relevant sources and get back to you shortly.
Based on the available sources, the
OPAL_DATA_CONFIG_SOURCES
configuration tells OPAL clients where to fetch dynamic data from (1) . The configuration uses the
DataSourceEntry
format, which acts as a "directive" on how to fetch data and where to place it in OPA's document tree . When using
OPAL_DATA_CONFIG_SOURCES
, you're configuring the "complete picture" data sources that are fetched after the OPAL client loads and after every disconnect between OPAL client and OPAL server . The format looks like this :
Copy code
{
  "config": {
    "entries": [
      {
        "url": "<https://api.permit.io/v1/policy-config>",
        "topics": [
          "policy_data"
        ],
        "config": {
          "headers": {
            "Authorization": "Bearer FAKE-SECRET"
          }
        }
      }
    ]
  }
}
Without this configuration, the OPAL client doesn't know where to fetch your user data from, which explains the empty
{}
response. Since you're following a specific tutorial, the docker-compose file for that tutorial should include the necessary
OPAL_DATA_CONFIG_SOURCES
configuration. If the tutorial's setup is missing this configuration or if there's an issue with how it's being passed to the container, that would cause the warning you're seeing. Can you share the exact docker-compose command or file you're using from the tutorial? This will help identify if there's a configuration issue with how
OPAL_DATA_CONFIG_SOURCES
is being set.
To learn more, see the sources I considered: [[1] How to configure external data sources](https://docs.opal.ac/tutorials/configure_external_data_sources)
p
@Ask Inkeep I'm using the docker compose curled from the permitio repo, as mentioned in the tutorial: curl -L https://raw.githubusercontent.com/permitio/opal/master/docker/docker-compose-example.yml \
docker-compose.yml && docker-compose up
a
Thanks for tagging me @phil.lewis. I'll look through relevant sources and get back to you shortly.
The warning you're seeing indicates that the
OPAL_DATA_CONFIG_SOURCES
environment variable isn't configured in the docker-compose file (1) . This variable tells OPAL clients where to fetch dynamic data from (2) . According to the documentation,
OPAL_DATA_CONFIG_SOURCES
contains directives on how to fetch the data configuration that loads into OPA cache when the OPAL client starts, and where to put it (1) . The value should be a JSON encoding of the ServerDataSourceConfig model . Here's an example of what the configuration should look like :
Copy code
{
  "config": {
    "entries": [
      {
        "url": "<https://api.permit.io/v1/policy-config>",
        "topics": [
          ...
        ],
        "config": {
          "headers": {
            "Authorization": "Bearer FAKE-SECRET"
          }
        }
      }
    ]
  }
}
Each entry is a directive that tells the OPAL client where to fetch data from (the
url
) , how to fetch it (optional
config
with headers) , and where to place it in OPA cache (using
dst_path
, which defaults to
/
if not specified) . To encode this as an environment variable, you can use Python's
json.dumps()
method to create a one-line string :
Copy code
export OPAL_DATA_CONFIG_SOURCES='{"config": {"entries": [{"url": "<https://api.permit.io/v1/policy-config>", "topics": ["policy_data"], "config": {"headers": {"Authorization": "Bearer FAKE-SECRET"}}}]}}'
If the docker-compose file from the tutorial is missing this configuration, you'll need to add it to the OPAL server's environment variables. Without it, the OPAL server serves a default empty data route, which explains why you're getting
{}
when querying
/v1/data/users
. Could you check if the docker-compose file has an
OPAL_DATA_CONFIG_SOURCES
environment variable set for the opal_server service? If not, that's the root cause of your issue.
To learn more, see the sources I considered: [[1] Data Source Configuration](https://docs.opal.ac/getting-started/running-opal/run-opal-server/data-sources) [[2] How to configure external data sources](https://docs.opal.ac/tutorials/configure_external_data_sources)
d
Hi @phil.lewis if you have any other questions, please let us know 🙂
p
Thanks, I have it working now with that config 🙂