This message was deleted Permit #opal

Join Slack

This message was deleted.

# opal

Slackbot

02/13/2023, 5:00 PM

This message was deleted.

Gage Miller

02/13/2023, 5:06 PM

I set headers in the config and I noticed that my responses have more properties in the config block than the opal server when it hands back data sources.

Copy code

url: <http://mock>
config: {
  fetcher: null,  
  headers: {
    Authorization: mock
  },
  is_json: true,
  process_data: true,
  method: get,
  data: null
}
...

Shuvy Ankor

02/13/2023, 5:17 PM

Hello @Gage Miller , I’ll check with the team members and get back to you

Or Weis

02/13/2023, 5:20 PM

Hi @Gage Miller can you share client and server logs ?

Gage Miller

02/13/2023, 7:09 PM

yeah I can get that to you, just need to redact some stuff

Gage Miller

02/13/2023, 7:16 PM

Copy code

opal_server_1        | 2023-02-13T18:51:12.637352+0000 | 10 | fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client connected
opal_server_1        | 2023-02-13T18:51:12.637407+0000 | 9 | fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client connected
opal_server_1        | 2023-02-13T18:51:12.637637+0000 | 9 | fastapi_websocket_pubsub.event_broadc...| INFO  | Starting broadcaster listener
opal_server_1        | 2023-02-13T18:51:12.638065+0000 | 10 | fastapi_websocket_pubsub.event_broadc...| INFO  | Starting broadcaster listener
opal_server_1        | 2023-02-13T18:51:12.640800+0000 | 9 | websockets.legacy.server                | INFO  | connection open
opal_server_1        | 2023-02-13T18:51:12.641807+0000 | 10 | websockets.legacy.server                | INFO  | connection open
opal_server_1        | 2023-02-13T18:51:12.687242+0000 | 9 | fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': 'c4f014a892c0465f998ac2f2a070f4f2', 'subscriber_id': 'b6f7f4c1f8014804b067cc166b0ba3a0', 'topic': 'policy:banking/src', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7f58b7219090>, 'notifier_id': None}
opal_server_1        | 2023-02-13T18:51:12.687730+0000 | 10 | fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': '5c3c90d6a0c44815818e805ca482599b', 'subscriber_id': '6827534c9b3c492088d76d25bd933ad8', 'topic': 'policy_data', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7f58b7219000>, 'notifier_id': None}
opal_client_1        | 2023-02-13T18:51:12.690284+0000 | 16 | opal_client.policy.updater              | INFO  | Connected to server
opal_client_1        | 2023-02-13T18:51:12.690565+0000 | 16 | opal_client.policy.updater              | INFO  | Refetching policy code (full bundle)
opal_client_1        | 2023-02-13T18:51:12.690940+0000 | 16 | opal_client.policy.fetcher              | INFO  | Fetching policy bundle from <http://opal_server:7002/policy>
opal_client_1        | 2023-02-13T18:51:12.693010+0000 | 16 | opal_client.data.updater                | INFO  | Connected to server
opal_client_1        | 2023-02-13T18:51:12.693165+0000 | 16 | opal_client.data.updater                | INFO  | Performing data configuration, reason: Initial load
opal_client_1        | 2023-02-13T18:51:12.693256+0000 | 16 | opal_client.data.updater                | INFO  | Getting data-sources configuration from '<http://opal_server:7002/data/config>'
opal_server_1        | 2023-02-13T18:51:12.696450+0000 | 9 | opal_server.data.api                    | INFO  | Source configuration is available at '<http://snapshot_api:8001/data/config>', redirecting with token=eyJhb...l9T5o (abbrv.)
opal_server_1        | 2023-02-13T18:51:12.696863+0000 | 9 | uvicorn.protocols.http.httptools_impl   | INFO  | 172.18.0.6:52412 - "GET /data/config HTTP/1.1" 307
opal_server_1        | 2023-02-13T18:51:12.707087+0000 | 10 | opal_common.git.bundle_maker            | INFO  | Using root manifest dir path (new-fashioned): '.'
opal_server_1        | 2023-02-13T18:51:12.707258+0000 | 10 | opal_common.git.bundle_maker            | INFO  | Compiling manifest file .manifest
snapshot_api_1       | 2023-02-13T18:51:12.707689+0000 | 1 | src.main                                | INFO  | <http://snapshot_api:8001>
snapshot_api_1       | 2023-02-13T18:51:12.709038+0000 | 1 | uvicorn.protocols.http.httptools_impl   | INFO  | 172.18.0.6:44888 - "GET /data/config?token=eyJhb[redacted]l9T5o HTTP/1.1" 200
opal_client_1        | 2023-02-13T18:51:12.710342+0000 | 16 | opal_client.data.updater                | INFO  | Triggering data update with id: 4f88fa8eb29648268410023dc4e7f969
opal_client_1        | 2023-02-13T18:51:12.710588+0000 | 16 | opal_client.data.updater                | INFO  | Fetching policy data
opal_client_1        | 2023-02-13T18:51:12.710962+0000 | 16 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server <ws://opal_server:7002/ws>
opal_server_1        | 2023-02-13T18:51:12.710725+0000 | 10 | opal_common.git.bundle_maker            | INFO  | Manifest file .manifest not found, assuming empty
opal_server_1        | 2023-02-13T18:51:12.725064+0000 | 10 | uvicorn.protocols.http.httptools_impl   | INFO  | 172.18.0.6:52406 - "GET /policy?path=banking/src HTTP/1.1" 200
opal_client_1        | 2023-02-13T18:51:12.726652+0000 | 16 | opal_client.policy.fetcher              | INFO  | Fetched valid bundle, id: ca7207c4b7afa2cd49e037722b3047aed59302f9
opal_client_1        | 2023-02-13T18:51:12.728005+0000 | 16 | opal_client.policy.updater              | INFO  | Got policy bundle with 13 rego files, 0 data files, commit hash: 'ca7207c4b7afa2cd49e037722b3047aed59302f9'
opal_client_1        | 2023-02-13T18:51:12.731241+0000 | 16 | opal_client.opa.logger                  | INFO  | Received request.    GET /v1/policies
opal_client_1        | 2023-02-13T18:51:12.732085+0000 | 16 | opal_client.opa.logger                  | INFO  | Sent response.       GET /v1/policies -> 200
opal_client_1        | 2023-02-13T18:51:12.734690+0000 | 16 | opal_client.opa.logger                  | INFO  | Received request.    PUT /v1/policies/banking/src/redact1.rego
opal_client_1        | 2023-02-13T18:51:12.739335+0000 | 16 | opal_client.opa.logger                  | INFO  | Sent response.       PUT /v1/policies/banking/src/redact1.rego -> 200
...
opal_client_1        | 2023-02-13T18:51:12.743492+0000 | 16 | opal_client.opa.logger                  | INFO  | Received request.    PUT /v1/policies/banking/src/redact2.rego
opal_client_1        | 2023-02-13T18:51:12.749069+0000 | 16 | opal_client.opa.logger                  | INFO  | Sent response.       PUT /v1/policies/banking/src/redact2.rego -> 200
opal_client_1        | 2023-02-13T18:51:12.961186+0000 | 16 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server <ws://opal_server:7002/ws>

Or Weis

02/13/2023, 7:38 PM

Hard to tell - but my guess is your data-config is malformed- perhaps missing the list of entries. Can you share an example of what

<http://snapshot_api:8001>

is returning ?

Gage Miller

02/13/2023, 7:42 PM

I switched my version to 0.3.1 and things started working again. Looks like it is an issue in 0.4.0 release

Gage Miller

02/13/2023, 7:42 PM

@Or Weis I tried using the raw data config sources instead of external server and it still did execute the fetch requests to my desired url

Or Weis

02/13/2023, 7:48 PM

Could be an issue with 0.4.0 - though I’m not aware of any changes in that area @Asaf Cohen

Or Weis

02/13/2023, 7:49 PM

it still did execute the fetch

did or didn’t ?

Gage Miller

02/13/2023, 7:58 PM

It did not fetch, this feels like a pretty big breaking change so I am surprised you guys have not run into it. Maybe I have some config missing that you guys don't?

Gage Miller

02/13/2023, 7:59 PM

I found this change relating to data entries https://github.com/permitio/opal/pull/339/files

Gage Miller

02/13/2023, 8:02 PM

I don't have OPAL_DATA_TOPICS set. The behavior before would subscribe to all topics without having to be explicit

Or Weis

02/13/2023, 8:22 PM

@Ro'e Katz

Or Weis

02/13/2023, 9:22 PM

Sorry for the delay here was In a meeting. Yeah the change here does seem relevant to what you’re experiencing - though I’d say you should always prefer to set data topics -otherwise you’ll have challenges with subsequent data updates And to quote the Zen of Python - Explicit is better than implicit. You can of course solve this by adding OPAL_DATA_TOPICS to your client; If you feel the implicit option is important we can add another flag to enable that effect; But as I said I do think the added explicitness is an improvement

Gage Miller

02/13/2023, 11:25 PM

I set data topics and still no luck. Do you have an example anywhere that I can cross reference?

Or Weis

02/13/2023, 11:50 PM

I suggest we wait for tomorrow morning (GMT+2) for @Asaf Cohen to chime in here; he’ll probably have better pointers; I’m assuming you can continue with 0.3.X for now.

👍 1

Or Weis

02/14/2023, 12:51 AM

There’s actually a PR I opened around this a earlier this month: https://github.com/permitio/opal/pull/376 I forgot all about it, and so did @Ro'e Katz it seems…

Asaf Cohen

02/14/2023, 7:34 AM

Hi @Gage Miller i will investigate with @Ro'e Katz and get back to you

👀 1

Ro'e Katz

02/14/2023, 12:02 PM

@Gage Miller Bottom line: 1. I’ll work on a fix and try to release it later today. 2. To make it work in the meantime - you can set the

topics

field in each data source entry to include some topic that the client listens to. a. clients listen to

"policy_data"

by default (if you don’t override

DATA_TOPICS

) - so you can use that. b. Let me know if that works :) Explanation: I’ve fixed a bug that on some cases caused clients to digest data updates not meant for them. As a side product the client won’t process a data update unless it has one of the subscribed topics, even if the data update doesn’t have topics at all. That’s indeed the desired behavior for updates triggered by server (git repos etc), I think that’s also the desired behavior for updates from

OPAL_DATA_CONFIG_SOURCES

, but I agree that’s breaking current behavior - as there was never an enforcement on those having non empty

topics

Ro'e Katz

02/14/2023, 3:38 PM

Hi @Gage Miller - Can you please confirm that setting a topic in all of your source entries (that matches the client’s subscribed topics) solves your issue? just to make sure we’re talking about the same issue. As for my fix - we haven’t been able to come up with a coherent way to support “topic-less” data source entries. I’ll look into maybe having

["policy_data"]

as a default value for

DataSourceEntry.topics

. Anyhow I’ll fix 0.4.0's docs to reflect the changes.

Gage Miller

02/14/2023, 3:42 PM

@Ro'e Katz I am setting the topic on data source entries but it isn't fetching them. I tried setting opal data topics and not setting data topics on the opal client but it still doesn't work.

Ro'e Katz

02/14/2023, 3:43 PM

What topic did you try to set?

Gage Miller

02/14/2023, 3:47 PM

I am setting this topic in my entry "topics":["myExample"] and then on the client I did OPAL_DATA_TOPICS=["myExample"]

Gage Miller

02/14/2023, 3:51 PM

I am always setting the topics on the entries, I was talking about explicitly setting OPAL_DATA_TOPICS on the clients.

Gage Miller

02/14/2023, 3:58 PM

I think I figured out the issue. I tried OPAL_DATA_TOPICS=["topic1","topic2"], I need to remove the brackets in the config for OPAL_DATA_TOPICS

Or Weis

02/14/2023, 4:09 PM

🤘

Gage Miller

02/14/2023, 5:21 PM

Yeah my bad here, can the docs be updated to show the OPAL_DATA_TOPICS being used? I think I got the wrong idea of how they worked because of the bug present in 0.3.1

Gage Miller

02/14/2023, 5:23 PM

I thought that the data config sources set the topics that the client listened to.

Or Weis

02/14/2023, 5:29 PM

For sure. @Ro'e Katz as you’re on a roll here - can you take care of that ?

Ro'e Katz

02/14/2023, 7:13 PM

Sure

🙏 3

Ro'e Katz

02/16/2023, 4:18 PM

Just merged that: https://github.com/permitio/opal/pull/389 Which I think should overall improve the experience of using (or forgetting to use) data topics. It also improves the documentation in a few places. @Gage Miller Regarding the use of

OPAL_DATA_TOPICS

- have you seen this page? Do you think it’s clear enough?

Open in Slack

Previous Next