https://linen.dev logo
Join SlackCommunities
Powered by
This message was deleted.
# opal
s
This message was deleted.
d
Here is the first request
Copy code
{
  "id": null,
  "entries": [
    {
      "url": "",
      "data": {
        "sensitivityLevel": 9
      },
      "config": {},
      "topics": [
        "policy_data"
      ],
      "dst_path": "/participants/me",
      "save_method": "POST"
    }
  ],
  "reason": "Sensitivity Level Updated",
  "callback": {
    "callbacks": []
  }
}
and here is the second request
Copy code
{
  "id": null,
  "entries": [
    {
      "url": "",
      "data": {
        "sensitivityLevel": 8
      },
      "config": {},
      "topics": [
        "policy_data"
      ],
      "dst_path": "/participants/you",
      "save_method": "POST"
    }
  ],
  "reason": "Sensitivity Level Updated",
  "callback": {
    "callbacks": []
  }
}
And here is the log. You can see the server sends the data both times but the client only sees the first one
Copy code
opal_server_1              | 2023-05-02T19:05:06.933607+0000 | 9 | opal_server.data.data_update_publisher  | INFO  | [9] Publishing data update to topics: {'policy_data'}, reason: Sensitivity Level Updated, entries: [{'url': '', 'method': 'POST', 'path': '/participants/me', 'inline_data': True, 'topics': ['policy_data']}]
opal_server_1              | 2023-05-02T19:05:06.933947+0000 | 9 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.0.1:58932 - "POST /data/config HTTP/1.1" 200
opal_server_1              | 2023-05-02T19:05:06.934412+0000 | 9 | fastapi_websocket_pubsub.event_notifier | INFO  | calling subscription callbacks: topic=policy_data, subscription_id=f9c9dcbcabe14ab4bae212ad23d00897, subscriber_id=41f4513ab94f46bfb84b9f54b4b8563c
opal_server_1              | 2023-05-02T19:05:06.934915+0000 | 9 | fastapi_websocket_pubsub.rpc_event_me...| INFO  | Notifying other side: subscription={'id': 'f9c9dcbcabe14ab4bae212ad23d00897', 'subscriber_id': '41f4513ab94f46bfb84b9f54b4b8563c', 'topic': 'policy_data', 'notifier_id': None}, data=id=None entries=[DataSourceEntry(url='', data={'sensitivityLevel': 9}, config={}, topics=['policy_data'], dst_path='/participants/me', save_method='POST')] reason='Sensitivity Level Updated' callback=UpdateCallback(callbacks=[]), channel_id=41f4513ab94f46bfb84b9f54b4b8563c
opal_server_1              | 2023-05-02T19:05:06.935600+0000 | 9 | fastapi_websocket_pubsub.event_notifier | INFO  | calling subscription callbacks: topic=policy_data (ALL_TOPICS), subscription_id=5895d392e1894dc2bfce77e9dd153ec9, subscriber_id=d03bca38d4a1497cbab91fdf8caa70b3
opal_server_1              | 2023-05-02T19:05:06.935739+0000 | 9 | fastapi_websocket_pubsub.event_broadc...| INFO  | Broadcasting incoming event: {'topic': 'policy_data', 'notifier_id': 'd03bca38d4a1497cbab91fdf8caa70b3'}
opal_client_1              | 2023-05-02T19:05:06.936836+0000 | 17 | opal_client.data.rpc                    | INFO  | Received notification of event: policy_data
opal_client_1              | 2023-05-02T19:05:06.936993+0000 | 17 | opal_client.data.updater                | INFO  | Updating policy data, reason: Sensitivity Level Updated
opal_client_1              | 2023-05-02T19:05:06.937185+0000 | 17 | opal_client.data.updater                | INFO  | Triggering data update with id: 9b1cc8dcabb5470c8afb9b70fa849fe1
opal_client_1              | 2023-05-02T19:05:06.937359+0000 | 17 | opal_client.data.updater                | INFO  | Fetching policy data
opal_client_1              | 2023-05-02T19:05:06.937498+0000 | 17 | opal_client.data.fetcher                | INFO  | Data provided inline for url: 
opal_client_1              | 2023-05-02T19:05:06.940266+0000 | 17 | opal_client.data.updater                | INFO  | Saving fetched data to policy-store: source url='', destination path='/participants/me'
opal_client_1              | 2023-05-02T19:05:06.942355+0000 | 17 | opal_client.opa.logger                  | INFO  | Received request.    PUT /v1/data/participants/me
opal_client_1              | 2023-05-02T19:05:06.943073+0000 | 17 | opal_client.opa.logger                  | INFO  | Sent response.       PUT /v1/data/participants/me -> 204
opal_client_1              | 2023-05-02T19:05:08.954261+0000 | 17 | opal_client.opa.logger                  | INFO  | Received request.    GET /v1/data/participants
opal_client_1              | 2023-05-02T19:05:08.954933+0000 | 17 | opal_client.opa.logger                  | INFO  | Sent response.       GET /v1/data/participants -> 200
kafka-ui                   | 2023-05-02 19:05:19,853 DEBUG [parallel-2] c.p.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: Default
kafka-ui                   | 2023-05-02 19:05:19,879 DEBUG [parallel-8] c.p.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: Default
opal_server_1              | 2023-05-02T19:05:39.738015+0000 | 8 | opal_server.data.data_update_publisher  | INFO  | [8] Publishing data update to topics: {'policy_data'}, reason: Sensitivity Level Updated, entries: [{'url': '', 'method': 'POST', 'path': '/participants/you', 'inline_data': True, 'topics': ['policy_data']}]
opal_server_1              | 2023-05-02T19:05:39.738553+0000 | 8 | uvicorn.protocols.http.httptools_impl   | INFO  | 192.168.0.1:58952 - "POST /data/config HTTP/1.1" 200
opal_server_1              | 2023-05-02T19:05:39.791368+0000 | 8 | fastapi_websocket_pubsub.event_notifier | INFO  | New subscription {'id': '5edd675326764b39abf7ac6dca906501', 'subscriber_id': '36fe701c1bd645a385e394002e973cf8', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7ff1287a5bd0>>, 'notifier_id': None}
opal_server_1              | 2023-05-02T19:05:39.791894+0000 | 8 | fastapi_websocket_pubsub.event_notifier | INFO  | calling subscription callbacks: topic=policy_data (ALL_TOPICS), subscription_id=5edd675326764b39abf7ac6dca906501, subscriber_id=36fe701c1bd645a385e394002e973cf8
opal_server_1              | 2023-05-02T19:05:39.792180+0000 | 8 | fastapi_websocket_pubsub.event_broadc...| INFO  | Broadcasting incoming event: {'topic': 'policy_data', 'notifier_id': '36fe701c1bd645a385e394002e973cf8'}
opal_server_1              | 2023-05-02T19:05:39.816939+0000 | 8 | fastapi_websocket_pubsub.event_notifier | INFO  | Removing Subscription of topic='__EventNotifier_ALL_TOPICS__' for subscriber=36fe701c1bd645a385e394002e973cf8
opal_client_1              | 2023-05-02T19:05:44.704751+0000 | 17 | opal_client.opa.logger                  | INFO  | Received request.    GET /v1/data/participants
opal_client_1              | 2023-05-02T19:05:44.705185+0000 | 17 | opal_client.opa.logger                  | INFO  | Sent response.       GET /v1/data/participants -> 200
o
Hi @David Hamilton, Quick question are you running with a broadcaster ? It is a must if you are running with more than one worker and/or more than one server.
d
yes, kafka broadcaster, using the docker-compose-with-kafka-example.yml
one server, one client
my goal is to transform my kafka topics containing data to EventNotifier format and update data via the kafka topic
i thought it was an issue with sending via kafka at first, but the kafka_server:7002 api behaves the same
o
This will require further investigation. @Ro'e Katz @Ori Shavit @Asaf Cohen - your assistance please. @David Hamilton - which version of OPAL are you using?
d
using latest tag, but docker shows two months ago so maybe 0.5.0
o
Worth trying 0.6.1 (latest)
d
i have to head out but I"ll try 0.6.1 tomorrow morning
👍 1
yup! will do
another thing i noticed, which happened with 0.5.0 and is still happening with 0.6.1 is that if you have the 
wait-for.sh
command on opal_server container, it fails at startup with 
nc command is missing!
then exit code 1
command: sh -c "exec /usr/wait-for.sh kafka:9092 --timeout=30 -- /start.sh"
directly from the example docker-compose-with-kafka-example.yml
BUT! the problem I had yesterday is resolved!
a
@Ro'e Katz can you please check out the issue with 
wait-for.sh
? it should work since it's being used by the docker compose examples
d
Thank you Asaf and Or!
I am now able to send mocked data to the EventNotifier topic via kafka-producer-perf-test.sh and confirm all data was created/updated in OPA!
💪 2
r
Actually I recall seeing that 
wait-for.sh
only works on client because we only install 
nc
on that image. So I think that’s an error that we still use 
wait-for.sh
there. Anyway I’ll add it to my tasks to make sure the kafka docker-compose example works. @David Hamilton Thanks for reporting, glad the other issue is now solved 🙂
👍 1
2 Views
Open in Slack
PreviousNext
Powered by