https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# opal
s
This message was deleted.
d
o
f
🫡 💪
d
As I see it, it is using the aiokafka library and calling 
aiokafka.helpers.create_ssl_context
with no parameters which ends up calling 
ssl.create_default_context
so the way i would have to set the kafka ssl connection key/cert would just be the system default for my opal-server container
I'm assuming that this would be /etc/ssl/private and /etc/ssl/certs/ca-certificates.crt by default, meaning I will need to mount my key/chain to those locations?
o
Yes I believe so, but it would be better to check the AIOKafka docs. Alternatively you can open a PR for broadcaster to add env-vars to configure the ssl context setup as well.
d
I was just about to say the same thing about the PR :-)
💜 1
I am not a python guy but this looks like it should do the trick
Don't know how to actually run these tests or if you can provide the environment variables to them
o
There isn’t a test for this, so unless you’re adding one (which would be ideal) - we’d need to settle for some manual testing. But let me start with some code review first
d
there is a basic kafka broadcaster test, so i am spinning up a local kafka instance and going to see if i can get ssl going on it
unfortunately the SSL instance I need to actually access is in k8s behind an AWS jumpbox and can't be easily accessed from my local
o
You could try Ngrok
The code seems looks good to me, at the minimum it won’t break backward compatibility
d
I ran the scripts and saw it did some reformatting, so i just pushed that
👍 1
o
Can you add a comment on the PR you are signing off on contributing the code to OSS
d
I can, I think I am getting close to getting this test workable
💪 1
Ok, got this working with a container using SSL
🔥 1
o
That’s awesome
Merged 💪
🫶 1
d
Any idea when this could get into a release version?
o
@Asaf Cohen, @Ro'e Katz?
d
Also curious if you have ever considered nightlies or weekly images based off 
development
snapshot
o
I’ll make sure to get an answer by tomorrow about the next release. Regarding nightly / weekly builds, while I’m 100% in favor of the idea; I feel the project is not quite there yet (especially with test coverage) - currently we do a lot of manual regression and integration tests as part of a release. We could use some help in improving tests, and improving coverage.
Next release of OPAL by (@Ro'e Katz) is planned for start of next week
d
Great, thank you! And that will include the latest from the 
broadcaster
repository?
o
Yes. Assuming no issues arise while doing the integration tests
r
Hi @David Hamilton, I’m releasing version 0.7.1 which also includes the newest broadcaster version. Would you please help me test how OPAL works with your new broadcaster feature? I’ve pushed a temporary image tag 
0.7.1-rc
(
permitio/opal-server:0.7.1-rc
& 
permitio/opal-client:0.7.1-rc
)
d
Sure thing! I'll pull it down now and try to get the ssl configured
r
Amazing, thanks!
d
Preliminary results it seems to be connected in that I"m not seeing any errors where I previously was. I sent a message to kafka for a data change but saw nothing in the server, so I'm going to try using 
opal-client publish-data-update
Looks like I made a mistake when requesting ACLs in our kafka instance and I don't have write access to the topic, so I'm going to get that requested and follow up
I have made the request, it is a separate team in my organization that manages it, so I will update when they have made the change. FYI when I do a data update through the OPAL Server API, I just get the following message, but no error. However, the data is not published to the topic, which I would expect at this point since the user ACL does not have write.
Copy code
20T15:12:08.310459+0000 | opal_server.data.data_update_publisher  | INFO  | [10] Publishing data update to topics: ...
I would expect there should be an error when the broadcast fails though
r
@David Hamilton That’s strange… and nothing printed afterwards? (maybe 
connecting to brokers: ...
). In the meantime I’ve managed to set up a docker-compose setup with Kafka in SSL - I did manage to make it work with the new OPAL version!
d
I get several sets of messages like this
Copy code
[32m2023-06-20T16:40:16.760331+0000[0m | [34mfastapi_websocket_pubsub.event_broadc...[0m|[1m INFO  | Listening for incoming events from broadcast channel (first listener started)[0m
[32m2023-06-20T16:40:16.822034+0000[0m | [34maiokafka.helpers                        [0m|[1m INFO  | Loading SSL CA from /etc/ssl/certs/ca-certificates.crt[0m
[32m2023-06-20T16:40:16.825038+0000[0m | [34maiokafka.helpers                        [0m|[1m INFO  | Loading SSL Cert from /etc/ssl/private/opal.baaz.dev.bip.va.gov.pem[0m
[32m2023-06-20T16:40:16.825212+0000[0m | [34maiokafka.helpers                        [0m|[1m INFO  | Loading SSL Key from /etc/ssl/private/opal.baaz.dev.bip.va.gov.key with password[0m
[32m2023-06-20T16:40:16.826155+0000[0m | [34mfastapi_websocket_pubsub.event_broadc...[0m|[1m INFO  | Starting broadcaster listener[0m
[32m2023-06-20T16:40:16.826369+0000[0m | [34maiokafka.helpers                        [0m|[1m INFO  | Loading SSL CA from /etc/ssl/certs/ca-certificates.crt[0m
[32m2023-06-20T16:40:16.827359+0000[0m | [34maiokafka.helpers                        [0m|[1m INFO  | Loading SSL Cert from /etc/ssl/private/opal.baaz.dev.bip.va.gov.pem[0m
[32m2023-06-20T16:40:16.827499+0000[0m | [34maiokafka.helpers                        [0m|[1m INFO  | Loading SSL Key from /etc/ssl/private/opal.baaz.dev.bip.va.gov.key with password[0m
[32m2023-06-20T16:40:16.828704+0000[0m | [34mwebsockets.legacy.server                [0m|[1m INFO  | connection open[0m
[32m2023-06-20T16:40:16.829367+0000[0m | [34mwebsockets.legacy.server                [0m|[1m INFO  | connection closed[0m
[32m2023-06-20T16:40:19.619582+0000[0m | [34mopal_server.data.api                    [0m|[1m INFO  | Serving source configuration[0m
the connection open/closed though appears to be a new thing with the client, so I am looking into that and if the keystore stuff somehow changed the client/server connection
Copy code
[32m2023-06-20T16:41:25.397039+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_e...[0m|[1m INFO  | Client connected[0m
[32m2023-06-20T16:41:25.397589+0000[0m | [34mwebsockets.legacy.server                [0m|[1m INFO  | connection open[0m
[32m2023-06-20T16:41:25.398237+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_e...[0m|[1m INFO  | Client connection failed - 35416 :: 1be0af7f2fea43de8db567317b64cf00[0m
[32m2023-06-20T16:41:25.401158+0000[0m | [34mwebsockets.legacy.server                [0m|[1m INFO  | connection closed[0m
[32m2023-06-20T16:41:28.203104+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_e...[0m|[1m INFO  | Client connected[0m
[32m2023-06-20T16:41:28.203725+0000[0m | [34mwebsockets.legacy.server                [0m|[1m INFO  | connection open[0m
[32m2023-06-20T16:41:28.204304+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_e...[0m|[1m INFO  | Client connection failed - 35418 :: 7d92e954a9e54052bb5bf683293a6b97[0m
[32m2023-06-20T16:41:28.206912+0000[0m | [34mwebsockets.legacy.server                [0m|[1m INFO  | connection closed[0m
r
The client should get disconnections when the server’s broadcaster connection is disconnected. So that adds up
d
ah, ok
In the meantime while I wait on my ACL request, I've added a new PR to broadcaster adding a test verifying the SSL params I added in PR #5. https://github.com/permitio/broadcaster/pull/6
💜 1
Open in Slack
PreviousNext
Powered by