https://linen.dev logo
Join Slack
Powered by
In case anyone is considering letting customers tr...
# general
g
In case anyone is considering letting customers train AI models on their K8s: https://pivot-to-ai.com/2024/07/28/sapwned-saps-ai-cloud-cracked-by-simple-kubernetes-configuration-errors/
👀 1
c
Not planning on letting people train AI on our cloud (I think AI is a bunch of nonsense, and yes I do use copilot and ChatGPT), but we do plan to allow running custom code. The security considerations are a doozy. Before I "okay" the release of this feature, we'll have to have: • OPA to enforce no containers run as root • OPA to enforce read-only filesystem for containers minus volumes • K8s 
NetworkPolicy
to ensure pods can't talk to things they shoudln't be able to • Ensure they don't have any 
ServiceAccount
allowing any unsavory access to stuff • Resource limits • Proper ACL's on any resources they do have access to (eg. kafka topics)
I find it hard to believe that a company as big as SAP could mess up that bigly
g
The Wiz article has all the technical details, which could be helpful as pointers: https://www.wiz.io/blog/sapwned-sap-ai-vulnerabilities-ai-security
I'm pretty impressed by Wiz's work, but then again... thats what they do.
c
Oh my goodness that's insane. Cluster admin access????
I guess if you can run arbitrary code by uploading a model, then you just give yourself a shell and have a field day...
g
to be fair, they had to get shell by getting access to the istio sidecar and from there get the istio identity which didn't have as many restrictions...
c
Yeah, reading that... (I guess I was live-tweeting my read)
g
It is truly wild. So many configs that look innocent...
c
The first is 
shareProcessNamespace
, which allowed us to share the process namespace with our sidecar container. Since our sidecar was the Istio proxy, we gained access to Istio’s configuration, including an access token to the cluster’s centralized Istiod server.
That seems like a weakness in Istio. There should be by default a 
ValidatingAdmissionWebhook
which rejects pods with that configuration
👍 1
First vulnerability seems to be "well I have Istio so my network is secure".
g
yeah... not if you allow custom code execution
c
Okay, I'm changing the material for our next Weekly Study Hall 😂
g
I am now wondering what you can do with Confluent cloud...
c
Y'all don't use Istio though? I mean, a self-managed bring your own connector is quite literally running a java application. The only thing is that allowing someone to create an Argo 
WorkflowTemplate
lets them configure some things about the pod that Confluent probably can hard-code.
Wow, they still use Tiller? 😱
g
Yeah. I was thinking about bring-your-connector. I am not sure if they use Istio, but there are other sidecars for sure.
There was a plan to run those connectors inside WASM sandboxes... not sure if that happened.
c
That's next-level
But still, a connector needs network connectivity (eg. JDBC must talk to customer's postgres)
So you can only sandbox so much.
g
yeah
A lot of people don't trust containers, so they run in MicroVMs (Qemu and such). But given the nature of the attack, I'm not sure Qemu isolation would help here.
Open in Slack
PreviousNext
Powered by