Using OPA to block malicious annotations in Kubernetes
August 21, 2021
My friend Jared Stroud recently wrote a great article about abusing annotations in Kubernetes as storage space for malicious payloads. It’s a great read and a clever idea, especially since most K8s admins probably pay little attention to what is going on with their annotations.
As I was reading his article, I kept thinking to myself: this seems like something that a ValidatingAdmissionWebhook could prevent. If you’re unfamiliar with admission controllers in Kubernetes, I’d recommend a look over the official docs. While ValidatingAdmissionWebhook may sound intimidating, it’s a simple concept: inspect API calls to Kubernetes for CRUD operations on resources, and make decisions about whether or not those actions should be permitted. Their implementation is also very easy: they’re just webservers that receive JSON requests from Kubernetes and reply back with their disposition (accept or reject).
I became passively interested in the ValidatingAdmissionWebhook pattern a few years ago when I saw an Open Source Summit presentation about the Open Policy Agent (OPA). OPA is a powerful project that provides a framework for authorization. The idea behind OPA tracks pretty closely with the strategy behind a ValidatingAdmissionWebhook: a JSON blob goes into OPA, it evaluates some rules, and it returns a response. While OPA isn’t limited to Kubernetes, it’s a natural fit for a ValidatingAdmissionWebhook (and the project itself provides the resources necessary to deploy it as one).
In this article, I’ll walk through how you could use OPA to mitigate the attack laid out in Jared’s article (again, if you haven’t read it: go do that first). While this approach might not hit every attack scenario, it’s a good start.
Environment Setup
Before I get started, let me go over my local environment that I used for this article. I’m running minikube version 1.20.0, which runs Kubernetes version 1.20.2, on my local laptop. However, these instructions should be applicable with any Kubernetes installation that supports OPA (e.g., has ValidatingAdmissionWebhook enabled, which is the default in the last several K8s versions).
I installed OPA using their Kubernetes tutorial, but be aware this isn’t the way you’d want to deploy it in production. You should use a real certificate signed by a real CA. The tutorial also uses --replicate and --replicate-cluster settings to force OPA to periodically cache the overall state of the Kubernetes cluster. This allows you to access existing Kubernetes objects in your OPA policies, instead of just looking at the fields sent in the admission request. For the sake of simplicity, this article doesn’t cover any of that (and you could subsequently remove those flags from the container arguments, if you wanted to).
I’m also running a simple nginx pod that I will use to demonstrate the admission control decisions throughout this article. The pod (or even Kubernetes resource) is irrelevant in this article, and just gives me something to test my kubectl annotate commands against.
For those familiar with OPA, you’ll also notice that I’m using “regular” OPA-Kubernetes, and not the newer OPA Gatekeeper. While Gatekeeper sounds like the future of the OPA project for K8s, it’s still in beta and the docs mostly involve “look at these examples on Github.” I’d love to revisit it once it’s a bit further along.
And that’s about it! If you’re running minikube (or any K8s distribution) and have OPA installed through their tutorial, you should be able to follow along with me here. I would strongly caution you against testing this in a production cluster, but YOLO is the official motto of Kubernetes, so do whatever you want.
A note about OPA
There are many robust, high-level languages (Python, Go, shudder Javascript) that provide clear, easily readable ways to express logic. So naturally, OPA needs its own DSL. Creating your own DSL is going to be this generation’s version of “should I invent a new binary format for this project?”
Anyway, joking aside, Rego takes some getting used to (and I’m definitely not great at it yet). I would highly encourage you to read the docs, but you can still follow along with this article if you don’t have any experience (you just might get lost). Thankfully, OPA provides an amazing Rego Playground where you can test out policies and learn through experimentation. I will provide a link to a Rego playground with my policy examples at the end of this article.
Denying bad annotations
Jared’s article describes a scenario where compromised Kubernetes API credentials are leveraged to annotate resources in an environment with a payload annotation, which is then consumed by a malicious C2 agent. How could you use OPA to prevent this?
It’s fairly easy with the policy below. These OPA rules below will inspect a request to the Kubernetes API, extract the annotations, and check to see if one of them contains the key payload. If it does, the request is rejected.
package kubernetes.admission
# Credit to https://github.com/open-policy-agent/opa/issues/1263 for this function
# which just checks to see if there is an element in an array
contains(array, element) {
array[_] == element
}
# Obtains all of the keys in the annotations of a resource
input_annotation_keys[key] {
some key
input.request.object.metadata.annotations[key]
}
# Obtains all of the keys in the annotation of a resources created via a template
# For example: a pod template in a deployment
input_annotation_keys[key] {
some key
input.request.object.spec.template.metadata.annotations[key]
}
# Evaluates to true if the key is "payload"
explicitly_denied_annotation_key[key] {
some key
input_annotation_keys[key]
key == "payload"
}
OPA policies can be stored as a ConfigMap in the opa namespace (or in other namespaces with the openpolicyagent.org/policy=rego label). The OPA sidecar will automatically pick up and load the policy.You can confirm that the policy has been successfully loaded by ensuring that the openpolicyagent.org/policy-status annotation for the ConfigMap reports a status of OK:
# Create the policy ConfigMap from the above policy file
$ kubectl -n opa create configmap annotation-policy --from-file=policy.rego
# Confirm that the sidecar has loaded the policy
$ kubectl -n opa get configmap annotation-policy -o jsonpath="{.metadata.annotations}" | jq
{
"openpolicyagent.org/policy-status": "{\"status\":\"ok\"}"
}
With this policy in place, any annotations that match this explicit deny are rejected by OPA:
# Attempt to annotate a pod with a forbidden annotation
$ kubectl annotate po nginx payload=somebadstuff
Error from server (Annotation 'payload' was found in the explicit deny list): admission webhook "validating-webhook.openpolicyagent.org" denied the request: Annotation 'payload' was found in the explicit deny list
# Attempt to annotate a pod with an innocuous annotation
$ kubectl annotate po nginx legitimate_annotation="this is fine"
pod/nginx annotated
You can also view the full request body of kubectl annotate po nginx payload=somebadstuff here. As you can see, it’s just JSON and the OPA policies are simply rules based on fields in this JSON.
Let’s build on this explicit deny capability a bit. Imagine that your company recently stopped using a container vulnerability scanner called h2o because they realized that, like the overwhelming majority of container security companies, it doesn’t actually do anything. You want to ensure that this product is unable to annotate anything in your environment. The h2o scanner annotates pods with an h2o- annotation, which may contain an arbitrary string after the h2o- designation. You can use regex.match to block annotations that match a regular expression:
# Deny annotation keys based on a regex match of h2o-*
explicitly_denied_annotation_key[key] {
some key
input_annotation_keys[key]
regex.match("h2O*", key)
}
You can now recreate the ConfigMap and test it out.
# Update the ConfigMap
$ kubectl -n opa create configmap annotation-policy --from-file=policy.rego --dry-run=client -o yaml | k apply -f
# Confirm that annotations matching h2o-* can't be added
$ kubectl annotate po nginx h2o-vulnerabilities="lol the sales guy said this product works?" h2o-confidence-score="series-a-funded"
Error from server (Annotation 'h2o-confidence-score' was found in the explicit deny list, Annotation 'h2o-vulnerabilities' was found in the explicit deny list): admission webhook "validating-webhook.openpolicyagent.org" denied the request: Annotation 'h2o-confidence-score' was found in the explicit deny list, Annotation 'h2o-vulnerabilities' was found in the explicit deny list
Allowing explictly permitted annotations
The previous examples are great, but they’re also naive: A bad actor isn’t going to conveniently set an annotation called “payload” on your Kubernetes resources. A better idea might be to explicitly allow only certain annotations. While chatting with Jared about his article, he pointed out that annotations are used in a variety of unexpected ways: many additional, third-party add-ons such as ingress controllers, tend to slap annotations onto K8s resources and use those annotations for their own purposes.
Going back to the container scanner example: imagine that your CISO has just gotten suckered into paying for a brand new (TM) container scanning platform (they promise this one actually works). It annotates pods with three simple annotations: container-scanner-confidence, container-scanner-result, and container-scanner-vulnerabilties. Your software engineering teams also annotate their pods with a softeng-team and oncall-email annotation, so you want to permit those as well.
Expanding our policy to use the rules below will meet this purpose.
# Regex match for allowed annotations
allowed_annotation_key[key] {
some key
input_annotation_keys[key]
regex.match("container-scanner-*", key)
}
# Allow kubectl annotations
allowed_annotation_key[key] {
some key
input_annotation_keys[key]
regex.match("kubectl.kubernetes.io/*", key)
}
# Literal match for allowed annotations
allowed_annotation_key[key] {
some key
input_annotation_keys[key]
contains(["softeng-team", "oncall-email"], key)
}
# Deny if we have any keys that aren't explicitly allowed
deny[msg] {
some key
input_annotation_keys[key]
not contains(allowed_annotation_key, input_annotation_keys[key])
msg := sprintf("Annotation '%v' not found in explicitly allowed key list", [key])
}
After updating the ConfigMap using the previous process, you can confirm that only allowed annotations will result in a successful response from OPA.
$ kubectl annotate pod nginx some-random-annotation="hey, you can just put whatever you want in here, can't you?"
Error from server (Key 'some-random-annotation' not found in explicitly allowed key list): admission webhook "validating-webhook.openpolicyagent.org" denied the request: Key 'some-random-annotation' not found in explicitly allowed key list
$ kubectl annotate pod nginx softeng-team="AgileScrumbanTribeofGreatness"
pod/nginx annotated
Matching on value patterns
Fully enumerating all of the acceptable annotations in a Kubernetes environment sounds like a great idea (and a reasonable expectation of robust third-party software), but it still wouldn’t prevent a bad actor from injecting a malicious payload into one of those annotations. This is likely a riskier operation for an attacker, as they won’t necessarily know what a particular annotation does. Therefore, they risk breaking something and being noticed.
You can take the concept of an allowlist one step further: what if you have an allowlist of keys and their permitted value patterns? For example: your new container scanner that the CISO purchased will annotate pods with the container-scanner-result annotation. The vendor of the scanner says that this will only ever be “passed” or “failed.” You can construct a rule to ensure that only these values are allowed for this particular annotation:
# Get a list of key/value pairs
input_annotation_kv_pairs[kv_pair] {
some key
input_annotation_keys[key]
value := input.request.object.metadata.annotations[key]
kv_pair := {"key": key, "value": value}
}
# True if the annotation key is "container-scanner-result"
# AND the value is not passed or failed
denied_annotation_key_value[kv_pair] {
kv_pair := input_annotation_kv_pairs[_]
kv_pair.key == "container-scanner-result"
not regex.match(kv_pair.value, "passed|failed")
}
# Deny the request if any annotations did not have the expected key/value pair
deny[msg] {
kv_pair := denied_annotation_key_value[_]
msg := sprintf("The value '%v' is not allowed for key '%v'", [kv_pair.value, kv_pair.key])
}
You can now see that only allowed key/value patterns are permitted:
# Confirm that an illegitimate value isn't permitted for a particular annotation
$ kubectl annotate pod nginx container-scanner-result=skipped
Error from server (The value 'skipped' is not allowed for key 'container-scanner-result'): admission webhook "validating-webhook.openpolicyagent.org" denied the request: The value 'skipped' is not allowed for key 'container-scanner-result'
# Confirm that a legitimate annotation is allowed through
$ kubectl annotate pod nginx container-scanner-result=passed
pod/nginx annotated
There are many directions that you could go with this. You could write policies that only allow annotations:
- With values containing letters, numbers, and “-“s
- Of a certain length
- That match a certain pattern. For example: only permit email-formatted strings in annotations related to email addresses
- Etc.
You’re really only limited by your imagination. I think it would be awesome if third party vendors, such as ingress controller creators, provided a list of their allowed annotations and the permissible regular expressions for those values. That’s probably a stretch, but anyone could start writing some basic OPA rules in the meantime to ensure that the annotations under their control are not abused in unknown ways.
I’ve probably already said too much. There’s enough half-baked ideas here for some idiot to get a few million in seed funding for a “Kubernetes security startup,” so I won’t share all of my secrets.
Conclusion
My friend Jared proposed a really interesting covert channel vector by abusing Kubernetes annotations using compromised Kubernetes credentials. Kubernetes admission controllers provide the perfect spot to help mitigate an attack like this. While the ideas I’ve proposed in this article aren’t perfect (how many administrators have any idea what annotations are used in their environment?), they do form a starting point for adding security measures around this channel.
If you found the policy syntax hard to follow, then play around with my examples on The Rego Playground (I’ve already loaded a request from this article, along with the Rego policies, at the linked Playground). Just don’t get beaten up by the Hipster DSL gang.
For the full contents of the policies in this article, see the Github repo.
Previous article: Automated OS Qualification with Ansible
Next article: Signals and the "kubectl delete" command