Kubernetes, the leading container orchestration platform, brings immense flexibility but also new security challenges. As adoption grows, robust security practices are no longer optional they’re critical. While it’s possible to rely on individual users to adopt good practices, this approach is often unrealistic. Security measures often add complexity to developers’ workflows, raising the barrier for project onboarding and delivery. From their perspective, ensuring the application functions correctly is the top priority, while security often takes a back seat. An effective solution is a tool that automatically verifies which objects are being applied to the cluster and how. In other words, Kubernetes administrators should be empowered to automatically control what is applied. One promising solution is Kyverno.
The Kubernetes community has long recognized the importance of security. Hence, attempts were made to implement tools to enhance the overall cluster security level. This led to the creation of PodSecurityPolicy (PSP).
Unfortunately, PSP didn’t gain widespread adoption, likely due to its complex and unintuitive configuration. It was removed in Kubernetes version 1.25 and replaced with a new mechanism PodSecurity. The new mechanism is significantly simpler. It serves as an excellent starting point for introducing basic security principles and helping users adapt. It defines three standards: privileged, baseline, and restricted, each of which can be enforced in one of three modes: enforce, audit, or warn.
If basic policy levels are sufficient, the native PodSecurity mechanism is worth using. However, as organizations grow, specific requirements may arise that call for custom policies. This is where Kyverno comes into play.
What is Kyverno?
Kyverno, a Kubernetes-native policy engine developed under CNCF, has earned significant recognition within the GitHub community for its simplicity and power. Kyverno allows you to:
- Create security policies.
- Validate objects against predefined policies and enforce compliance.
- Manage metadata.
- Generate reports.
- Test policies before deployment (using Kyverno CLI), with the option to integrate this step into CI/CD pipelines.
- Scan existing objects.
Additionally, all configurations are handled using YAML files, eliminating the need to learn a new language.
Admission Controller
Before diving into Kyverno’s capabilities, let’s explore how it works and what mechanism underpins its functionality. The key component here is the Admission Controller. Operating at the API server level, it determines whether resource requests should be accepted or rejected. This native Kubernetes mechanism requires no additional configuration or installation.
Among Kubernetes’ many controllers, Kyverno primarily leverages:
- ValidatingAdmissionWebhook - Validates requests for creating or updating Kubernetes resources, ensuring compliance with specified policies;
- MutatingAdmissionWebhook - Modifies requests for creating or updating resources, automatically adding configurations or tweaking settings.
Source: https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/
Practical Setup - Prerequisites
The demonstration will be conducted in a local environment. The following software needs to be installed, and for macOS users, the brew package manager will be utilized. Linux is also supported. Windows users can opt for WSL2, although some issues may arise.
Docker
Docker is used to install a lightweight Kubernetes cluster.
brew install docker
Kubectl
Kubectl will be needed to communicate with the Kubernetes API.
brew install kubernetes-cli
KiND
One of the simplest and fastest ways to create Kubernetes clusters is by using the KiND (Kubernetes in Docker) tool. Since we won’t be running a large number of applications on the cluster, the node requirements are not demanding. A minimum of 2 GB is suggested, although 4 GB is recommended.
brew install kind
Helm
Using the Helm package manager, we can easily install Kyverno and a set of Kyverno policies.
brew install helm
Creating a Kubernetes Cluster
KiND will be used to create the cluster. Without specifying parameters, it will install a Kubernetes cluster in the latest available version - at the time of writing this article, it was version v1.30. After a moment, the following message should appear:
kind create cluster --name kyverno-sourcemation
Creating cluster "kyverno-sourcemation" ...
✓ Ensuring node image (kindest/node:v1.30.0) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-kyverno-sourcemation"
You can now use your cluster with:
kubectl cluster-info --context kind-kyverno-sourcemation
Have a nice day! 👋
As you can see, the cluster is available and ready for the next steps.
kubectl cluster-info --context kind-kyverno-sourcemation
(...)
Kubernetes control plane is running at https://127.0.0.1:55291
CoreDNS is running at https://127.0.0.1:55291/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Installation of Kyverno
There are two ways to install Kyverno. For the purposes of this article, I will use the method with a Helm Chart. Kyverno will be installed in Standalone mode.
For production purposes, it is recommended to install Kyverno in High Availability mode.
Adding and updating the Helm repository:
helm repo add kyverno https://kyverno.github.io/kyverno/
"kyverno" has been added to your repositories
helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kyverno" chart repository
Update Complete. ⎈Happy Helming!⎈
The table below shows the compatibility between Kubernetes and Kyverno versions. It is important to keep this in mind before proceeding with the installation.
| Kyverno Version | Kubernetes Min | Kubernetes Max |
|---|---|---|
| 1.8.x | 1.23 | 1.25 |
| 1.9.x | 1.24 | 1.26 |
| 1.10.x | 1.24 | 1.26 |
| 1.11.x | 1.25 | 1.28 |
| 1.12.x | 1.26 | 1.29 |
Installation of Kyverno Helm Chart
If we run the default command, Helm will install Kyverno in the latest
available version. At the time of writing this article, it is version
v1.12.5. A dedicated namespace, kyverno-example, will also
be automatically created.
helm install kyverno kyverno/kyverno -n kyverno-example --create-namespace
NAME: kyverno
LAST DEPLOYED: Tue Jul 30 15:23:08 2024
NAMESPACE: kyverno-example
STATUS: deployed
REVISION: 1
NOTES:
Chart version: 3.2.6
Kyverno version: v1.12.5
Thank you for installing kyverno! Your release is named kyverno.
The following components have been installed in your cluster:
- CRDs
- Admission controller
- Reports controller
- Cleanup controller
- Background controller
With the kubectl command, we can check what has been installed.
kubectl get pod -o name -n kyverno-example
NAME
kyverno-admission-controller-7547cf7664-92zw5
kyverno-background-controller-86b9f95c96-wmwkr
kyverno-cleanup-controller-7f6f5447bf-md6f8
kyverno-reports-controller-665ccb5b65-r2969
The default Kyverno installation creates the following objects (Custom Resource Definitions):
kubectl get crd -o name | grep kyverno
admissionreports.kyverno.io
backgroundscanreports.kyverno.io
cleanuppolicies.kyverno.io
clusteradmissionreports.kyverno.io
clusterbackgroundscanreports.kyverno.io
clustercleanuppolicies.kyverno.io
clusterephemeralreports.reports.kyverno.io
clusterpolicies.kyverno.io
ephemeralreports.reports.kyverno.io
globalcontextentries.kyverno.io
policies.kyverno.io
policyexceptions.kyverno.io
updaterequests.kyverno.io
Kyverno CLI
The Kyverno CLI simplifies policy testing, allowing users to validate configurations before deploying them to Kubernetes.
brew install kyverno
Kyverno Policies
One of the major advantages of Kyverno is its rich collection of predefined security policies. The community can contribute to the development of new policies, which also positively impacts the growth of the project. The selection of policies can be tailored to specific environments, such as AWS, Openshift, or Istio. There are also sets of best practices, such as Best Practices and Software Supply Chain Security. Policies can be applied across the entire cluster or within a single namespace, offering flexibility to meet the organization’s requirements.
Before proceeding with the installation of the policy set, it is worth
checking what the values.yaml file from the
kyverno/kyverno-policies chart contains. We will use the helm inspect command and redirect the output to a local values.yaml
file.
helm inspect values kyverno/kyverno-policies > values.yaml
Since this file is quite long, I will only cover a portion of the available configuration options.
# -- Policy kind (`ClusterPolicy`, `Policy`)
# Set to `Policy` if you need namespaced policies and not cluster policies
policyKind: ClusterPolicy
# -- Pod Security Standard profile (`baseline`, `restricted`, `privileged`, `custom`).
# For more info https://kyverno.io/policies/pod-security.
podSecurityStandard: baseline
# -- Pod Security Standard (`low`, `medium`, `high`).
podSecuritySeverity: medium
# -- Policies to include when `podSecurityStandard` is `custom`.
podSecurityPolicies: []
# -- Additional policies to include from `other`.
includeOtherPolicies: []
# - require-non-root-groups
# -- Additional policies to include from `restricted`.
includeRestrictedPolicies: []
# - require-run-as-non-root-user
As seen from the commented lines in the code block above, there is quite a bit of flexibility in configuration. We can choose whether the policy applies to the entire cluster or only to a specific namespace (policyKind). It’s possible to select one of the security profiles (podSecurityStandard), as well as the severity level (podSecuritySeverity). Additional policies not included in the default profiles can also be added.
Take some time to explore the values.yaml file — it’s a great way
to uncover Kyverno’s full potential.
Policy Installation
Let’s install the default baseline policy package in Audit mode.
helm install kyverno-policies kyverno/kyverno-policies -n kyverno-example
NAME: kyverno-policies
LAST DEPLOYED: Wed Jul 31 09:58:34 2024
NAMESPACE: kyverno-example
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing kyverno-policies 3.2.5 😀
We have installed the “baseline” profile of Pod Security Standards and set them in Audit mode. Visit https://kyverno.io/policies/ to find more sample policies
The kubectl command returns the following policies. We are
displaying the new clusterpolicies object.
kubectl get clusterpolicies.kyverno.io
NAME ADMISSION BACKGROUND VALIDATE ACTION READY AGE MESSAGE
disallow-capabilities true true Audit True 73s Ready
disallow-host-namespaces true true Audit True 73s Ready
disallow-host-path true true Audit True 73s Ready
disallow-host-ports true true Audit True 73s Ready
disallow-host-process true true Audit True 73s Ready
disallow-privileged-containers true true Audit True 73s Ready
disallow-proc-mount true true Audit True 73s Ready
disallow-selinux true true Audit True 73s Ready
restrict-apparmor-profiles true true Audit True 73s Ready
restrict-seccomp true true Audit True 73s Ready
restrict-sysctls true true Audit True 73s Ready
Practical operation
We will focus on the practical operation of one of the installed policies. As an example, we will use the policy named disallow-privileged-containers. The structure of the yaml file can be checked with the following command:
kubectl get clusterpolicies.kyverno.io disallow-privileged-containers -o yaml |less
The disallow-privileged-containers policy blocks the execution of
containers with privileged mode enabled. In the code block below, I’ve
prepared a deployment that should violate the established rules. The
most important part here is the section
spec.template.spec.containers.securityContext.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-kyverno-testing
spec:
selector:
matchLabels:
app: nginx-kyverno
replicas: 1
template:
metadata:
labels:
app: nginx-kyverno
spec:
containers:
- name: nginx
securityContext:
privileged: true
image: nginx:latest
ports:
- containerPort: 80
Surprisingly, despite applying the above manifest, the nginx-kyverno container started successfully. Why? Kyverno should not have allowed its execution, as the deployment clearly violates the policy of running containers in privileged mode.
Remember, this policy was created in Audit mode. In practice, this means that Kyverno will display a warning to the user but allow the object to be created. This feature allows users to adapt gradually to the enforced changes.
kubectl describe deployments.apps nginx-kyverno-testing -n nginx-kyverno
.
.
.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning PolicyViolation 77s kyverno-admission policy disallow-privileged-containers/autogen-privileged-containers fail: validation error: Privileged mode is disallowed.
The fields spec.containers[*].securityContext.privileged and spec.initContainers[*].securityContext.privileged must be unset
or set to `false`. rule autogen-privileged-containers failed at path /spec/template/spec/containers/0/securityContext/privileged/
I will delete the nginx-kyverno-testing deployment and then recreate it. However, before doing so, I will change the disallow-privileged-containers policy to Enforce mode.
kubectl delete -f nginx.yaml -n nginx-kyverno
kubectl patch clusterpolicy disallow-privileged-containers -p '{"spec": {"validationFailureAction": "Enforce"}}' --type=merge
After the change, when attempting to recreate the deployment, Kyverno does not allow it. You immediately receive the following message:
kubectl apply -f nginx.yaml -n nginx-kyverno
Error from server: error when creating "nginx.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:
resource Deployment/nginx-kyverno/nginx-kyverno-testing was blocked due to the following policies
disallow-privileged-containers:
autogen-privileged-containers: 'validation error: Privileged mode is disallowed.
The fields spec.containers[*].securityContext.privileged and spec.initContainers[*].securityContext.privileged
must be unset or set to `false`. rule autogen-privileged-containers failed at
path /spec/template/spec/containers/0/securityContext/privileged/'
Removing the test Kubernetes cluster
To remove the containers creating the test cluster, we will use the following command:
kind delete cluster --name kyverno-sourcemation
Summary
This article introduces Kyverno. It includes instructions for setting up the environment and installation. A practical usage example is also shown. There are many more use cases, but the goal of this article was not to make it too long or hard to read. This article aimed to introduce Kyverno and inspire further exploration of its capabilities in securing Kubernetes clusters.