Article reprinted from: https://www.kuboard.cn/learning/k8s-advanced/policy/lr.html
By default, when the container runs on the Kubernetes cluster, it is not limited by computing resources. Using the Resource quota, cluster administrators can limit the use of resources for namespaces. Within a namespace, the resource consumption of a Pod (or container) is unlimited. The concern at this point is that there may be a Pod (or container) that monopolizes most of the resources of the namespace. Limit Range is a Policy used to limit the amount of resources that a Pod (or container) in a namespace can consume.
The Kubernetes LimitRange object can:
- Limit the minimum and maximum computing resources for each Pod or container in the namespace
- Limit the minimum and maximum storage space that each PersistentVolumeClaim in the namespace can use
- Limit the ratio between the calculated resource request request and the qualified limit in the namespace
- Set the request/limit of the default computing resource in the namespace and inject it into the container automatically at run time
Enable Limit Range
Execute the command kubectl API resources to check whether your cluster supports Limit Range. The output results are as follows. Generally, Limit Range is enabled by default.
NAME SHORTNAMES APIGROUP NAMESPACED KIND bindings true Binding componentstatuses cs false ComponentStatus configmaps cm true ConfigMap endpoints ep true Endpoints events ev true Event limitranges limits true LimitRange namespaces ns false Namespace nodes no false Node persistentvolumeclaims pvc true PersistentVolumeClaim persistentvolumes pv false PersistentVolume pods po true Pod podtemplates true PodTemplate
Basic introduction
- The Cluster Administrator creates a LimitRange object in the namespace
- Users create objects such as workloads in the namespace, such as Pod, Container, PersistentVolumeClaim, and so on
- For those pods and containers that do not have a computational resource request request and limit set, the LimitRanger sets the default resource request and response according to the LimitRange object in the namespace, and ensures that the actual consumption of computational resources by pods and containers does not exceed the specified value
- If the request to create or update an object (Pod, Container, PersistentVolumeClaim) conflicts with the Limit Range, apiserver will return the HTTP status code 403 FORBIDDEN and the corresponding error message
- If limit range is activated in the namespace to limit the use of computing resources such as cpu and memory, the user must specify the request and limit of cpu or memory when creating Pod and Container, otherwise the system will refuse to create Pod
- Kubernetes only checks the qualification of LimitRange during Pod creation, and does not perform any checks when Pod is running
After the LimitRange object is activated, create a Pod, Container, and PersistentVolumeClaim. You must specify the request and limit of cpu or memory, otherwise the system will refuse to create a Pod
First create the Pod, Container and PersistentVolumeClaim, and then create the active LimitRange object. The Pod and Container for calculating the resource request request and limit are not set. The LimitRanger sets the default resource request and response according to the LimitRange object in the namespace
Examples of using LimitRange are:
- On a 2-node cluster with a total capacity of 8G memory and 16 core CPU, the Pod in a namespace is limited to use 100m CPU request and no more than 500m CPU limit, 200Mi memory request and no more than 600Mi memory online limit
- For containers without defined CPU and memory requests, specify that the default CPU request and limit are 150m and the default memory request is 300Mi
When the total limit of the namespace is less than the sum of the limits of Pod/Container in the namespace, resource contention will occur, and the container or Pod cannot be created.
When resource contention occurs or limit range is modified, the created Pod/Container will not be affected.
Limit the computing resources of the container
Suppose a Pod contains four containers, and each container defines spec.resource. At this time, the LimitRanger management controller handles the four containers in the Pod differently.
The demonstration steps are as follows:
1. Execute the following command to create the namespace limitrange demo
kubectl create namespace limitrange-demo
Switch the kubectl default namespace to limitrange demo
kubectl config set-context --current --namespace=limitrange-demo
2. The yaml file of the limitrange object is as follows:
apiVersion: v1
kind: LimitRange
metadata:
name: limit-mem-cpu-per-container
spec:
limits:
- max:
cpu: "800m"
memory: "1Gi"
min:
cpu: "100m"
memory: "99Mi"
default:
cpu: "700m"
memory: "900Mi"
defaultRequest:
cpu: "110m"
memory: "111Mi"
type: Container
This object defines the following for containers in the namespace:
- Maximum and minimum CPU / memory
- Default CPU / memory limit
- Default CPU / memory request
Execute the command to create the object:
kubectl create -f https://kuboard.cn/statics/learning/policy/lr-container-limit-range.yaml -n limitrange-demo
Execute commands to view results
kubectl describe limitrange/limit-mem-cpu-per-container -n limitrange-demo Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Container cpu 100m 800m 110m 700m - Container memory 99Mi 1Gi 111Mi 900Mi -
3. The yaml file of the Pod with 4 containers mentioned above is as follows:
apiVersion: v1
kind: Pod
metadata:
name: busybox1
spec:
containers:
- name: busybox-cnt01
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello from cnt01; sleep 10;done"]
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "200Mi"
cpu: "500m"
- name: busybox-cnt02
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello from cnt02; sleep 10;done"]
resources:
requests:
memory: "100Mi"
cpu: "100m"
- name: busybox-cnt03
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello from cnt03; sleep 10;done"]
resources:
limits:
memory: "200Mi"
cpu: "500m"
- name: busybox-cnt04
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello from cnt04; sleep 10;done"]
Execute the command to create the Pod
kubectl apply -f https://kuboard.cn/statics/learning/policy/lr-container-pod.yaml
The container contains valid CPU / memory requests/limits
Execute the following command to view the configuration information of busybox-cnt01
kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[0].resources"
{
"limits": {
"cpu": "500m",
"memory": "200Mi"
},
"requests": {
"cpu": "100m",
"memory": "100Mi"
}
}
- The container busybox-cnt01 in busybox Pod defines requests CPU = 100m and requests memory=100Mi
- The cpu limit (500m) of the 100m < = 500m < = 800m container is within the range specified by the namespace LimitRange
- 99mi < = 200Mi < = 1gi the memory limit (200Mi) of the container is within the range specified by the namespace LimitRange
- request/limit ratio is not specified for CPU / memory
- At this point, the container definition is valid and will be created
The container contains valid CPU / memory requests and limits are not specified
Execute the following command to view the configuration information of busybox-cnt02
kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[1].resources"
{
"limits": {
"cpu": "700m",
"memory": "900Mi"
},
"requests": {
"cpu": "100m",
"memory": "100Mi"
}
}
- The container busybox-cnt02 in busybox Pod defines requests CPU = 100m and requests Memory = 100mi, and is the maximum limit of the specified CPU / memory
- Since the container does not define limits, the namespace's LimitRange defines limits CPU = 700mi and limits Memory = 900mi is injected into the container
- The maximum CPU limit (700m) of 100m < = 700m < = 800m container is within the range specified by namespace LimitRange
- 99mi < = 900Mi < = 1gi the memory limit (900Mi) of the container is within the range specified by the namespace LimitRange
- request/limit ratio is not specified for CPU / memory
- At this point, the container definition is valid and will be created
The container contains valid CPU / memory limits and does not specify requests
Execute the following command to view the configuration information of busybox-cnt03
kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[2].resources"
{
"limits": {
"cpu": "500m",
"memory": "200Mi"
},
"requests": {
"cpu": "500m",
"memory": "200Mi"
}
}
- The container busybox-cnt03 in busybox Pod defines limits CPU = 500m and limits Memory = 200mi, and no CPU / memory requests are specified
- Since the container does not define requests, the defaultRequest defined by LimitRange in the namespace is not injected into the request field of the container. Instead, the limits defined by the container is set to its requests field: limits CPU = 500m and limits memory=200Mi
- The maximum cpu limit (500m) of the 100m < = 500m < = 800m container is within the range specified by the namespace LimitRange
- 99mi < = 200Mi < = 1gi the maximum memory limit (200Mi) of the container is within the range specified by the namespace LimitRange
- request/limit ratio is not specified for CPU / memory
- At this point, the container definition is valid and will be created
The container does not contain requests/limits for CPU / memory
Execute the following command to view the configuration information of busybox-cnt04
kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[3].resources"
{
"limits": {
"cpu": "700m",
"memory": "900Mi"
},
"requests": {
"cpu": "110m",
"memory": "111Mi"
}
}
- The container busybox-cnt04 in busybox Pod defines neither request nor limits
- Since the container does not define limits, the namespace's LimitRange defines limits CPU = 700mi and limits Memory = 900mi is injected into the container
- Since the container does not define requests, the LimitRange of the namespace defines requests CPU = 110m and requests Memory = 110mi is injected into the container
- The maximum cpu limit (700m) of the 100m < = 700m < = 800m container is within the range specified in the namespace LimitRange
- 99mi < = 900Mi < = 1gi the memory limit (900Mi) of the container is within the range specified by the namespace LimitRange
- request/limit ratio is not specified for CPU / memory
- At this point, the container definition is valid and will be created
All containers in the Pod busybox have passed the namespace LimitRange check, and this Pod will be created
Limit Pod's computing resources
The following is a LimitRange object used to limit the use of Pod resources.
apiVersion: v1
kind: LimitRange
metadata:
name: limit-mem-cpu-per-pod
spec:
limits:
- max:
cpu: "2"
memory: "2Gi"
type: Pod
Before starting, complete the calculation resources of the container, and ensure that the limitrange, limit MEM CPU per container and Pod busybox1 in this tutorial have been created.
1. Execute the following command to create the LimitRange in yaml above the limit MEM CPU Pod, which limits the CPU usage of each Pod to no more than 2 cores and memory to no more than 2Gi.
kubectl apply -f https://kuboard.cn/statics/learning/policy/lr-pod-limit-range.yaml -n limitrange-demo
Execute the command to view the creation result of limit MEM CPU per pod:
kubectl describe limitrange/limit-mem-cpu-per-pod -n limitrange-demo Name: limit-mem-cpu-per-pod Namespace: limitrange-demo Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Pod cpu - 2 - - - Pod memory - 2Gi - - -
2. Create a second Pod, yaml file as follows:
apiVersion: v1
kind: Pod
metadata:
name: busybox2
spec:
containers:
- name: busybox-cnt01
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello from cnt01; sleep 10;done"]
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "200Mi"
cpu: "500m"
- name: busybox-cnt02
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello from cnt02; sleep 10;done"]
resources:
requests:
memory: "100Mi"
cpu: "100m"
- name: busybox-cnt03
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello from cnt03; sleep 10;done"]
resources:
limits:
memory: "200Mi"
cpu: "500m"
- name: busybox-cnt04
image: busybox
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello from cnt04; sleep 10;done"]
Execute the following command to create the Pod
kubectl apply -f https://kuboard.cn/statics/learning/policy/lr-pod-pod.yaml -n limitrange-demo
The definition of Pod busybox2 is exactly the same as that of busybox1, but the following errors will be encountered when executing the create command, because the resources available to Pod are now limited:
Error from server (Forbidden): error when creating "limit-range-pod-2.yaml": pods "busybox2" is forbidden: [maximum cpu usage per Pod is 2, but limit is 2400m., maximum memory usage per Pod is 2Gi, but limit is 2306867200.]
Execute the command to view the resource usage of busybox1
kubectl get po/busybox1 -n limitrange-demo -o json | jq ".spec.containers[].resources.limits.memory" "200Mi" "900Mi" "200Mi" "900Mi"
Pod busybox2 will not be created in the cluster because the sum of the memory limits of all containers in it exceeds the limit in limitrange, limit MEM CPU per pod. busybox1 will not be expelled because the pod has been created when the limitrange limit MEM CPU per pod is created.
Limited storage resources
Through the LimitRange object, the cluster administrator can limit the minimum and maximum storage space that can be used by each PersistentVolumeClaim (storage volume declaration) in the namespace.
Please refer to the following example:
apiVersion: v1
kind: LimitRange
metadata:
name: storagelimits
spec:
limits:
- type: PersistentVolumeClaim
max:
storage: 2Gi
min:
storage: 1Gi
Execute the command to create the LimitRange:
kubectl create -f https://kuboard.cn/statics/learning/policy/lr-storage-limit.yaml -n limitrange-demo kubectl describe limits/storagelimits -n limitrange-demo Name: storagelimits Namespace: limitrange-demo Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- PersistentVolumeClaim storage 1Gi 2Gi - - -
Now suppose there is a PVC (storage volume declaration), and the definition file is as follows:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-limit-lower
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
Execute the command to create the PVC (storage volume declaration)
kubectl create -f https://kuboard.cn/statics/learning/policy/lr-storage-pvc-lower.yaml -n limitrange-demo
Due to the field requests defined in PVC Storage is greater than limits [0] in LimitRange storagelimits The definition of min.storage is small, so creating the PVC will fail:
Error from server (Forbidden): error when creating "lr-storage-pvc-lower.yaml": persistentvolumeclaims "pvc-limit-lower" is forbidden: minimum storage usage per PersistentVolumeClaim is 1Gi, but request is 500Mi.
If PVC requests Storage is greater than limits [0] in LimitRange Max.storage cannot be created successfully. Please refer to the following example:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-limit-greater
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
Execute the command to create the PVC (storage volume declaration)
kubectl create -f https://kuboard.cn/statics/learning/policy/lr-storage-pvc-greater.yaml Error from server (Forbidden): error when creating "lr-storage-pvc-greater.yaml": persistentvolumeclaims "pvc-limit-greater" is forbidden: maximum storage usage per PersistentVolumeClaim is 2Gi, but request is 5Gi.
Limit/Request ratio
If spec.limits.of the LimitRange object is specified Maxlimitrequestratio field, the request and limit of Pod / container in namespace cannot be 0, and the result of limit divided by request must be less than or equal to spec.limits of LimitRange maxLimitRequestRatio
In the following example, LimitRange defines that the maximum memory limit of any Pod in the namespace cannot exceed twice the minimum memory request:
apiVersion: v1
kind: LimitRange
metadata:
name: limit-memory-ratio-pod
spec:
limits:
- maxLimitRequestRatio:
memory: 2
type: Pod
Execute the command to create the LimitRange:
kubectl create -f https://kuboard.cn/statics/learning/policy/lr-ratio-limit-range.yaml -n limitrange-demo
Execute the command to view the creation results:
kubectl describe limitrange/limit-memory-ratio-pod -n limitrange-demo Name: limit-memory-ratio-pod Namespace: limitrange-demo Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Pod memory - - - - 2
At this point, if we create a Pod containing the following attribute requests Memory = 100mi and limits memory=300Mi:
apiVersion: v1
kind: Pod
metadata:
name: busybox3
spec:
containers:
- name: busybox-cnt01
image: busybox
resources:
limits:
memory: "300Mi"
requests:
memory: "100Mi"
Execute the command to create the Pod:
kubectl apply -f https://kuboard.cn/statics/learning/policy/lr-ratio-pod.yaml -n limitrange-demo
Since the memory limit request ratio of the Pod is 3, which exceeds 2 defined in the LimitRange, the Pod cannot be created successfully:
Error from server (Forbidden): error when creating "lr-ratio-pod.yaml": pods "busybox3" is forbidden: memory max limit to request ratio per Pod is 2, but provided ratio is 3.000000.