Article catalog
- Kubernetes cluster distributed storage plug-in Rook Ceph deployment
Kubernetes cluster distributed storage plug-in Rook Ceph deployment
1, Foreword
We often say that containers and pods are transient. It means that their life cycle may be very short and will be destroyed and created frequently. When the container is destroyed, the data saved in the file system inside the container will be cleared. In order to persistently save the data of the container, you can use the storage plug-in to mount a remote data volume based on the network or other mechanisms in the container, so that the files created in the container are actually saved on the remote storage server or on multiple nodes in a distributed manner, and have no binding relationship with the current host. In this way, the specified persistent storage volume can be requested to be mounted no matter on which node the new container is started.
Due to Kubernetes' loose coupling design, most storage projects, such as Ceph, GlusterFS, NFS, etc., can provide Kubernetes with persistent storage capacity. In this deployment practice, choose a very important production level storage plug-in project: Rook.
2, Rook introduction
1. Introduction
Rook project is a Kubernetes storage plug-in based on Ceph (support for more storage will also be added later). However, unlike the simple encapsulation of Ceph, rook has added a large number of enterprise level functions such as horizontal expansion, migration, disaster backup and monitoring to its implementation, making the project a highly scalable distributed storage solution, providing object, file and block storage.
Rook currently supports the establishment of Ceph, NFS, Minio Object Store, Edegefs, Cassandra and CockroachDB storage.
Rook mechanism:
- Rook provides a volume plug-in to extend the K8S storage system. The Kubelet agent Pod can be used to mount block devices and file systems managed by rook.
- Rook Operator is responsible for starting and monitoring the whole underlying storage system, such as Ceph Pod, Ceph OSD, etc. at the same time, it also manages CRD, object storage and file system.
- The Rook Agent agent is deployed on each node of K8S and runs in a Pod container. Each agent Pod is configured with a Flexvolume driver, which is mainly used to integrate with the volume control framework of K8S. The agent is responsible for relevant operations on each node, such as adding storage devices, mounting, formatting, deleting storage, etc.
For more information, please refer to the following official website:
- https://rook.io
- https://ceph.com/
2. Rook architecture
3, Rook deployment
1. Preliminary planning
2. Preparatory work
To configure Ceph storage clusters, you need at least one of the following local storage options:
- Raw device (non partitioned or formatted file system)
- Raw partition (unformatted file system)
- PV can be obtained from storage categories through block mode
You can use the following command to confirm that a partition or device is a formatted file system:
$ lsblk -f NAME FSTYPE LABEL UUID MOUNTPOINT vda ├─vda1 xfs e16ad84e-8cef-4cb1-b19b-9105c57f97b1 /boot ├─vda2 LVM2_member Vg3nyB-iW9Q-4xp0-LEIO-gzHc-2eax-D1razB │ └─centos-root xfs 0bb4bfa4-b315-43ca-a789-2b43e726c10c / ├─vda3 LVM2_member VZMibm-DJ8e-apig-YhR3-a1dF-wHYQ-8pjKan │ └─centos-root xfs 0bb4bfa4-b315-43ca-a789-2b43e726c10c / └─vda4
If the FSTYPE field is not empty, there is a file system at the top of the corresponding device. In this case, vda4 can be used for Ceph.
3. Get YAML
git clone --single-branch --branch master https://github.com/rook/rook.git
4. Deploy Rook Operator
This experiment uses k8s-node1, k8s-node2 and k8s-node3 nodes, so it needs to be modified as follows:
kubectl label nodes {k8s-node1,k8s-node2,k8s-node3} ceph-osd=enabled
kubectl label nodes {k8s-node1,k8s-node2,k8s-node3} ceph-mon=enabled
kubectl label nodes k8s-node1 ceph-mgr=enabled
Note: mgr in the current version of rook can only support one node.
Execute script:
cd rook/cluster/examples/kubernetes/ceph kubectl create -f common.yaml kubectl create -f operator.yaml
Note: the corresponding basic services (such as serviceaccounts) are created above, and the rook CEPH operator will create rook CEPH agent and rook discover at each node.
5. Deploy cluster
Configure cluster yaml
vi cluster.yaml
After modification, it is as follows:
#################################################################################################################
# Define the settings for the rook-ceph cluster with common settings for a production cluster.
# All nodes with available raw devices will be used for the Ceph cluster. At least three nodes are required
# in this example. See the documentation for more details on storage settings available.
# For example, to create the cluster:
# kubectl create -f common.yaml
# kubectl create -f operator.yaml
# kubectl create -f cluster.yaml
#################################################################################################################
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephVersion:
# The container image used to launch the Ceph daemon pods (mon, mgr, osd, mds, rgw).
# v13 is mimic, v14 is nautilus, and v15 is octopus.
# RECOMMENDATION: In production, use a specific version tag instead of the general v14 flag, which pulls the latest release and could result in different
# versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/.
# If you want to be more precise, you can always use a timestamp tag such ceph/ceph:v14.2.5-20190917
# This tag might not contain a new Ceph version, just security fixes from the underlying operating system, which will reduce vulnerabilities
image: ceph/ceph:v15.2.3
# Whether to allow unsupported versions of Ceph. Currently mimic and nautilus are supported, with the recommendation to upgrade to nautilus.
# Octopus is the version allowed when this is set to true.
# Do not set to true in production.
allowUnsupported: false
# The path on the host where configuration files will be persisted. Must be specified.
# Important: if you reinstall the cluster, make sure you delete this directory from each host or else the mons will fail to start on the new cluster.
# In Minikube, the '/data' directory is configured to persist across reboots. Use "/data/rook" in Minikube environment.
dataDirHostPath: /var/lib/rook
# Whether or not upgrade should continue even if a check fails
# This means Ceph's status could be degraded and we don't recommend upgrading but you might decide otherwise
# Use at your OWN risk
# To understand Rook's upgrade process of Ceph, read https://rook.io/docs/rook/master/ceph-upgrade.html#ceph-version-upgrades
skipUpgradeChecks: false
# Whether or not continue if PGs are not clean during an upgrade
continueUpgradeAfterChecksEvenIfNotHealthy: false
# set the amount of mons to be started
mon:
count: 3
allowMultiplePerNode: false
mgr:
modules:
# Several modules should not need to be included in this list. The "dashboard" and "monitoring" modules
# are already enabled by other settings in the cluster CR and the "rook" module is always enabled.
- name: pg_autoscaler
enabled: true
# enable the ceph dashboard for viewing cluster status
dashboard:
enabled: true
# serve the dashboard under a subpath (useful when you are accessing the dashboard via a reverse proxy)
# urlPrefix: /ceph-dashboard
# serve the dashboard at the given port.
# port: 8443
# serve the dashboard using SSL
ssl: true
# enable prometheus alerting for cluster
monitoring:
# requires Prometheus to be pre-installed
enabled: false
# namespace to deploy prometheusRule in. If empty, namespace of the cluster will be used.
# Recommended:
# If you have a single rook-ceph cluster, set the rulesNamespace to the same namespace as the cluster or keep it empty.
# If you have multiple rook-ceph clusters in the same k8s cluster, choose the same namespace (ideally, namespace with prometheus
# deployed) to set rulesNamespace for all the clusters. Otherwise, you will get duplicate alerts with multiple alert definitions.
rulesNamespace: rook-ceph
network:
# enable host networking
#provider: host
# EXPERIMENTAL: enable the Multus network provider
#provider: multus
#selectors:
# The selector keys are required to be `public` and `cluster`.
# Based on the configuration, the operator will do the following:
# 1. if only the `public` selector key is specified both public_network and cluster_network Ceph settings will listen on that interface
# 2. if both `public` and `cluster` selector keys are specified the first one will point to 'public_network' flag and the second one to 'cluster_network'
#
# In order to work, each selector value must match a NetworkAttachmentDefinition object in Multus
#
#public: public-conf --> NetworkAttachmentDefinition object name in Multus
#cluster: cluster-conf --> NetworkAttachmentDefinition object name in Multus
# enable the crash collector for ceph daemon crash collection
crashCollector:
disable: false
cleanupPolicy:
# cleanup should only be added to the cluster when the cluster is about to be deleted.
# After any field of the cleanup policy is set, Rook will stop configuring the cluster as if the cluster is about
# to be destroyed in order to prevent these settings from being deployed unintentionally.
# To signify that automatic deletion is desired, use the value "yes-really-destroy-data". Only this and an empty
# string are valid values for this field.
confirmation: ""
# To control where various services will be scheduled by kubernetes, use the placement configuration sections below.
# The example under 'all' would have all services scheduled on kubernetes nodes labeled with 'role=storage-node' and
# tolerate taints with a key of 'storage-node'.
placement:
mon:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: ceph-mon
operator: In
values:
- enabled
podAffinity:
podAntiAffinity:
topologySpreadConstraints:
tolerations:
- key: ceph-mon
operator: Exists
osd:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: ceph-osd
operator: In
values:
- enabled
podAffinity:
podAntiAffinity:
topologySpreadConstraints:
tolerations:
- key: ceph-osd
operator: Exists
mgr:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: ceph-mgr
operator: In
values:
- enabled
podAffinity:
podAntiAffinity:
topologySpreadConstraints:
tolerations:
- key: ceph-mgr
operator: Exists
# cleanup:
annotations:
# all:
# mon:
# osd:
# cleanup:
# If no mgr annotations are set, prometheus scrape annotations will be set by default.
# mgr:
resources:
# The requests and limits set here, allow the mgr pod to use half of one CPU core and 1 gigabyte of memory
# mgr:
# limits:
# cpu: "500m"
# memory: "1024Mi"
# requests:
# cpu: "500m"
# memory: "1024Mi"
# The above example requests/limits can also be added to the mon and osd components
# mon:
# osd:
# prepareosd:
# crashcollector:
# cleanup:
# The option to automatically remove OSDs that are out and are safe to destroy.
removeOSDsIfOutAndSafeToRemove: false
# priorityClassNames:
# all: rook-ceph-default-priority-class
# mon: rook-ceph-mon-priority-class
# osd: rook-ceph-osd-priority-class
# mgr: rook-ceph-mgr-priority-class
storage: # cluster level storage configuration and selection
useAllNodes: false #Turn off all nodes used
useAllDevices: false #Turn off all devices
deviceFilter: vda4
config:
# metadataDevice: "md0" # specify a non-rotational storage so ceph-volume will use it as block db device of bluestore.
# databaseSizeMB: "1024" # uncomment if the disks are smaller than 100 GB
# journalSizeMB: "1024" # uncomment if the disks are 20 GB or smaller
# osdsPerDevice: "1" # this value can be overridden at the node or device level
# encryptedDevice: "true" # the default value for this option is "false"
# Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named
# nodes below will be used as storage resources. Each node's 'name' field should match their 'kubernetes.io/hostname' label.
nodes:
- name: "k8s-node1" #Specify storage node host
devices:
- name: "vda4" #Specify the disk as sdb
config:
storeType: bluestore
- name: "k8s-node2"
devices:
- name: "vda4"
config:
storeType: bluestore
- name: "k8s-node3"
devices:
- name: "vda4"
config:
storeType: bluestore
# The section for configuring management of daemon disruptions during upgrade or fencing.
disruptionManagement:
# If true, the operator will create and manage PodDisruptionBudgets for OSD, Mon, RGW, and MDS daemons. OSD PDBs are managed dynamically
# via the strategy outlined in the [design](https://github.com/rook/rook/blob/master/design/ceph/ceph-managed-disruptionbudgets.md). The operator will
# block eviction of OSDs by default and unblock them safely when drains are detected.
managePodBudgets: false
# A duration in minutes that determines how long an entire failureDomain like `region/zone/host` will be held in `noout` (in addition to the
# default DOWN/OUT interval) when it is draining. This is only relevant when `managePodBudgets` is `true`. The default value is `30` minutes.
osdMaintenanceTimeout: 30
# If true, the operator will create and manage MachineDisruptionBudgets to ensure OSDs are only fenced when the cluster is healthy.
# Only available on OpenShift.
manageMachineDisruptionBudgets: false
# Namespace in which to watch for the MachineDisruptionBudgets.
machineDisruptionBudgetNamespace: openshift-machine-api
More CRD configuration references for cluster:
- https://github.com/rook/rook/blob/master/Documentation/ceph-cluster-crd.md
- https://blog.gmem.cc/rook-based-k8s-storage-solution
Execute cluster yaml
kubectl create -f cluster.yaml # View deployment log $ kubectl logs -f -n rook-ceph rook-ceph-operator-567d7945d6-t9rd4 # After waiting for a certain time, some intermediate containers may fluctuate [7d@k8s-master ceph]$ kubectl get pods -n rook-ceph -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES csi-cephfsplugin-dr4dq 3/3 Running 0 24h 172.16.106.239 k8s-node2 <none> <none> csi-cephfsplugin-provisioner-6bcb7cdd75-dtzmn 5/5 Running 0 24h 10.244.2.51 k8s-node3 <none> <none> csi-cephfsplugin-provisioner-6bcb7cdd75-zbfsc 5/5 Running 0 24h 10.244.1.43 k8s-node1 <none> <none> csi-cephfsplugin-vsfst 3/3 Running 0 24h 172.16.106.205 k8s-node3 <none> <none> csi-cephfsplugin-vtcmj 3/3 Running 0 24h 172.16.106.209 k8s-node1 <none> <none> csi-rbdplugin-2vwrm 3/3 Running 0 24h 172.16.106.205 k8s-node3 <none> <none> csi-rbdplugin-4slnj 3/3 Running 0 24h 172.16.106.239 k8s-node2 <none> <none> csi-rbdplugin-flrcp 3/3 Running 0 24h 172.16.106.209 k8s-node1 <none> <none> csi-rbdplugin-provisioner-5bcf44d764-4fzxh 6/6 Running 0 24h 10.244.1.42 k8s-node1 <none> <none> csi-rbdplugin-provisioner-5bcf44d764-dl8tn 6/6 Running 0 24h 10.244.3.27 k8s-node2 <none> <none> rook-ceph-crashcollector-k8s-node1-5bd88b5f5-qcpdp 1/1 Running 0 24h 10.244.1.47 k8s-node1 <none> <none> rook-ceph-crashcollector-k8s-node2-98c4cc686-z8jsl 1/1 Running 0 24h 10.244.3.34 k8s-node2 <none> <none> rook-ceph-crashcollector-k8s-node3-66fc75bc8f-cck7l 1/1 Running 0 24h 10.244.2.55 k8s-node3 <none> <none> rook-ceph-mgr-a-674699969f-x929t 1/1 Running 0 24h 10.244.3.31 k8s-node2 <none> <none> rook-ceph-mon-a-76856b5d7c-z69sv 1/1 Running 0 24h 10.244.3.29 k8s-node2 <none> <none> rook-ceph-mon-b-85c9bb4fdb-2s5mz 1/1 Running 0 24h 10.244.2.53 k8s-node3 <none> <none> rook-ceph-mon-c-5f8b654578-mmvgk 1/1 Running 0 24h 10.244.1.45 k8s-node1 <none> <none> rook-ceph-operator-567d7945d6-t9rd4 1/1 Running 0 24h 10.244.2.49 k8s-node3 <none> <none> rook-ceph-osd-0-6c8d48b89-jzxtk 1/1 Running 0 24h 10.244.3.35 k8s-node2 <none> <none> rook-ceph-osd-1-6df8796948-xgb7q 1/1 Running 0 24h 10.244.1.48 k8s-node1 <none> <none> rook-ceph-osd-2-5f47685c7f-k8dnr 1/1 Running 0 24h 10.244.2.56 k8s-node3 <none> <none> rook-ceph-osd-prepare-k8s-node1-lgvjl 0/1 Completed 0 24h 10.244.1.46 k8s-node1 <none> <none> rook-ceph-osd-prepare-k8s-node2-fghpx 0/1 Completed 0 24h 10.244.3.33 k8s-node2 <none> <none> rook-ceph-osd-prepare-k8s-node3-hp6xg 0/1 Completed 0 24h 10.244.2.54 k8s-node3 <none> <none> rook-discover-2sql9 1/1 Running 0 24h 10.244.3.25 k8s-node2 <none> <none> rook-discover-fcmvx 1/1 Running 0 24h 10.244.2.50 k8s-node3 <none> <none> rook-discover-r7zpr 1/1 Running 0 24h 10.244.1.40 k8s-node1 <none> <none>
Note: if deployment fails
master node execution
kubectl delete -f ./
All node nodes perform the following cleanup operations:
rm -rf /var/lib/rook /dev/mapper/ceph-* dmsetup ls dmsetup remove_all dd if=/dev/zero of=/dev/vda4 bs=512k count=1 wipefs -af /dev/vda4
6. Deploy Toolbox
Toolbox is a tool set container of Rook. The commands in this container can be used to debug and test Rook. The operation of Ceph temporary test is generally executed in this container.
# Start the rook CEPH tools Pod: $ kubectl create -f toolbox.yaml deployment.apps/rook-ceph-tools created # Wait for rook CEPH tools to load its container and enter the running state: $ kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" NAME READY STATUS RESTARTS AGE rook-ceph-tools-6d659f5579-knt6x 1/1 Running 0 7s
7. Test Rook
$ kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
# View Ceph status
[root@rook-ceph-tools-6d659f5579-knt6x /]# ceph status
cluster:
id: 550e2978-26a6-4f3b-b101-10369ab63cf4
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 2m)
mgr: a(active, since 85s)
osd: 3 osds: 3 up (since 114s), 3 in (since 114s)
data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 3.0 GiB used, 147 GiB / 150 GiB avail
pgs: 1 active+clean
[root@rook-ceph-tools-6d659f5579-knt6x /]# ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 k8s-node2 1026M 48.9G 0 0 0 0 exists,up
1 k8s-node1 1026M 48.9G 0 0 0 0 exists,up
2 k8s-node3 1026M 48.9G 0 0 0 0 exists,up
[root@rook-ceph-tools-6d659f5579-knt6x /]# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 150 GiB 147 GiB 6.2 MiB 3.0 GiB 2.00
TOTAL 150 GiB 147 GiB 6.2 MiB 3.0 GiB 2.00
--- POOLS ---
POOL ID STORED OBJECTS USED %USED MAX AVAIL
device_health_metrics 1 0 B 0 0 B 0 46 GiB
[root@rook-ceph-tools-6d659f5579-knt6x /]# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
device_health_metrics 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B
total_objects 0
total_used 3.0 GiB
total_avail 147 GiB
total_space 150 GiB
# View all Keyrings of Ceph
[root@rook-ceph-tools-6d659f5579-knt6x /]# ceph auth ls
installed auth entries:
osd.0
key: AQAjofFe1j9pGhAABnjTXAYZeZdwo2FGHIFv+g==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.1
key: AQAjofFeY0LaHhAAMVLxrH1lqXqyYsZE9yJ5dg==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.2
key: AQAkofFeBjtoDBAAVYW7FursqpbttekW54u2rA==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQDhoPFeqLE4ORAAvBeGwV7p1YY25owP8nS02Q==
caps: [mds] allow *
caps: [mgr] allow *
caps: [mon] allow *
caps: [osd] allow *
client.bootstrap-mds
key: AQABofFeQclcCxAAEtA9Y4+yF3I6H9RM0/f1DQ==
caps: [mon] allow profile bootstrap-mds
client.bootstrap-mgr
key: AQABofFeaeBcCxAA9SEnt+RV7neC4uy/xQb5qg==
caps: [mon] allow profile bootstrap-mgr
client.bootstrap-osd
key: AQABofFe0/NcCxAAqCKwJpzPlav8MuajRk8xmw==
caps: [mon] allow profile bootstrap-osd
client.bootstrap-rbd
key: AQABofFesAZdCxAAZyWJg+Pa3F0g5Toy4LamPw==
caps: [mon] allow profile bootstrap-rbd
client.bootstrap-rbd-mirror
key: AQABofFejRpdCxAA/9NbTQDJILdSoYJZdol7bQ==
caps: [mon] allow profile bootstrap-rbd-mirror
client.bootstrap-rgw
key: AQABofFeAi5dCxAAKu67ZyM8PRRPcluTXR3YRw==
caps: [mon] allow profile bootstrap-rgw
client.crash
key: AQAcofFeLDr3KBAAw9UowFd26JiQSGjCFyhx8w==
caps: [mgr] allow profile crash
caps: [mon] allow profile crash
client.csi-cephfs-node
key: AQAcofFeFRh4DBAA7Z8kgcHGM92vHj6cvGbXXg==
caps: [mds] allow rw
caps: [mgr] allow rw
caps: [mon] allow r
caps: [osd] allow rw tag cephfs *=*
client.csi-cephfs-provisioner
key: AQAbofFemLuJMRAA4WlGWBjONb1av48rox1q6g==
caps: [mgr] allow rw
caps: [mon] allow r
caps: [osd] allow rw tag cephfs metadata=*
client.csi-rbd-node
key: AQAbofFepu7rFhAA+vdit2ipDgVFc/yKUpHHug==
caps: [mon] profile rbd
caps: [osd] profile rbd
client.csi-rbd-provisioner
key: AQAaofFe3Yw9OxAAiJzZ6HQne/e9Zob5G311OA==
caps: [mgr] allow rw
caps: [mon] profile rbd
caps: [osd] profile rbd
mgr.a
key: AQAdofFeh4VZHhAA8VL9gH5jgOxzjTDtEaFWBQ==
caps: [mds] allow *
caps: [mon] allow profile mgr
caps: [osd] allow *
[root@rook-ceph-tools-6d659f5579-knt6x /]# ceph version
ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable)
[root@rook-ceph-tools-6d659f5579-knt6x /]# exit
exit
In this way, a Rook based persistent storage cluster runs as a container, Next, all pods created on the Kubernetes project can mount the data volumes provided by Ceph in the container through Persistent Volume (PV) and Persistent Volume Claim (PVC). The Rook project will be responsible for the life cycle management, disaster backup and other operation and maintenance of these data volumes.
4, Setting up dashboard
dashboard is a very useful tool, which allows you to roughly understand the status of Ceph cluster, including overall health status, single arbitration status, mgr, osd and other Ceph daemons, view pool and PG status, display logs for daemons, and so on. Rook makes it easy to enable dashboards.
1. Deploy Node SVC
Modify dashboard external HTTPS yaml
$ vi dashboard-external-https.yaml apiVersion: v1 kind: Service metadata: name: rook-ceph-mgr-dashboard-external-https namespace: rook-ceph labels: app: rook-ceph-mgr rook_cluster: rook-ceph spec: ports: - name: dashboard port: 8443 protocol: TCP targetPort: 8443 selector: app: rook-ceph-mgr rook_cluster: rook-ceph sessionAffinity: None type: NodePort
Create Node SVC
$ kubectl create -f dashboard-external-https.yaml service/rook-ceph-mgr-dashboard-external-https created $ kubectl get svc -n rook-ceph NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE csi-cephfsplugin-metrics ClusterIP 10.102.32.77 <none> 8080/TCP,8081/TCP 17m csi-rbdplugin-metrics ClusterIP 10.101.121.5 <none> 8080/TCP,8081/TCP 17m rook-ceph-mgr ClusterIP 10.99.155.138 <none> 9283/TCP 16m rook-ceph-mgr-dashboard ClusterIP 10.97.61.135 <none> 8443/TCP 16m rook-ceph-mgr-dashboard-external-https NodePort 10.108.210.25 <none> 8443:32364/TCP 6s rook-ceph-mon-a ClusterIP 10.102.116.81 <none> 6789/TCP,3300/TCP 16m rook-ceph-mon-b ClusterIP 10.101.141.241 <none> 6789/TCP,3300/TCP 16m rook-ceph-mon-c ClusterIP 10.101.157.247 <none> 6789/TCP,3300/TCP 16m
Rook operator will enable the CEPH Mgr dashboard module. A service object will be created to expose the port in the Kubernetes cluster. Rook will enable port 8443 for https access.
5, Confirmation verification
Login to dashboard requires secure access. Rook creates a default user admin in the namespace running Rook Ceph cluster and generates a secret Rook Ceph dashboard admin password called.
To retrieve the generated password, run the following command:
kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
6, Ceph block storage application
1. Create StorageClass
Before providing block storage, you need to create StorageClass and storage pool. K8S needs these two kinds of resources to interact with Rook and allocate persistent volumes (PV).
cd rook/cluster/examples/kubernetes/ceph/csi/rbd kubectl create -f csi/rbd/storageclass.yaml
Interpretation: a storage pool named replicapool and a storageClass of rook CEPH block will be created in the following configuration files.
apiVersion: ceph.rook.io/v1 kind: CephBlockPool metadata: name: replicapool namespace: rook-ceph spec: failureDomain: host replicated: size: 3 # Disallow setting pool with replica 1, this could lead to data loss without recovery. # Make sure you're *ABSOLUTELY CERTAIN* that is what you want requireSafeReplicaSize: true # gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool # for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size #targetSizeRatio: .5 --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rook-ceph-block provisioner: rook-ceph.rbd.csi.ceph.com parameters: # clusterID is the namespace where the rook cluster is running # If you change this namespace, also change the namespace below where the secret namespaces are defined clusterID: rook-ceph # If you want to use erasure coded pool with RBD, you need to create # two pools. one erasure coded and one replicated. # You need to specify the replicated pool here in the `pool` parameter, it is # used for the metadata of the images. # The erasure coded pool must be set as the `dataPool` parameter below. #dataPool: ec-data-pool pool: replicapool # RBD image format. Defaults to "2". imageFormat: "2" # RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature. imageFeatures: layering # The secrets contain Ceph admin credentials. These are generated automatically by the operator # in the same namespace as the cluster. csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph # Specify the filesystem type of the volume. If not specified, csi-provisioner # will set default as `ext4`. csi.storage.k8s.io/fstype: ext4 # uncomment the following to use rbd-nbd as mounter on supported nodes # **IMPORTANT**: If you are using rbd-nbd as the mounter, during upgrade you will be hit a ceph-csi # issue that causes the mount to be disconnected. You will need to follow special upgrade steps # to restart your application pods. Therefore, this option is not recommended. #mounter: rbd-nbd allowVolumeExpansion: true reclaimPolicy: Delete
$ kubectl get storageclasses.storage.k8s.io NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE rook-ceph-block rook-ceph.rbd.csi.ceph.com Delete Immediate true 44m
2. Create PVC
$ kubectl create -f pvc.yaml $ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE rbd-pvc Bound pvc-b2b7ce1d-7cad-4b7b-afac-dcf6cd597e88 1Gi RWO rook-ceph-block 45m $ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-b2b7ce1d-7cad-4b7b-afac-dcf6cd597e88 1Gi RWO Delete Bound default/rbd-pvc rook-ceph-block 45m
Interpretation: create the corresponding PVC as above. storageClassName: is a rook Ceph block based on the rook Ceph cluster.
pvc.yaml:
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: rbd-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: rook-ceph-block
3. Consumer block device
$ kubectl create -f rookpod01.yaml $ kubectl get pods NAME READY STATUS RESTARTS AGE rookpod01 0/1 Completed 0 4m1s
Interpretation: create the above Pod and mount the PVC created before, waiting for the execution to be completed
rookpod01.yaml:
apiVersion: v1 kind: Pod metadata: name: rookpod01 spec: restartPolicy: OnFailure containers: - name: test-container image: busybox volumeMounts: - name: block-pvc mountPath: /var/test command: ['sh', '-c', 'echo "Hello World" > /var/test/data; exit 0'] volumes: - name: block-pvc persistentVolumeClaim: claimName: rbd-pvc readOnly: false
4. Test persistence
# Delete rookpod01 $ kubectl delete -f rookpod01.yaml pod "rookpod01" deleted # Create rookpod02 $ kubectl create -f rookpod02.yaml pod/rookpod02 created $ kubectl get pods NAME READY STATUS RESTARTS AGE rookpod02 0/1 Completed 0 59s $ kubectl logs rookpod02 Hello World
Interpretation: create a rookpod02 and use the created PVC to test the persistence.
rookpod02.yaml:
apiVersion: v1 kind: Pod metadata: name: rookpod02 spec: restartPolicy: OnFailure containers: - name: test-container image: busybox volumeMounts: - name: block-pvc mountPath: /var/test command: ['sh', '-c', 'cat /var/test/data; exit 0'] volumes: - name: block-pvc persistentVolumeClaim: claimName: rbd-pvc readOnly: false
7, Problems encountered
1. dashboard click overview 500 Internal Server Error
terms of settlement:
Create a new copy of the built-in administrator role, remove iscsi permissions from the role, and then assign the new role to the administrator. After the problem is solved upstream, you can delete the new role and reassign the administrator role to the admin user.
ceph dashboard ac-role-create admin-no-iscsi
for scope in dashboard-settings log rgw prometheus grafana nfs-ganesha manager hosts rbd-image config-opt rbd-mirroring cephfs user osd pool monitor; do
ceph dashboard ac-role-add-scope-perms admin-no-iscsi ${scope} create delete read update;
done
ceph dashboard ac-user-set-roles admin admin-no-iscsi
[root@rook-ceph-tools-6d659f5579-knt6x /]# ceph dashboard ac-role-create admin-no-iscsi
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {}}
[root@rook-ceph-tools-6d659f5579-knt6x /]# for scope in dashboard-settings log rgw prometheus grafana nfs-ganesha manager hosts rbd-image config-opt rbd-mirroring cephfs user osd pool monitor; do
> ceph dashboard ac-role-add-scope-perms admin-no-iscsi ${scope} create delete read update;
> done;
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"], "rgw": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"], "rgw": ["create", "delete", "read", "update"], "prometheus": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"], "rgw": ["create", "delete", "read", "update"], "prometheus": ["create", "delete", "read", "update"], "grafana": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"], "rgw": ["create", "delete", "read", "update"], "prometheus": ["create", "delete", "read", "update"], "grafana": ["create", "delete", "read", "update"], "nfs-ganesha": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"], "rgw": ["create", "delete", "read", "update"], "prometheus": ["create", "delete", "read", "update"], "grafana": ["create", "delete", "read", "update"], "nfs-ganesha": ["create", "delete", "read", "update"], "manager": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"], "rgw": ["create", "delete", "read", "update"], "prometheus": ["create", "delete", "read", "update"], "grafana": ["create", "delete", "read", "update"], "nfs-ganesha": ["create", "delete", "read", "update"], "manager": ["create", "delete", "read", "update"], "hosts": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"], "rgw": ["create", "delete", "read", "update"], "prometheus": ["create", "delete", "read", "update"], "grafana": ["create", "delete", "read", "update"], "nfs-ganesha": ["create", "delete", "read", "update"], "manager": ["create", "delete", "read", "update"], "hosts": ["create", "delete", "read", "update"], "rbd-image": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"], "rgw": ["create", "delete", "read", "update"], "prometheus": ["create", "delete", "read", "update"], "grafana": ["create", "delete", "read", "update"], "nfs-ganesha": ["create", "delete", "read", "update"], "manager": ["create", "delete", "read", "update"], "hosts": ["create", "delete", "read", "update"], "rbd-image": ["create", "delete", "read", "update"], "config-opt": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"], "rgw": ["create", "delete", "read", "update"], "prometheus": ["create", "delete", "read", "update"], "grafana": ["create", "delete", "read", "update"], "nfs-ganesha": ["create", "delete", "read", "update"], "manager": ["create", "delete", "read", "update"], "hosts": ["create", "delete", "read", "update"], "rbd-image": ["create", "delete", "read", "update"], "config-opt": ["create", "delete", "read", "update"], "rbd-mirroring": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"], "rgw": ["create", "delete", "read", "update"], "prometheus": ["create", "delete", "read", "update"], "grafana": ["create", "delete", "read", "update"], "nfs-ganesha": ["create", "delete", "read", "update"], "manager": ["create", "delete", "read", "update"], "hosts": ["create", "delete", "read", "update"], "rbd-image": ["create", "delete", "read", "update"], "config-opt": ["create", "delete", "read", "update"], "rbd-mirroring": ["create", "delete", "read", "update"], "cephfs": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"], "rgw": ["create", "delete", "read", "update"], "prometheus": ["create", "delete", "read", "update"], "grafana": ["create", "delete", "read", "update"], "nfs-ganesha": ["create", "delete", "read", "update"], "manager": ["create", "delete", "read", "update"], "hosts": ["create", "delete", "read", "update"], "rbd-image": ["create", "delete", "read", "update"], "config-opt": ["create", "delete", "read", "update"], "rbd-mirroring": ["create", "delete", "read", "update"], "cephfs": ["create", "delete", "read", "update"], "user": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"], "rgw": ["create", "delete", "read", "update"], "prometheus": ["create", "delete", "read", "update"], "grafana": ["create", "delete", "read", "update"], "nfs-ganesha": ["create", "delete", "read", "update"], "manager": ["create", "delete", "read", "update"], "hosts": ["create", "delete", "read", "update"], "rbd-image": ["create", "delete", "read", "update"], "config-opt": ["create", "delete", "read", "update"], "rbd-mirroring": ["create", "delete", "read", "update"], "cephfs": ["create", "delete", "read", "update"], "user": ["create", "delete", "read", "update"], "osd": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"], "rgw": ["create", "delete", "read", "update"], "prometheus": ["create", "delete", "read", "update"], "grafana": ["create", "delete", "read", "update"], "nfs-ganesha": ["create", "delete", "read", "update"], "manager": ["create", "delete", "read", "update"], "hosts": ["create", "delete", "read", "update"], "rbd-image": ["create", "delete", "read", "update"], "config-opt": ["create", "delete", "read", "update"], "rbd-mirroring": ["create", "delete", "read", "update"], "cephfs": ["create", "delete", "read", "update"], "user": ["create", "delete", "read", "update"], "osd": ["create", "delete", "read", "update"], "pool": ["create", "delete", "read", "update"]}}
{"name": "admin-no-iscsi", "description": null, "scopes_permissions": {"dashboard-settings": ["create", "delete", "read", "update"], "log": ["create", "delete", "read", "update"], "rgw": ["create", "delete", "read", "update"], "prometheus": ["create", "delete", "read", "update"], "grafana": ["create", "delete", "read", "update"], "nfs-ganesha": ["create", "delete", "read", "update"], "manager": ["create", "delete", "read", "update"], "hosts": ["create", "delete", "read", "update"], "rbd-image": ["create", "delete", "read", "update"], "config-opt": ["create", "delete", "read", "update"], "rbd-mirroring": ["create", "delete", "read", "update"], "cephfs": ["create", "delete", "read", "update"], "user": ["create", "delete", "read", "update"], "osd": ["create", "delete", "read", "update"], "pool": ["create", "delete", "read", "update"], "monitor": ["create", "delete", "read", "update"]}}
[root@rook-ceph-tools-6d659f5579-knt6x /]# ceph dashboard ac-user-set-roles admin admin-no-iscsi
{"username": "admin", "password": "$2b$12$UZUaLaTtHOrB5rUZvgqt7OHKEFIa0F76RlKaosnUtY.dDAGvGod6C", "roles": ["admin-no-iscsi"], "name": null, "email": null, "lastUpdate": 1592897567, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": false}
Source address:
- https://github.com/zuozewei/blog-example/tree/master/Kubernetes/k8s-rook-ceph