前文我们聊到了k8s上的ingress资源相关话题，回顾请参考：https://www.cnblogs.com/qiuhom-1874/p/14167581.html；今天们来聊一下k8s上volume相关话题；
在说k8s上的volume的使用前，我们先来回顾下docker里的volume；对于docker容器来说，镜像是分成构建的且每一层都是只读的，只读就意味着不能修改数据；只有当一个镜像运行为容器以后，在镜像的最顶端才会加上一个可写层，一旦这个容器被删除，对应可写层上的数据也随之被删除；为了解决docker容器上的数据持久化的问题；docker使用了volume；在docker上volume有两种管理方式，第一种是用户手动指定把宿主机（对于宿主机上的目录可能是挂载存储系统上的某目录）上的某个目录挂载到容器某个目录，这种管理方式叫做绑定挂载卷；还有一种就是docker自身维护把某个目录挂载到容器某个目录，这种叫docker自己管理的卷；不管使用那种方式管理的volume，它都是容器直接关联宿主机上的某个目录或文件；docker中的volume解决了容器生命周期内产生的数据在容器终止后能够持久化保存的问题；同样k8s也有同样的烦恼，不同的是k8s面对的是pod；我们知道pod是k8s上最小调度单元，一个pod被删除以后，pod里运行的容器也随之被删除，那么pod里容器产生的数据该如何被持久化呢？要想解决这个问题，我们先来看看pod的组成；
提示：在k8s上pod里可以运行一个或多个容器，运行多个容器，其中一个容器我们叫主容器，其他的容器是用来辅助主容器，我们叫做sidecar；对于pod来说，不管里面运行多少个容器，在最底层都会运行一个pause容器，该容器最主要用来为pod提供基础架构支撑；并且位于同一个pod中的容器都共享pause容器的网络名称空间以及IPC和UTS；这样一来我们要想给pod里的容器提供存储卷，首先要把存储卷关联到pause容器，然后在容器里挂载pause里的存储卷即可；如下图所示
提示：如上图所示，对于pause容器来说它可以关联存储A，也可以关联存储B；对于pause关联某个存储，其位于同一pod中的其他容器就也可以挂载pause里关联的存储目录或文件；对于k8s来说存储本来就不属于k8s内部组件，它是一个外来系统，这也意味着我们要想k8s使用外部存储系统，首先pause容器要有适配其对应存储系统的驱动；我们知道同一宿主机上运行的多个容器都是共享同一内核，即宿主机内核上有某个存储系统的驱动，那么pause就可以使用对应的驱动去适配对应的存储；
volumes类型
我们知道要想在k8s上使用存储卷，我们需要在对应的节点上提供对应存储系统的驱动，对应运行在该节点上的所有pod就可以使用对应的存储系统，那么问题来了，pod怎么使用对应的存储系统呢？该怎么向其驱动程序传递参数呢？我们知道在k8s上一切皆对象，要在k8s上使用存储卷，我们还需要把对应的驱动抽象成k8s上的资源；在使用时，我们直接初始化对应的资源为对象即可；为了在k8s上简化使用存储卷的复杂度，k8s内置了一些存储接口，对于不同类型的存储，其使用的接口、传递的参数也有所不同；除此之外在k8s上也支持用户使用自定义存储，通过csi接口来定义；
查看k8s上支持的volumes接口
[root@master01 ~]# kubectl explain pod.spec.volumes
KIND: Pod
VERSION: v1
RESOURCE: volumes <[]Object>
DESCRIPTION:
List of volumes that can be mounted by containers belonging to the pod.
More info: https://kubernetes.io/docs/concepts/storage/volumes
Volume represents a named volume in a pod that may be accessed by any
container in the pod.
FIELDS:
awsElasticBlockStore <Object>
AWSElasticBlockStore represents an AWS Disk resource that is attached to a
kubelet's host machine and then exposed to the pod. More info:
https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
azureDisk <Object>
AzureDisk represents an Azure Data Disk mount on the host and bind mount to
the pod.
azureFile <Object>
AzureFile represents an Azure File Service mount on the host and bind mount
to the pod.
cephfs <Object>
CephFS represents a Ceph FS mount on the host that shares a pod's lifetime
cinder <Object>
Cinder represents a cinder volume attached and mounted on kubelets host
machine. More info: https://examples.k8s.io/MysqL-cinder-pd/README.md
configMap <Object>
ConfigMap represents a configMap that should populate this volume
csi <Object>
CSI (Container Storage Interface) represents ephemeral storage that is
handled by certain external CSI drivers (Beta feature).
downwardAPI <Object>
DownwardAPI represents downward API about the pod that should populate this
volume
emptyDir <Object>
EmptyDir represents a temporary directory that shares a pod's lifetime.
More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir
ephemeral <Object>
Ephemeral represents a volume that is handled by a cluster storage driver
(Alpha feature). The volume's lifecycle is tied to the pod that defines it
- it will be created before the pod starts,and deleted when the pod is
removed.
Use this if: a) the volume is only needed while the pod runs,b) features
of normal volumes like restoring from snapshot or capacity tracking are
needed,c) the storage driver is specified through a storage class,and d)
the storage driver supports dynamic volume provisioning through a
PersistentVolumeClaim (see EphemeralVolumeSource for more information on
the connection between this volume type and PersistentVolumeClaim).
Use PersistentVolumeClaim or one of the vendor-specific APIs for volumes
that persist for longer than the lifecycle of an individual pod.
Use CSI for light-weight local ephemeral volumes if the CSI driver is meant
to be used that way - see the documentation of the driver for more
information.
A pod can use both types of ephemeral volumes and persistent volumes at the
same time.
fc <Object>
FC represents a Fibre Channel resource that is attached to a kubelet's host
machine and then exposed to the pod.
flexVolume <Object>
FlexVolume represents a generic volume resource that is
provisioned/attached using an exec based plugin.
flocker <Object>
Flocker represents a Flocker volume attached to a kubelet's host machine.
This depends on the Flocker control service being running
gcePersistentDisk <Object>
GCEPersistentDisk represents a GCE Disk resource that is attached to a
kubelet's host machine and then exposed to the pod. More info:
https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
gitRepo <Object>
GitRepo represents a git repository at a particular revision. DEPRECATED:
GitRepo is deprecated. To provision a container with a git repo,mount an
EmptyDir into an InitContainer that clones the repo using git,then mount
the EmptyDir into the Pod's container.
glusterfs <Object>
Glusterfs represents a Glusterfs mount on the host that shares a pod's
lifetime. More info: https://examples.k8s.io/volumes/glusterfs/README.md
hostPath <Object>
HostPath represents a pre-existing file or directory on the host machine
that is directly exposed to the container. This is generally used for
system agents or other privileged things that are allowed to see the host
machine. Most containers will NOT need this. More info:
https://kubernetes.io/docs/concepts/storage/volumes#hostpath
iscsi <Object>
ISCSI represents an ISCSI Disk resource that is attached to a kubelet's
host machine and then exposed to the pod. More info:
https://examples.k8s.io/volumes/iscsi/README.md
name <string> -required-
Volume's name. Must be a DNS_LABEL and unique within the pod. More info:
https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
nfs <Object>
NFS represents an NFS mount on the host that shares a pod's lifetime More
info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
persistentVolumeClaim <Object>
PersistentVolumeClaimVolumeSource represents a reference to a
PersistentVolumeClaim in the same namespace. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
photonPersistentDisk <Object>
PhotonPersistentDisk represents a PhotonController persistent disk attached
and mounted on kubelets host machine
portworxVolume <Object>
PortworxVolume represents a portworx volume attached and mounted on
kubelets host machine
projected <Object>
Items for all in one resources secrets,configmaps,and downward API
quobyte <Object>
Quobyte represents a Quobyte mount on the host that shares a pod's lifetime
rbd <Object>
RBD represents a Rados Block Device mount on the host that shares a pod's
lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md
scaleIO <Object>
ScaleIO represents a ScaleIO persistent volume attached and mounted on
Kubernetes nodes.
secret <Object>
Secret represents a secret that should populate this volume. More info:
https://kubernetes.io/docs/concepts/storage/volumes#secret
storageos <Object>
StorageOS represents a StorageOS volume attached and mounted on Kubernetes
nodes.
vsphereVolume <Object>
VsphereVolume represents a vSphere volume attached and mounted on kubelets
host machine
[root@master01 ~]#
提示：从上面的帮助信息可以看到k8s上支持的存储接口还是很多，每一个存储接口都是一种类型；对于这些存储类型我们大致可以分为云存储，分布式存储，网络存储、临时存储，节点本地存储，特殊类型存储、用户自定义存储等等；比如awsElasticBlockStore、azureDisk、azureFile、gcePersistentDisk、vshperVolume、cinder这些类型划分为云存储；cephfs、glusterfs、rbd这些划分为分布式存储；nfs、iscsi、fc这些划分为网络存储；enptyDIR划分为临时存储；hostPath、local划分为本地存储；自定义存储csi；特殊存储configMap、secret、downwardAPId；持久卷申请persistentVolumeClaim等等；
volumes的使用
示例：创建使用hostPath类型存储卷Pod
[root@master01 ~]# cat hostPath-demo.yaml apiVersion: v1 kind: Pod Metadata: name: vol-hostpath-demo namespace: default spec: containers: - name: Nginx image: Nginx:1.14-alpine volumeMounts: - name: webhtml mountPath: /usr/share/Nginx/html readOnly: true volumes: - name: webhtml hostPath: path: /vol/html/ type: DirectoryOrCreate [root@master01 ~]#
提示：以上配置清单表示创建一个名为Nginx的pod，对应使用Nginx:1.14-alpine的镜像；并且定义了一个存储卷，该存储卷的名称为webhtml，其类型为hostPath；在定义存储卷时，我们需要在spec字段下使用volumes字段来指定，该字段的值为一个对象列表；其中name是必须字段，用于指定存储卷的名称，方便容器挂载其存储卷时引用的标识符；其次我们需要对应的存储卷类型来表示使用对应的存储接口；hostPath表示使用hostPath类型存储接口；该类型存储接口需要我们手动传递两个参数，第一个是path指定宿主机的某个目录或文件路径；type用来指定当宿主机上的指定的路径不存在时该怎么办，这个值有7个值；其中DirectoryOrCteate表示对应path字段指定的文件必须是一个目录，当这个目录在宿主机上不存在时就创建；Directory表示对应path字段指定的文件必须是一个已存在的目录；FileOrCreate表示对应path字段指定的文件必须是一个文件，如果文件不存在就创建；File表示对应path字段必须为一个已存在的文件；Socket表示对应path必须为一个已存在的Socket文件；CharDevice表示对应path字段指定的文件必须是一个已存在的字符设备；BlockDevice表示对应path字段指定的是一个已存在的块设备；定义volumes相当于把外部存储关联到对应pod的pause容器，至于pod里的其他容器是否要使用，怎么使用，取决于volumeMounts字段是否定义；spec.containers.volumeMounts字段用于指定对应pod里的容器存储卷挂载配置，其中name和mountPath是必选字段，name字段用于指定引用的存储卷名称；mountPath字段用于指定在容器内部的挂载点，readOnly用于指定是否为只读，默认是读写，即readOnly的值为false；
应用配置清单
[root@master01 ~]# kubectl apply -f hostPath-demo.yaml pod/vol-hostpath-demo created [root@master01 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE myapp-6479b786f5-9d4mh 1/1 Running 1 47h myapp-6479b786f5-k252c 1/1 Running 1 47h vol-hostpath-demo 1/1 Running 0 11s [root@master01 ~]# kubectl describe pod/vol-hostpath-demo Name: vol-hostpath-demo Namespace: default Priority: 0 Node: node03.k8s.org/192.168.0.46 Start Time: Wed,23 Dec 2020 23:14:35 +0800 Labels: <none> Annotations: <none> Status: Running IP: 10.244.3.92 IPs: IP: 10.244.3.92 Containers: Nginx: Container ID: docker://eb8666714b8697457ce2a88271a4615f836873b4729b6a0938776e3d527c6536 Image: Nginx:1.14-alpine Image ID: docker-pullable://Nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 Port: <none> Host Port: <none> State: Running Started: Wed,23 Dec 2020 23:14:37 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /usr/share/Nginx/html from webhtml (ro) /var/run/secrets/kubernetes.io/serviceaccount from default-token-xvd4c (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhtml: Type: HostPath (bare host directory volume) Path: /vol/html/ HostPathType: DirectoryOrCreate default-token-xvd4c: Type: Secret (a volume populated by a Secret) SecretName: default-token-xvd4c Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 43s default-scheduler Successfully assigned default/vol-hostpath-demo to node03.k8s.org Normal Pulled 42s kubelet Container image "Nginx:1.14-alpine" already present on machine Normal Created 41s kubelet Created container Nginx Normal Started 41s kubelet Started container Nginx [root@master01 ~]#
提示：可以看到对应pod里以只读方式挂载了webhtml存储卷，对应webhtm存储卷类型为HostPath，对应path是/vol/html/；
查看对应pod所在节点
[root@master01 ~]# kubectl get pod vol-hostpath-demo -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES vol-hostpath-demo 1/1 Running 0 3m39s 10.244.3.92 node03.k8s.org <none> <none> [root@master01 ~]#
在node03上查看对应目录是否创建？
[root@node03 ~]# ll / total 16 lrwxrwxrwx. 1 root root 7 Sep 15 20:33 bin -> usr/bin dr-xr-xr-x. 5 root root 4096 Sep 15 20:39 boot drwxr-xr-x 20 root root 3180 Dec 23 23:10 dev drwxr-xr-x. 80 root root 8192 Dec 23 23:10 etc drwxr-xr-x. 2 root root 6 Nov 5 2016 home lrwxrwxrwx. 1 root root 7 Sep 15 20:33 lib -> usr/lib lrwxrwxrwx. 1 root root 9 Sep 15 20:33 lib64 -> usr/lib64 drwxr-xr-x. 2 root root 6 Nov 5 2016 media drwxr-xr-x. 2 root root 6 Nov 5 2016 mnt drwxr-xr-x. 4 root root 35 Dec 8 14:25 opt dr-xr-xr-x 141 root root 0 Dec 23 23:09 proc dr-xr-x---. 4 root root 213 Dec 21 22:46 root drwxr-xr-x 26 root root 780 Dec 23 23:13 run lrwxrwxrwx. 1 root root 8 Sep 15 20:33 sbin -> usr/sbin drwxr-xr-x. 2 root root 6 Nov 5 2016 srv dr-xr-xr-x 13 root root 0 Dec 23 23:09 sys drwxrwxrwt. 9 root root 251 Dec 23 23:11 tmp drwxr-xr-x. 13 root root 155 Sep 15 20:33 usr drwxr-xr-x. 19 root root 267 Sep 15 20:38 var drwxr-xr-x 3 root root 18 Dec 23 23:14 vol [root@node03 ~]# ll /vol total 0 drwxr-xr-x 2 root root 6 Dec 23 23:14 html [root@node03 ~]# ll /vol/html/ total 0 [root@node03 ~]#
提示：可以看到对应节点上已经创建/vol/html/目录，对应目录下没有任何文件；
在对应节点对应目录下创建一个网页文件，访问对应pod看看是否对应网页文件是否能够被访问到？
[root@node03 ~]# echo "this is test page from node03 /vol/html/test.html" > /vol/html/test.html [root@node03 ~]# cat /vol/html/test.html this is test page from node03 /vol/html/test.html [root@node03 ~]# exit logout Connection to node03 closed. [root@master01 ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myapp-6479b786f5-9d4mh 1/1 Running 1 47h 10.244.2.99 node02.k8s.org <none> <none> myapp-6479b786f5-k252c 1/1 Running 1 47h 10.244.4.21 node04.k8s.org <none> <none> vol-hostpath-demo 1/1 Running 0 7m45s 10.244.3.92 node03.k8s.org <none> <none> [root@master01 ~]# curl 10.244.3.92/test.html this is test page from node03 /vol/html/test.html [root@master01 ~]#
提示：可以看到在对应节点上创建网页文件，访问pod能够正常被访问到；
[root@master01 ~]# kubectl delete -f hostPath-demo.yaml pod "vol-hostpath-demo" deleted [root@master01 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE myapp-6479b786f5-9d4mh 1/1 Running 1 47h myapp-6479b786f5-k252c 1/1 Running 1 47h [root@master01 ~]# ssh node03 Last login: Wed Dec 23 23:18:51 2020 from master01 [root@node03 ~]# ll /vol/html/ total 4 -rw-r--r-- 1 root root 50 Dec 23 23:22 test.html [root@node03 ~]# exit logout Connection to node03 closed. [root@master01 ~]#
提示：可以看到删除了pod以后，在对应节点上的目录并不会被删除；对应的网页文件还是完好无损；
测试：重新引用配置清单，访问对应的pod，看看是否能够访问到对应的网页文件内容？
[root@master01 ~]# kubectl apply -f hostPath-demo.yaml pod/vol-hostpath-demo created [root@master01 ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myapp-6479b786f5-9d4mh 1/1 Running 1 47h 10.244.2.99 node02.k8s.org <none> <none> myapp-6479b786f5-k252c 1/1 Running 1 47h 10.244.4.21 node04.k8s.org <none> <none> vol-hostpath-demo 1/1 Running 0 7s 10.244.3.93 node03.k8s.org <none> <none> [root@master01 ~]# curl 10.244.3.93/test.html this is test page from node03 /vol/html/test.html [root@master01 ~]#
提示：可以看到对应pod被调度到node03上了，访问对应的pod能够访问到我们创建的网页文件；假如我们明确指定将此pod运行在node02上，对应pod是否还可以访问到对应的网页文件呢？
测试：绑定pod运行在node02.k8s.org上
[root@master01 ~]# cat hostPath-demo.yaml apiVersion: v1 kind: Pod Metadata: name: vol-hostpath-demo namespace: default spec: nodeName: node02.k8s.org containers: - name: Nginx image: Nginx:1.14-alpine volumeMounts: - name: webhtml mountPath: /usr/share/Nginx/html readOnly: true volumes: - name: webhtml hostPath: path: /vol/html/ type: DirectoryOrCreate [root@master01 ~]#
提示：绑定pod运行为某个节点上，我们可以在spec字段中用nodeName字段来指定对应节点的主机名即可；
删除原有pod，重新应用新资源清单
[root@master01 ~]# kubectl delete pod/vol-hostpath-demo pod "vol-hostpath-demo" deleted [root@master01 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE myapp-6479b786f5-9d4mh 1/1 Running 1 47h myapp-6479b786f5-k252c 1/1 Running 1 47h [root@master01 ~]# kubectl apply -f hostPath-demo.yaml pod/vol-hostpath-demo created [root@master01 ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myapp-6479b786f5-9d4mh 1/1 Running 1 47h 10.244.2.99 node02.k8s.org <none> <none> myapp-6479b786f5-k252c 1/1 Running 1 47h 10.244.4.21 node04.k8s.org <none> <none> vol-hostpath-demo 1/1 Running 0 8s 10.244.2.100 node02.k8s.org <none> <none> [root@master01 ~]#
提示：可以看到重新应用新资源清单，对应pod运行在node02上；
访问对应pod，看看test.html是否能够被访问到？
[root@master01 ~]# curl 10.244.2.100/test.html <html> <head><title>404 Not Found</title></head> <body bgcolor="white"> <center><h1>404 Not Found</h1></center> <hr><center>Nginx/1.14.2</center> </body> </html> [root@master01 ~]#
提示：可以看到现在访问pod，对应网页文件就不能被访问到；其实原因很简单；hostPath类型存储卷是将宿主机上的某个目录或文件当作存储卷映射进pause容器，然后供pod里的容器挂载使用；这种类型的存储卷不能跨节点；所以在node02上创建的pod，node03上的文件肯定是不能被访问到的；为此，如果要使用hostPath类型的存储卷，我们就必须绑定节点；除此之外我们就应该在k8s节点上创建相同的文件或目录；
示例：创建使用emptyDir类型存储卷pod
[root@master01 ~]# cat emptyDir-demo.yaml apiVersion: v1 kind: Pod Metadata: name: vol-emptydir-demo namespace: default spec: containers: - name: Nginx image: Nginx:1.14-alpine volumeMounts: - name: web-cache-dir mountPath: /usr/share/Nginx/html readOnly: true readOnly: true - name: alpine image: alpine volumeMounts: - name: web-cache-dir mountPath: /Nginx/html command: ["/bin/sh","-c"] args: - while true; do echo $(hostname) $(date) >> /Nginx/html/index.html; sleep 10; done volumes: - name: web-cache-dir emptyDir: medium: Memory sizeLimit: "10Mi" [root@master01 ~]#
提示：以上清单表示定义运行一个名为vol-emptydir-demo的pod；在其pod内部运行两个容器，一个名为Nginx，一个名为alpine；同时这两个容器都同时挂载一个名为web-cache-dir的存储卷，其类型为emptyDir，如下图所示；定义empytDir类型的存储卷，我们需要在spec.volumes字段下使用name指定其存储卷的名称；用emptyDir指定其存储卷类型为emptyDir；对于empytDir类型存储卷，它有两个属性，medium字段用于指定媒介类型，Memory表示使用内存作为存储媒介；默认该字段的值为“”，表示使用默认的对应节点默认的存储媒介；sizeLimit字段是用来限制其对应存储大小，默认是空，表示不限制；
提示：如上图，其pod内部有两个容器，一个名为alpine的容器，它会每隔10往/Nginx/html/inde.html文件中写入对应主机名+时间；而Nginx容器挂载对应的empytDir类型存储卷到本地的网页存储目录；简单讲就是alpine容器往/Nginx/html/index.html写数据，Nginx容器挂载对应文件到网页目录；
应用资源清单
[root@master01 ~]# kubectl apply -f emptyDir-demo.yaml pod/vol-emptydir-demo created [root@master01 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myapp-6479b786f5-9d4mh 1/1 Running 1 2d 10.244.2.99 node02.k8s.org <none> <none> myapp-6479b786f5-k252c 1/1 Running 1 2d 10.244.4.21 node04.k8s.org <none> <none> vol-emptydir-demo 0/2 ContainerCreating 0 8s <none> node03.k8s.org <none> <none> vol-hostpath-demo 1/1 Running 0 72m 10.244.2.100 node02.k8s.org <none> <none> [root@master01 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myapp-6479b786f5-9d4mh 1/1 Running 1 2d 10.244.2.99 node02.k8s.org <none> <none> myapp-6479b786f5-k252c 1/1 Running 1 2d 10.244.4.21 node04.k8s.org <none> <none> vol-emptydir-demo 2/2 Running 0 16s 10.244.3.94 node03.k8s.org <none> <none> vol-hostpath-demo 1/1 Running 0 72m 10.244.2.100 node02.k8s.org <none> <none> [root@master01 ~]# kubectl describe pod vol-emptydir-demo Name: vol-emptydir-demo Namespace: default Priority: 0 Node: node03.k8s.org/192.168.0.46 Start Time: Thu,24 Dec 2020 00:46:56 +0800 Labels: <none> Annotations: <none> Status: Running IP: 10.244.3.94 IPs: IP: 10.244.3.94 Containers: Nginx: Container ID: docker://58af9ef80800fb22543d1c80be58849f45f3d62f3b44101dbca024e0761cead5 Image: Nginx:1.14-alpine Image ID: docker-pullable://Nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 Port: <none> Host Port: <none> State: Running Started: Thu,24 Dec 2020 00:46:57 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /usr/share/Nginx/html from web-cache-dir (ro) /var/run/secrets/kubernetes.io/serviceaccount from default-token-xvd4c (ro) alpine: Container ID: docker://327f110a10e8ef9edb5f86b5cb3dad53e824010b52b1c2a71d5dbecab6f49f05 Image: alpine Image ID: docker-pullable://alpine@sha256:3c7497bf0c7af93428242d6176e8f7905f2201d8fc5861f45be7a346b5f23436 Port: <none> Host Port: <none> Command: /bin/sh -c Args: while true; do echo $(hostname) $(date) >> /Nginx/html/index.html; sleep 10; done State: Running Started: Thu,24 Dec 2020 00:47:07 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /Nginx/html from web-cache-dir (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-xvd4c (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: web-cache-dir: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory SizeLimit: 10Mi default-token-xvd4c: Type: Secret (a volume populated by a Secret) SecretName: default-token-xvd4c Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 51s default-scheduler Successfully assigned default/vol-emptydir-demo to node03.k8s.org Normal Pulled 51s kubelet Container image "Nginx:1.14-alpine" already present on machine Normal Created 51s kubelet Created container Nginx Normal Started 50s kubelet Started container Nginx Normal Pulling 50s kubelet Pulling image "alpine" Normal Pulled 40s kubelet Successfully pulled image "alpine" in 10.163157508s Normal Created 40s kubelet Created container alpine Normal Started 40s kubelet Started container alpine [root@master01 ~]#
提示：可以看到对应pod已经正常运行起来，其内部有2个容器；其中Nginx容器一只读方式挂载名为web-cache-dir的存储卷，alpine以读写方式挂载web-cache-dir的存储卷；对应存储卷类型为emptyDir；
访问对应pod，看看是否能够访问到对应存储卷中index.html的内容？
[root@master01 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myapp-6479b786f5-9d4mh 1/1 Running 1 2d 10.244.2.99 node02.k8s.org <none> <none> myapp-6479b786f5-k252c 1/1 Running 1 2d 10.244.4.21 node04.k8s.org <none> <none> vol-emptydir-demo 2/2 Running 0 4m38s 10.244.3.94 node03.k8s.org <none> <none> vol-hostpath-demo 1/1 Running 0 77m 10.244.2.100 node02.k8s.org <none> <none> [root@master01 ~]# curl 10.244.3.94 vol-emptydir-demo Wed Dec 23 16:47:07 UTC 2020 vol-emptydir-demo Wed Dec 23 16:47:17 UTC 2020 vol-emptydir-demo Wed Dec 23 16:47:27 UTC 2020 vol-emptydir-demo Wed Dec 23 16:47:37 UTC 2020 vol-emptydir-demo Wed Dec 23 16:47:47 UTC 2020 vol-emptydir-demo Wed Dec 23 16:47:57 UTC 2020 vol-emptydir-demo Wed Dec 23 16:48:07 UTC 2020 vol-emptydir-demo Wed Dec 23 16:48:17 UTC 2020 vol-emptydir-demo Wed Dec 23 16:48:27 UTC 2020 vol-emptydir-demo Wed Dec 23 16:48:37 UTC 2020 vol-emptydir-demo Wed Dec 23 16:48:47 UTC 2020 vol-emptydir-demo Wed Dec 23 16:48:57 UTC 2020 vol-emptydir-demo Wed Dec 23 16:49:07 UTC 2020 vol-emptydir-demo Wed Dec 23 16:49:17 UTC 2020 vol-emptydir-demo Wed Dec 23 16:49:27 UTC 2020 vol-emptydir-demo Wed Dec 23 16:49:37 UTC 2020 vol-emptydir-demo Wed Dec 23 16:49:47 UTC 2020 vol-emptydir-demo Wed Dec 23 16:49:57 UTC 2020 vol-emptydir-demo Wed Dec 23 16:50:07 UTC 2020 vol-emptydir-demo Wed Dec 23 16:50:17 UTC 2020 vol-emptydir-demo Wed Dec 23 16:50:27 UTC 2020 vol-emptydir-demo Wed Dec 23 16:50:37 UTC 2020 vol-emptydir-demo Wed Dec 23 16:50:47 UTC 2020 vol-emptydir-demo Wed Dec 23 16:50:57 UTC 2020 vol-emptydir-demo Wed Dec 23 16:51:07 UTC 2020 vol-emptydir-demo Wed Dec 23 16:51:17 UTC 2020 vol-emptydir-demo Wed Dec 23 16:51:27 UTC 2020 vol-emptydir-demo Wed Dec 23 16:51:37 UTC 2020 vol-emptydir-demo Wed Dec 23 16:51:47 UTC 2020 [root@master01 ~]#
提示：可以看到能够访问到index.html文件内容，并且该文件内容是alpine容器动态生成的内容；从上面的示例，不难理解，在同一个pod内部可以共享同一存储卷；
示例：创建使用nfs类型的存储卷pod
[root@master01 ~]# cat nfs-demo.yaml apiVersion: v1 kind: Pod Metadata: name: vol-nfs-demo namespace: default spec: containers: - name: Nginx image: Nginx:1.14-alpine volumeMounts: - name: webhtml mountPath: /usr/share/Nginx/html readOnly: true volumes: - name: webhtml nfs: path: /data/html/ server: 192.168.0.99 [root@master01 ~]#
提示：定义nfs类型存储卷，对应spec.volumes.nfs字段下必须定义path字段，该字段用于指定其nfs文件系统的导出文件路径；server字段是用于指定其nfs服务器地址；在使用nfs存储作为pod的后端存储，首先我们要准备好nfs服务器，并导出对应的目录；
准备nfs服务器，在192.168.0.99这台服务器上安装nfs-utils包
[root@docker_registry ~]# ip a|grep 192.168.0.99
inet 192.168.0.99/24 brd 192.168.0.255 scope global enp3s0
[root@docker_registry ~]# yum install nfs-utils -y
Loaded plugins: fastestmirror,langpacks
Repository epel is listed more than once in the configuration
Repository epel-debuginfo is listed more than once in the configuration
Repository epel-source is listed more than once in the configuration
base | 3.6 kB 00:00:00
docker-ce-stable | 3.5 kB 00:00:00
epel | 4.7 kB 00:00:00
extras | 2.9 kB 00:00:00
kubernetes/signature | 844 B 00:00:00
kubernetes/signature | 1.4 kB 00:00:00 !!!
mariadb-main | 2.9 kB 00:00:00
mariadb-maxscale | 2.4 kB 00:00:00
mariadb-tools | 2.9 kB 00:00:00
mongodb-org | 2.5 kB 00:00:00
proxysql_repo | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
(1/6): docker-ce-stable/x86_64/primary_db | 51 kB 00:00:00
(2/6): kubernetes/primary | 83 kB 00:00:01
(3/6): mongodb-org/primary_db | 26 kB 00:00:01
(4/6): epel/x86_64/updateinfo | 1.0 MB 00:00:02
(5/6): updates/7/x86_64/primary_db | 4.7 MB 00:00:01
(6/6): epel/x86_64/primary_db | 6.9 MB 00:00:02
Determining fastest mirrors
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
kubernetes 612/612
Resolving Dependencies
--> Running transaction check
---> Package nfs-utils.x86_64 1:1.3.0-0.66.el7_8 will be updated
---> Package nfs-utils.x86_64 1:1.3.0-0.68.el7 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
=============================================================================================================================================
Package Arch Version Repository Size
=============================================================================================================================================
Updating:
nfs-utils x86_64 1:1.3.0-0.68.el7 base 412 k
Transaction Summary
=============================================================================================================================================
Upgrade 1 Package
Total download size: 412 k
Downloading packages:
No Presto Metadata available for base
nfs-utils-1.3.0-0.68.el7.x86_64.rpm | 412 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : 1:nfs-utils-1.3.0-0.68.el7.x86_64 1/2
Cleanup : 1:nfs-utils-1.3.0-0.66.el7_8.x86_64 2/2
Verifying : 1:nfs-utils-1.3.0-0.68.el7.x86_64 1/2
Verifying : 1:nfs-utils-1.3.0-0.66.el7_8.x86_64 2/2
Updated:
nfs-utils.x86_64 1:1.3.0-0.68.el7
Complete!
[root@docker_registry ~]#
创建/data/html目录
[root@docker_registry ~]# mkdir /data/html -pv mkdir: created directory ‘/data/html’ [root@docker_registry ~]#
配置该目录能够被k8s集群节点所访问
[root@docker_registry ~]# cat /etc/exports /data/html 192.168.0.0/24(rw,no_root_squash) [root@docker_registry ~]#
提示：以上配置表示把/data/html这个目录以读写，不压榨root权限共享给192.168.0.0/24这个网络中的所有主机使用；
启动nfs
[root@docker_registry ~]# systemctl start nfs [root@docker_registry ~]# ss -tnl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 127.0.0.1:1514 *:* LISTEN 0 128 *:111 *:* LISTEN 0 128 *:20048 *:* LISTEN 0 64 *:42837 *:* LISTEN 0 5 192.168.122.1:53 *:* LISTEN 0 128 *:22 *:* LISTEN 0 128 192.168.0.99:631 *:* LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 64 *:2049 *:* LISTEN 0 128 *:59396 *:* LISTEN 0 128 :::34922 :::* LISTEN 0 128 :::111 :::* LISTEN 0 128 :::20048 :::* LISTEN 0 128 :::80 :::* LISTEN 0 128 :::22 :::* LISTEN 0 100 ::1:25 :::* LISTEN 0 128 :::443 :::* LISTEN 0 128 :::4443 :::* LISTEN 0 64 :::2049 :::* LISTEN 0 64 :::36997 :::* [root@docker_registry ~]#
提示：nfs监听在tcp的2049端口，启动请确保该端口能够正常处于监听状态；到此nfs服务器就准备好了；
在k8s节点上安装nfs-utils包，为其使用nfs提供所需驱动
yum install nfs-utils -y
验证：在node01上，看看能不能正常挂载nfs服务器共享出来的目录
[root@node01 ~]# showmount -e 192.168.0.99 Export list for 192.168.0.99: /data/html 192.168.0.0/24 [root@node01 ~]# mount -t nfs 192.168.0.99:/data/html /mnt [root@node01 ~]# mount |grep /data/html 192.168.0.99:/data/html on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.44,local_lock=none,addr=192.168.0.99) [root@node01 ~]# umount /mnt [root@node01 ~]# mount |grep /data/html [root@node01 ~]#
提示：可以看到在node01上能够正常看到nfs服务器共享出来的目录，并且也能正常挂载使用；等待其他节点把nfs-utils包安装完成以后，接下来就可以在master上应用配置清单了；
应用资源清单
[root@master01 ~]# kubectl apply -f nfs-demo.yaml pod/vol-nfs-demo created [root@master01 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myapp-6479b786f5-9d4mh 1/1 Running 1 2d1h 10.244.2.99 node02.k8s.org <none> <none> myapp-6479b786f5-k252c 1/1 Running 1 2d1h 10.244.4.21 node04.k8s.org <none> <none> vol-hostpath-demo 1/1 Running 0 141m 10.244.2.100 node02.k8s.org <none> <none> vol-nfs-demo 1/1 Running 0 10s 10.244.3.101 node03.k8s.org <none> <none> [root@master01 ~]# kubectl describe pod vol-nfs-demo Name: vol-nfs-demo Namespace: default Priority: 0 Node: node03.k8s.org/192.168.0.46 Start Time: Thu,24 Dec 2020 01:55:51 +0800 Labels: <none> Annotations: <none> Status: Running IP: 10.244.3.101 IPs: IP: 10.244.3.101 Containers: Nginx: Container ID: docker://72227e3a94622a4ea032a1ab0d7d353aef167d5a0e80c3739e774050eaea3914 Image: Nginx:1.14-alpine Image ID: docker-pullable://Nginx@sha256:485b610fefec7ff6c463ced9623314a04ed67e3945b9c08d7e53a47f6d108dc7 Port: <none> Host Port: <none> State: Running Started: Thu,24 Dec 2020 01:55:52 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /usr/share/Nginx/html from webhtml (ro) /var/run/secrets/kubernetes.io/serviceaccount from default-token-xvd4c (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhtml: Type: NFS (an NFS mount that lasts the lifetime of a pod) Server: 192.168.0.99 Path: /data/html/ ReadOnly: false default-token-xvd4c: Type: Secret (a volume populated by a Secret) SecretName: default-token-xvd4c Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 28s default-scheduler Successfully assigned default/vol-nfs-demo to node03.k8s.org Normal Pulled 27s kubelet Container image "Nginx:1.14-alpine" already present on machine Normal Created 27s kubelet Created container Nginx Normal Started 27s kubelet Started container Nginx [root@master01 ~]#
提示：可以看到对应pod已经正常运行，并且其内部容器已经正常挂载对应目录；
在nfs服务器对应目录，创建一个index.html文件
[root@docker_registry ~]# cd /data/html [root@docker_registry html]# echo "this is test file from nfs server ip addr is 192.168.0.99" > index.html [root@docker_registry html]# cat index.html this is test file from nfs server ip addr is 192.168.0.99 [root@docker_registry html]#
[root@master01 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myapp-6479b786f5-9d4mh 1/1 Running 1 2d2h 10.244.2.99 node02.k8s.org <none> <none> myapp-6479b786f5-k252c 1/1 Running 1 2d2h 10.244.4.21 node04.k8s.org <none> <none> vol-hostpath-demo 1/1 Running 0 145m 10.244.2.100 node02.k8s.org <none> <none> vol-nfs-demo 1/1 Running 0 4m6s 10.244.3.101 node03.k8s.org <none> <none> [root@master01 ~]# curl 10.244.3.101 this is test file from nfs server ip addr is 192.168.0.99 [root@master01 ~]#
删除pod
[root@master01 ~]# kubectl delete -f nfs-demo.yaml pod "vol-nfs-demo" deleted [root@master01 ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myapp-6479b786f5-9d4mh 1/1 Running 1 2d2h 10.244.2.99 node02.k8s.org <none> <none> myapp-6479b786f5-k252c 1/1 Running 1 2d2h 10.244.4.21 node04.k8s.org <none> <none> vol-hostpath-demo 1/1 Running 0 149m 10.244.2.100 node02.k8s.org <none> <none> [root@master01 ~]#
绑定pod运行在node02.k8s.org上，重新应用配置文件创建pod，再次访问对应pod，看看对应文件是否能够正常访问到呢？
[root@master01 ~]# cat nfs-demo.yaml apiVersion: v1 kind: Pod Metadata: name: vol-nfs-demo namespace: default spec: nodeName: node02.k8s.org containers: - name: Nginx image: Nginx:1.14-alpine volumeMounts: - name: webhtml mountPath: /usr/share/Nginx/html readOnly: true volumes: - name: webhtml nfs: path: /data/html/ server: 192.168.0.99 [root@master01 ~]# kubectl apply -f nfs-demo.yaml pod/vol-nfs-demo created [root@master01 ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myapp-6479b786f5-9d4mh 1/1 Running 1 2d2h 10.244.2.99 node02.k8s.org <none> <none> myapp-6479b786f5-k252c 1/1 Running 1 2d2h 10.244.4.21 node04.k8s.org <none> <none> vol-hostpath-demo 1/1 Running 0 151m 10.244.2.100 node02.k8s.org <none> <none> vol-nfs-demo 1/1 Running 0 8s 10.244.2.101 node02.k8s.org <none> <none> [root@master01 ~]# curl 10.244.2.101 this is test file from nfs server ip addr is 192.168.0.99 [root@master01 ~]#
提示：可以看到把对应pod绑定到node02上，访问对应pod也能正常访问到nfs服务器上的文件；从上述测试过程来看，nfs这种类型的存储卷能够脱离pod的生命周期，跨节点将pod里容器产生的数据持久化到对应的nfs文件系统服务器上；当然nfs此时是单点，一旦nfs服务器宕机挂掉，对应pod运行时产生的数据将全部丢失；所以对应外部存储系统，我们应该选择一个对数据有冗余，且k8s集群支持的类型的存储系统，比如cephfs，glusterfs等等；
