CentOS 7 安装Kubernetes记录(带ca认证)
一、环境准备
1. 所有节点配置host,使各个Node间可以通过主机名称进行解析。
$ vi /etc/hosts
#加入如下片段(ip地址和servername替换成自己的)
172.16.136.201 server01
172.16.136.202 server02
172.16.136.203 server03
2.所有节点关闭防火墙
systemctl disable firewalld systemctl stop firewalld
3.所有节点禁用SELinux
setenforce 0
4.所有节点关闭NetworkManager(虚拟机环境)
systemctl stop NetworkManager systemctl disable NetworkManager
在虚拟机环境有时会出现
Failed to start LSB: Bring up/down networking.
错误,network无法启动。
5.关闭swap
为了性能考虑
swapoff -a
二、安装Docker (所有节点)
1.安装脚本
wget -qO- https://get.docker.com/ | sh
2.修改docker的网络转发规则
vi /lib/systemd/system/docker.service
#找到ExecStart=xxx,在这行上面加入一行,内容如下:(k8s的网络需要)
ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT
3.(可选)给docker配置加速
curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://0ed63f2c.m.daocloud.io
4.使配置生效并启动docker
systemctl daemon-reload systemctl start docker systemctl enable docker
5.验证是否安装完成
$ docker version
Client:
Version: 18.03.0-ce
API version: 1.37
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:09:15 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.0-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:13:03 2018
OS/Arch: linux/amd64
Experimental: false
三、Kubernetes安装
1. 设置系统参数 (所有节点)
- 允许路由转发,不对bridge的数据进行处理
#写入配置文件
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
#生效配置文件
sysctl -p /etc/sysctl.d/k8s.conf
2. 下载kubernetes二进制文件 (所有节点)
下载完成后上传到服务器用户目录并解压,并将其目录加入到PATH环境变量中。
tar -zxf kubernetes-bins.tar.gz;rm -rf kubernetes-bins.tar.gz;mv kubernetes-bins bin
#添加环境变量
vim /etc/profile
export PATH=$PATH:/root/bin
source /etc/profile
3. 下载配置文件并生成配置文件 (所有节点)
使用
yum install git
安装Git
git clone https://github.com/KingBoyWorld/kubernetes-starter
3.1 生成带有权限认证的配置文件 (所有节点)
cd ~/kubernetes-starter && vi config.properties && ./gen-config.sh with-ca
需要注意的是其中主节点使用https, 如下示例配置
#kubernetes二进制文件目录,eg: /home/michael/bin
BIN_PATH=/root/bin
#当前节点ip,eg: 192.168.1.102
NODE_IP=172.16.136.201
#etcd服务集群列表,eg: http://192.168.1.102:2379
#如果已有etcd集群可以填写现有的。没有的话填写:http://${MASTER_IP}:2379 (MASTER_IP自行替换成自己的主节点ip)
ETCD_ENDPOINTS=https://172.16.136.202:2379
#kubernetes主节点ip地址,eg: 192.168.1.102
MASTER_IP=172.16.136.202
4. 安装cfssl并生成根证书 (所有节点)
4.1 安装cfssl(所有节点)
用来生成证书
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 && wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 && chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 && mv cfssl_linux-amd64 /usr/local/bin/cfssl && mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
cfssl version
4.2 生成根证书 (主节点)
mkdir -p /etc/kubernetes/ca && cp ~/kubernetes-starter/target/ca/ca-config.json /etc/kubernetes/ca && cp ~/kubernetes-starter/target/ca/ca-csr.json /etc/kubernetes/ca && cd /etc/kubernetes/ca && cfssl gencert -initca ca-csr.json | cfssljson -bare ca && ls && cd ~/kubernetes-starter
5. 安装ETCD服务 (主节点)
5.1 生成ETCD证书 (主节点)
mkdir -p /etc/kubernetes/ca/etcd && cp ~/kubernetes-starter/target/ca/etcd/etcd-csr.json /etc/kubernetes/ca/etcd/ && cd /etc/kubernetes/ca/etcd/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd && ls && cd ~/kubernetes-starter
5.2 安装并启动ETCD (主节点)
mkdir -p /var/lib/etcd && cp ~/kubernetes-starter/target/master-node/etcd.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable etcd && systemctl start etcd
验证是否安装完成
$ ETCDCTL_API=3 etcdctl \
--endpoints=https://172.16.136.202:2379 \
--cacert=/etc/kubernetes/ca/ca.pem \
--cert=/etc/kubernetes/ca/etcd/etcd.pem \
--key=/etc/kubernetes/ca/etcd/etcd-key.pem \
endpoint health
#如下输出说明成功
https://172.16.136.202:2379 is healthy: successfully committed proposal: took = 1.04668ms
6. 安装APIServer (主节点)
6.1 生成证书 (主节点)
mkdir -p /etc/kubernetes/ca/kubernetes && cp ~/kubernetes-starter/target/ca/kubernetes/kubernetes-csr.json /etc/kubernetes/ca/kubernetes/ && cd /etc/kubernetes/ca/kubernetes/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes && ls && cd ~/kubernetes-starter
6.2 生成token认证文件 (主节点)
$ head -c 16 /dev/urandom | od -An -t x | tr -d ' '
8afdf3c4eb7c74018452423c29433609
按照固定格式写入token.csv,注意替换token内容
echo "8afdf3c4eb7c74018452423c29433609,kubelet-bootstrap,10001,\"system:kubelet-bootstrap\"" > /etc/kubernetes/ca/kubernetes/token.csv
6.3 启动APIServer (主节点)
cp ~/kubernetes-starter/target/master-node/kube-apiserver.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable kube-apiserver.service && systemctl start kube-apiserver && cd ~/kubernetes-starter
验证
journalctl -f -u kube-apiserver
7. 安装ControllerManager (主节点)
安装
cp ~/kubernetes-starter/target/master-node/kube-controller-manager.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl start kube-controller-manager
验证
journalctl -f -u kube-controller-manager
8. 安装Scheduler (主节点)
安装
cp ~/kubernetes-starter/target/master-node/kube-scheduler.service /lib/systemd/system/ && systemctl enable kube-scheduler.service && systemctl start kube-scheduler
验证
journalctl -f -u kube-scheduler
9. 配置kubectl (主节点)
9.1 生成证书 (主节点)
mkdir -p /etc/kubernetes/ca/admin && cp ~/kubernetes-starter/target/ca/admin/admin-csr.json /etc/kubernetes/ca/admin/ && cd /etc/kubernetes/ca/admin/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin && ls && cd ~/kubernetes-starter
9.2 配置kubectl (主节点)
注意修改其中的IP地址
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ca/ca.pem --embed-certs=true --server=https://172.16.136.202:6443 && kubectl config set-credentials admin --client-certificate=/etc/kubernetes/ca/admin/admin.pem --embed-certs=true --client-key=/etc/kubernetes/ca/admin/admin-key.pem && kubectl config set-context kubernetes --cluster=kubernetes --user=admin && kubectl config use-context kubernetes
cat ~/.kube/config
验证
$ kubectl get componentstatus
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
scheduler Healthy ok
10 配置calico-node (所有节点)
10.1 生成证书 (主节点)
mkdir -p /etc/kubernetes/ca/calico && cp ~/kubernetes-starter/target/ca/calico/calico-csr.json /etc/kubernetes/ca/calico/ && cd /etc/kubernetes/ca/calico/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes calico-csr.json | cfssljson -bare calico && ls && cd ~/kubernetes-starter
10.2 拷贝证书到其它机器上 (主节点)
可能需要在两台机器上建立
/etc/kubernetes
目录
scp -r /etc/kubernetes/ca/ root@172.16.136.201:/etc/kubernetes/ca/
scp -r /etc/kubernetes/ca/ root@172.16.136.201:/etc/kubernetes/ca/
10.2 启动calio-node (所有节点)
需要下载docker镜像,会慢一些
cp ~/kubernetes-starter/target/all-node/kube-calico.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable kube-calico && systemctl start kube-calico
查看日志
journalctl -f -u kube-calico
验证
calicoctl node status
11 安装kubelet (工作节点)
11.1 配置查看 (主节点)
#可以通过下面命令查询clusterrole列表
$ kubectl -n kube-system get clusterrole
#可以回顾一下token文件的内容
$ cat /etc/kubernetes/ca/kubernetes/token.csv
8afdf3c4eb7c74018452423c29433609,10001,"system:kubelet-bootstrap"
11.2 环境准备
(工作节点)
mkdir -p /var/lib/kubelet && mkdir -p /etc/kubernetes && mkdir -p /etc/cni/net.d
(主节点)
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
11.3 生成bootstrap.kubeconfig配置文件 (工作节点)
注意替换IP地址 和 token
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ca/ca.pem --embed-certs=true --server=https://172.16.136.202:6443 --kubeconfig=bootstrap.kubeconfig && kubectl config set-credentials kubelet-bootstrap --token=8afdf3c4eb7c74018452423c29433609 --kubeconfig=bootstrap.kubeconfig && kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=bootstrap.kubeconfig && kubectl config use-context default --kubeconfig=bootstrap.kubeconfig && mv bootstrap.kubeconfig /etc/kubernetes/
11.4 cni配置 (工作节点)
cp ~/kubernetes-starter/target/worker-node/10-calico.conf /etc/cni/net.d/
11.5 启动kubelet服务 (工作节点)
cp ~/kubernetes-starter/target/worker-node/kubelet.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable kubelet && systemctl start kubelet
#启动kubelet之后到master节点允许worker加入(批准worker的tls证书请求)
#--------*在主节点执行*---------
$ kubectl get csr|grep 'Pending' | awk '{print $1}'| xargs kubectl certificate approve
#-----------------------------
检查日志
journalctl -f -u kubelet
12 安装kube-proxy (工作节点)
创建工作目录 (工作节点)
mkdir -p /var/lib/kube-proxy
12.1 生成证书 (工作节点)
mkdir -p /etc/kubernetes/ca/kube-proxy && cp ~/kubernetes-starter/target/ca/kube-proxy/kube-proxy-csr.json /etc/kubernetes/ca/kube-proxy/ && cd /etc/kubernetes/ca/kube-proxy/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy && ls && cd ~/kubernetes-starter
12.2 生成kube-proxy.kubeconfig配置 (工作节点)
注意替换IP地址
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ca/ca.pem --embed-certs=true --server=https://172.16.136.202:6443 --kubeconfig=kube-proxy.kubeconfig && kubectl config set-credentials kube-proxy --client-certificate=/etc/kubernetes/ca/kube-proxy/kube-proxy.pem --client-key=/etc/kubernetes/ca/kube-proxy/kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig && kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig && kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig && mv kube-proxy.kubeconfig /etc/kubernetes/kube-proxy.kubeconfig
12.3 启动kube-proxy (工作节点)
cp ~/kubernetes-starter/target/worker-node/kube-proxy.service /lib/systemd/system/ && systemctl daemon-reload && yum -y install conntrack && systemctl enable kube-proxy && systemctl start kube-proxy
检查
journalctl -f -u kube-proxy
13 部署kube-dns (主节点)
kubectl create -f ~/kubernetes-starter/target/services/kube-dns.yaml
验证
kubectl -n kube-system get pods