二进制部署k8s
kubernetes,docker
容器技术
发布日期:   2019-04-30

Centos7系统使用二进制包部署kubernetes1.13集群

一 简述
Kubernetes1.13 的核心特性包括：利用kubeadm简化集群管理、容器存储接口(CSI)以及将CoreDNS作为默认DNS。
利用kubeadm简化集群管理功能
Kubeadm是管理集群生命周期的重要工具，能够帮助从我们从创建到配置再到升级的整个过程，随着1.13版本的发布，kubeadm功能进入GA版本，正式普遍可用。kubeadm处理现有硬件上的生产集群的引导，并以最佳实践方式配置kubernetes核心组件，以便为新节点提供安全而简单的连接流程并支持轻松升级。
该GA版本中最值得注意的是已经毕业的高级功能，尤其是可插拔性和可配置性。kubeadm旨在为管理员与高级自动化系统提供一套工具箱，如今已迈出重要一步。
容器存储接口(CSI)
容器存储接口最初于1.9版本中作为alpha测试功能引入，在1.10版本中进入beta测试，如今终于进入GA阶段正式普遍可用。在CSI的帮助下，kubernetes卷层将真正实现可扩展性。通过CSI，第三方存储供应商将可以直接编写可与kubernetes互操作的代码，而无需触及任何kubernetes核心代码。事实上，相关规范也已经同步进入1.0阶段。随着CSI的稳定，插件作者将能够按照自己的节奏开发核心存储插件，详见CSI文档。
CoreDNS称为Kubernetes的默认DNS服务器
在1.11版本中，开发团队宣布CoreDNS已实现基于DNS服务发现的普遍可用性。在最新的1.13版本中，CoreDNS正式去掉kuber-dns成为Kubernetes中的默认DNS服务器。CoreDNS是一种通用的、权威的DNS服务器，能够提供与Kubernetes向下兼容且具备可扩展性的集成能力。由于CoreDNS自身单一可执行文件与单一进程的特性，因此CoreDNS的活动部件数量会少于之前的DNS服务器，且能够通过创建自定义DNS条目来支持各类灵活的用例。此外，由于CoreDNS采用Go语言编写，它具有强大的内存安全性。

CoreDNS现在是kubernetes1.13急后续版本推荐的DNS解决方案，Kubernetes已将常用测试基础设施架构切换为默认使用CoreDNS,因此，开发团队建议用户也尽快完成切换。KubeDNS仍将至少支持一个版本，但现在是时候开始规划迁移了。

centos7系统的内网服务器三台：master:192.168.1.119;node1:192.168.1.118;node2:192.168.1.163

一 初始化系统环境和安装指定版本的docker(三台机器同时进行如下部署)(墙外环境)
1 关闭防火墙和SELINUX

systemctl stop firewalld.service && systemctl disable firewalld.service
# 禁用SELINUX,临时生效
setenforce 0
# 修改配置文件，重启后生效
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
或者打开配置文件修改
vi /etc/selinux/config
SELINUX=disabled

2 关闭swap

swapoff -a && sysctl -w vm.swappiness=0
vi /etc/fstab
#UUID=7bff6243-324c-4587-b550-55dc34018ebf swap   swap defaults   0 0

3 设置Docker所需参数

cat << EOF | tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf

4 安装Docker

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# 如果没有yum-config-manager命令的话，请安装，步骤如下
-------------------------------------- start -------------------------
[root@rancherserver ~]# yum search yum-config-manager
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.uhost.hk
 * epel: mirror.pregi.net
 * extras: centos.uhost.hk
 * updates: centos.uhost.hk
======================================================================================================================
匹配：yum-config-manager ======================================================================================================================
yum-utils.noarch : Utilities based around the yum package manager

yum -y install yum-utils # 安装yum-config-manager的依赖包就可以了
------------------------------------ end ------------------------------

yum list docker-ce --showduplicates | sort -r
yum install docker-ce-18.06.0.ce-3.el7 -y
systemctl start docker && systemctl enable docker

二 下面只需在192.168.1.119 master主机上部署

2.1 创建k8s目录和下载自制证书的工具

###  2.1.1 创建安装目录
mkdir -p  /home/k8s/etcd/{bin,cfg,ssl}
mkdir -p  /home/k8s/kubernetes/{bin,cfg,ssl}
mkdir -p  /home/k8s/cfssl/{etcd,kubernetes}  #(这两个目录用来存放相关证书文件)
### 2.1.2 安装配置CFSSL(CFSSL是自己创建证书的工具)
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

2.2 开始创建证书

2.2.1创建ETCD证书(目录为/home/k8s/cfssl/etcd/)

cat << EOF | tee ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

2.2.2 创建ETCD CA配置文件

cat << EOF | tee ca-csr.json
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Guangzhou",
            "ST": "Guangzhou"
        }
    ]
}
EOF

2.2.3创建ETCD Server证书

cat << EOF | tee server-csr.json
{
    "CN": "etcd",
    "hosts": [
    "192.168.1.119",
    "192.168.1.118",
    "192.168.1.63"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Guangzhou",
            "ST": "Guangzhou"
        }
    ]
}
EOF

2.2.4生成ETCD CA证书和私钥

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

2.2.5创建Kubernetes CA证书

cat << EOF | tee ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat << EOF | tee ca-csr.json
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Guangzhou",
            "ST": "Guangzhou",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

2.2.6 生成API_SERVER证书

cat << EOF | tee server-csr.json
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "192.168.1.119",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Guangzhou",
            "ST": "Guangzhou",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

2.2.7 创建Kubernetes Proxy证书

cat << EOF | tee kube-proxy-csr.json
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Guangzhou",
      "ST": "Guangzhou",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

2.3 Master主机到其它两个node节点做ssh-key认证

# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:FQjjiRDp8IKGT+UDM+GbQLBzF3DqDJ+pKnMIcHGyO/o root@qas-k8s-master01
The key's randomart image is:
+---[RSA 2048]----+
|o.==o o. ..      |
|ooB+o+ o.  .     |
|B++@o o   .      |
|=X**o    .       |
|o=O. .  S        |
|..+              |
|oo .             |
|* .              |
|o+E              |
+----[SHA256]-----+

# ssh-copy-id 192.168.1.119
# ssh-copy-id 192.168.1.63
# 上面一步要输入这两台机器的密码

三 部署ETCD服务集群(三台机器同时都需要部署)

三个节点都要下载etcd源码包，三台机器通过xshell同时连接

cd /usr/local/src/
wget http://down.cdn1688.net/k8s1.13/etcd-v3.3.10-linux-amd64.tar.gz
tar -xvf etcd-v3.3.10-linux-amd64.tar.gz
cd etcd-v3.3.10-linux-amd64/ && cp etcd etcdctl /home/k8s/etcd/bin/

192.168.1.119 Master节点部署如下

vim /home/k8s/etcd/cfg/etcd

#[Member]
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.119:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.119:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.119:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.119:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.119:2380,etcd2=https://192.168.1.118:2380,etcd3=https://192.168.1.63:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

创建etcd的systemd unit文件

vim /usr/lib/systemd/system/etcd.service 
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/home/k8s/etcd/cfg/etcd
ExecStart=/home/k8s/etcd/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/home/k8s/etcd/ssl/server.pem \
--key-file=/home/k8s/etcd/ssl/server-key.pem \
--peer-cert-file=/home/k8s/etcd/ssl/server.pem \
--peer-key-file=/home/k8s/etcd/ssl/server-key.pem \
--trusted-ca-file=/home/k8s/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/home/k8s/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

把目录/home/k8s/cfssl/etcd/下制作的证书文件拷贝到/home/k8s/etcd/ssl目录下用于启动etcd服务

cp /home/k8s/cfssl/etcd/  /home/k8s/etcd/ssl/

192.168.1.118 node1节点部署如下

把master的etcd的配置文件和启动文件拷贝到node1上
# 进入192.168.1.119(master)的/home/k8s/目录
scp -r /home/k8s/etcd    192.168.1.118:/home/k8s/
scp -r /usr/lib/systemd/system/etcd.service 192.168.1.118:/usr/lib/systemd/system/etcd.service
# 进入192.168.1.118(node1)需要对/home/k8s/etcd/cfg/etcd配置文件进行修改
vim /home/k8s/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.118:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.118:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.118:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.118:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.119:2380,etcd2=https://192.168.1.118:2380,etcd3=https://192.168.1.63:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

node1节点的 etcd的 systemd unit 文件(启动文件)不需要修改

192.168.1.163 node2节点部署如下

把master的etcd的配置文件和启动文件拷贝到node1上
# 进入192.168.1.119(master)的/home/k8s/目录
scp -r /home/k8s/etcd    192.168.1.163:/home/k8s/
scp -r /usr/lib/systemd/system/etcd.service 192.168.1.63:/usr/lib/systemd/system/etcd.service
# 进入192.168.1.63(node2)需要对/home/k8s/etcd/cfg/etcd配置文件进行修改
vim /home/k8s/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.63:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.63:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.63:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.63:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.119:2380,etcd2=https://192.168.1.118:2380,etcd3=https://192.168.1.63:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

node2节点的 etcd的 systemd unit 文件(启动文件)也不需要修改

三节点同时启动etcd服务（注意事项：ETCD集群启动时，必须两个或者三个节点同时启动，启动一个节点集群是无法正常启动的）

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd

###
验证集群是否正常运行

/home/k8s/etcd/bin/etcdctl \
--ca-file=/home/k8s/etcd/ssl/ca.pem \
--cert-file=/home/k8s/etcd/ssl/server.pem \
--key-file=/homek8s/etcd/ssl/server-key.pem \
--endpoints="https://192.168.1.119:2379,\
https://192.168.1.118:2379,\
https://192.168.1.63:2379" cluster-health

四 三节点部署Flannel网络

先在master上部署Flannel网络，登录master机器

向etcd写入集群Pod网段信息

cd /home/k8s/etcd/ssl/
/home/k8s/etcd/bin/etcdctl \
--ca-file=/home/k8s/etcd/ssl/ca.pem  \
--cert-file=/home/k8s/etcd/ssl/server.pem \
--key-file=/home/k8s/etcd/ssl/server-key.pem \
--endpoints="https://192.168.1.119:2379,\
https://192.168.1.118:2379,https://192.168.1.63:2379" \
set /coreos.com/network/config  '{ "Network": "172.18.0.0/16", "Backend": {"Type": "vxlan"}}'

下面这两句话部署的过程中我不懂啥意思，尤其是第一句简直就是懵逼了

  • flanneld当前版本(v0.10.0)不支持etcdv3,故使用etcd v2 API写入配置key和网段数据
  • 写入Pod网段 ${CLUSTER_CIDR}必须是/16段地址，必须与kube-controller-的-cluster-cidr参数值一致;

下载安装flannel软件

cd /usr/local/src/ && wget http://down.52zbz.com/k8s/flannel-v0.10.0-linux-amd64.tar.gz
tar -xvf flannel-v0.10.0-linux-amd64.tar.gz
mv flannel mk-docker-opts.sh /home/k8s/kubernetes/bin/
创建Flannel配置文件
vim /home/k8s/kubernetes/cfg/flanneld
FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.1.119:2379,https://192.168.1.118:2379,https://192.168.1.63:2379 -etcd-cafile=/home/k8s/etcd/ssl/ca.pem -etcd-certfile=/home/k8s/etcd/ssl/server.pem -etcd-keyfile=/home/k8s/etcd/ssl/server-key.pem"

创建flanneld的systemd unit文件

vim /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/home/k8s/kubernetes/cfg/flanneld
ExecStart=/home/k8s/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/home/k8s/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

[Install]
WantedBy=multi-user.target
  • mk-docker-opts.sh脚本分配给flanneld的Pod子网网段信息写入/run/flannel/docker文件，后续docker启动时使用这个文件中的环境变量配置docker0网桥;
  • flanneld使用系统缺省路由所在的接口与其它节点通信，对于有多个网络接口(如内网和公网)的节点，可以用-iface参数指定通信接口，如上面的eth0接口;
  • flanneld运行时需要root权限
    配置Docker启动指定子网段
vim /usr/lib/systemd/system/docker.service 
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

注意：对上面的文件内容做一下解析：

  • FLannel网络必须在宿主机网络能对外(其它node节点)正常通信的情况下启动才有意义，所以这里定义After=network.target
  • 只有当Flannel网络启动之后，才能创建一个与其它节点不会冲突的网络，而docker的网络需要和fannel网络相同才能保证跨主机通信，所以docker必须要在flannel网络创建后才能启动，这里定义before=docker.service
    将flanneld systemd unit文件拷贝到所有节点(在master上操作)
#把对应的配置文件拷贝到node1和node2上
scp -r /home/k8s/kubernetes    192.168.1.118:/home/k8s/
scp -r /home/k8s/kubernetes    192.168.1.63:/home/k8s/
scp /home/k8s/kubernetes/cfg/flanneld   192.168.1.118:/home/k8s/kubernetes/cfg/flanneld
scp /home/k8s/kubernetes/cfg/flanneld   192.168.1.63:/home/k8s/kubernetes/cfg/flanneld
scp /usr/lib/systemd/system/docker.server 192.168.1.118:/usr/lib/systemd/system/docker.servier
scp /usr/lib/systemd/system/docker.server   192.168.1.63:/usr/lib/systemd/system/docker.servier
scp /usr/lib/systemd/system/flanneld.service  192.168.1.118:/usr/lib/systemd/system/flanneld.servier
scp /usr/lib/systemd/system/flanneld.service  192.168.1.63:/usr/lib/systemd/system/flanneld.servier

三节点都执行下面命令：启动服务和重启docker

systemctl daemon-reload
systemctl start flanneld && systemctl enable flanneld
systemctl restart docker

可以在各个节点上查看是否生效(下面是在master、node1、node2上的操作结果)

[root@docker-1 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 80:18:44:e3:61:e4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.119/24 brd 192.168.1.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::4e7f:7ff9:fbc0:d131/64 scope link 
       valid_lft forever preferred_lft forever
3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 80:18:44:e3:61:e5 brd ff:ff:ff:ff:ff:ff
4: em3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 80:18:44:e3:61:e6 brd ff:ff:ff:ff:ff:ff
5: em4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 80:18:44:e3:61:e7 brd ff:ff:ff:ff:ff:ff
6: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP 
    link/ether 02:42:24:16:72:04 brd ff:ff:ff:ff:ff:ff
    inet 172.18.89.1/24 brd 172.18.89.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:24ff:fe16:7204/64 scope link 
       valid_lft forever preferred_lft forever
7: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN 
    link/ether 4a:a6:d0:d3:f2:ce brd ff:ff:ff:ff:ff:ff
    inet 172.18.89.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
11: vethea3f23b@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP 
    link/ether be:59:a5:d3:cd:41 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::bc59:a5ff:fed3:cd41/64 scope link 
       valid_lft forever preferred_lft forever
[root@docker-2 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 80:18:44:e3:56:c0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.118/24 brd 192.168.1.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::4585:be92:8558:dbf6/64 scope link 
       valid_lft forever preferred_lft forever
3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 80:18:44:e3:56:c1 brd ff:ff:ff:ff:ff:ff
4: em3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 80:18:44:e3:56:c2 brd ff:ff:ff:ff:ff:ff
5: em4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 80:18:44:e3:56:c3 brd ff:ff:ff:ff:ff:ff
6: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP 
    link/ether 02:42:e2:a7:53:8f brd ff:ff:ff:ff:ff:ff
    inet 172.18.99.1/24 brd 172.18.99.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:e2ff:fea7:538f/64 scope link 
       valid_lft forever preferred_lft forever
7: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN 
    link/ether ba:7d:4e:4b:6a:ef brd ff:ff:ff:ff:ff:ff
    inet 172.18.99.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
9: vethd9144bb@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP 
    link/ether c2:39:12:a7:be:cd brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::c039:12ff:fea7:becd/64 scope link 
       valid_lft forever preferred_lft forever
11: vethec20706@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP 
    link/ether de:d4:47:90:8d:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::dcd4:47ff:fe90:8da5/64 scope link 
       valid_lft forever preferred_lft forever
#######################################################################################
[root@docker-3 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 20:47:47:8d:f4:9c brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.63/24 brd 192.168.1.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::976b:ad75:61c6:8c2d/64 scope link 
       valid_lft forever preferred_lft forever
3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 20:47:47:8d:f4:9d brd ff:ff:ff:ff:ff:ff
4: em3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 20:47:47:8d:f4:9e brd ff:ff:ff:ff:ff:ff
5: em4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 20:47:47:8d:f4:9f brd ff:ff:ff:ff:ff:ff
6: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP 
    link/ether 02:42:79:c7:6b:62 brd ff:ff:ff:ff:ff:ff
    inet 172.18.72.1/24 brd 172.18.72.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:79ff:fec7:6b62/64 scope link 
       valid_lft forever preferred_lft forever
7: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN 
    link/ether 4e:cd:29:4b:df:1b brd ff:ff:ff:ff:ff:ff
    inet 172.18.72.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
13: veth1e4d6e8@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP 
    link/ether fe:6c:77:28:be:eb brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::fc6c:77ff:fe28:beeb/64 scope link 
       valid_lft forever preferred_lft forever
17: veth7a9dd40@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP 
    link/ether 7a:05:16:52:c1:4a brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::7805:16ff:fe52:c14a/64 scope link 
       valid_lft forever preferred_lft forever

这时三个节点都部署了etcd、flannled、docker服务了,下面是master、node1、node2上的显示

[root@docker-1 ~]# netstat -lntp |grep etcd
tcp        0      0 192.168.1.119:2379      0.0.0.0:*               LISTEN      12651/etcd          
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      12651/etcd          
tcp        0      0 192.168.1.119:2380      0.0.0.0:*               LISTEN      12651/etcd   

[root@docker-1 ~]# ps -ef |grep flannel
root      12903      1  0 4月29 ?       00:08:32 /home/k8s/kubernetes/bin/flanneld --ip-masq --etcd-endpoints=https://192.168.1.119:2379,https://192.168.1.118:2379,https://192.168.1.63:2379 -etcd-cafile=/home/k8s/etcd/ssl/ca.pem -etcd-certfile=/home/k8s/etcd/ssl/server.pem -etcd-keyfile=/home/k8s/etcd/ssl/server-key.pem
root      64750  64185  0 15:46 pts/0    00:00:00 grep --color=auto flannel

[root@docker-1 ~]# ps -ef |grep docker
root      13017      1  0 4月29 ?       01:25:11 /usr/bin/dockerd --bip=172.18.89.1/24 --ip-masq=false --mtu=1450
root      13031  13017  0 4月29 ?       00:55:16 docker-containerd --config /var/run/docker/containerd/containerd.toml
root      65318  64185  0 15:51 pts/0    00:00:00 grep --color=auto docker

#######################################################################################
[root@docker-2 ~]# 
[root@docker-2 ~]# netstat -lntp |grep etcd
tcp        0      0 192.168.1.118:2379      0.0.0.0:*               LISTEN      11976/etcd          
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      11976/etcd          
tcp        0      0 192.168.1.118:2380      0.0.0.0:*               LISTEN      11976/etcd   

[root@docker-2 ~]# ps -ef |grep flanneld
root      12131      1  0 4月29 ?       00:08:42 /home/k8s/kubernetes/bin/flanneld --ip-masq --etcd-endpoints=https://192.168.1.119:2379,https://192.168.1.118:2379,https://192.168.1.63:2379 -etcd-cafile=/home/k8s/etcd/ssl/ca.pem -etcd-certfile=/home/k8s/etcd/ssl/server.pem -etcd-keyfile=/home/k8s/etcd/ssl/server-key.pem
root     114125 114001  0 15:47 pts/0    00:00:00 grep --color=auto flanneld

[root@docker-2 ~]# ps -ef |grep docker
root      12243      1  0 4月29 ?       01:40:32 /usr/bin/dockerd --bip=172.18.99.1/24 --ip-masq=false --mtu=1450
root      12258  12243  1 4月29 ?       02:13:34 docker-containerd --config /var/run/docker/containerd/containerd.toml

#######################################################################################
[root@docker-3 ~]# netstat -lntp |grep etcd
tcp        0      0 192.168.1.63:2379       0.0.0.0:*               LISTEN      11645/etcd          
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      11645/etcd          
tcp        0      0 192.168.1.63:2380       0.0.0.0:*               LISTEN      11645/etcd   

[root@docker-3 ~]# ps -ef |grep flanneld
root      11781      1  0 4月29 ?       00:08:47 /home/k8s/kubernetes/bin/flanneld --ip-masq --etcd-endpoints=https://192.168.1.119:2379,https://192.168.1.118:2379,https://192.168.1.63:2379 -etcd-cafile=/home/k8s/etcd/ssl/ca.pem -etcd-certfile=/home/k8s/etcd/ssl/server.pem -etcd-keyfile=/home/k8s/etcd/ssl/server-key.pem
root      35948  35741  0 15:45 pts/0    00:00:00 grep --color=auto flanneld

[root@docker-3 ~]# ps -ef |grep docker
root      11892      1  0 4月29 ?       01:42:50 /usr/bin/dockerd --bip=172.18.72.1/24 --ip-masq=false --mtu=1450
root      11906  11892  1 4月29 ?       02:25:14 docker-containerd --config /var/run/docker/containerd/containerd.toml

五 部署master节点

kubernetes master节点运行如下组件：

  • kuber-apiserver
  • kube-scheduler
  • kube-controller-manager
  • 注意：kuber-scheduler和kube-controlller-manager可以以集群模式运行，通过leader选举产生一个工作进程，其它进程处于阻塞模式。

将kubernetes-server二进制文件解压拷贝到master节点

cd /usr/local/src && wget http://down.cdn1688.net/k8s1.13/kubernetes-server-linux-amd64.tar.gz
# 特别注意下面这目录
cd /usr/local/src/kubernetes/server/bin 
cp kube-scheduler kube-apiserver kube-controller-manager kubectl /home/k8s/kubernetes/bin/
## 拷贝证书
cp /home/k8s/cfssl/kubernetes/*pem  /home/k8s/kubernetes/ssl/

部署kube-apiserver组件

# 创建TLS Bootstrapping Token
> head -c 16 /dev/urandom | od -An -t x | tr -d ' '
b2d81324fb83862f435f66498b90e5bd
vim /home/k8s/kubernetes/cfg/token.csv
b2d81324fb83862f435f66498b90e5bd,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

创建apiserver配置文件

vim /home/k8s/kubernetes/cfg/kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://192.168.1.119:2379,https://192.168.1.118:2379,https://192.168.1.63:2379 \
--bind-address=192.168.1.119 \
--secure-port=6443 \
--advertise-address=192.168.1.119 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth \
--token-auth-file=/home/k8s/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/home/k8s/kubernetes/ssl/server.pem  \
--tls-private-key-file=/home/k8s/kubernetes/ssl/server-key.pem \
--client-ca-file=/home/k8s/kubernetes/ssl/ca.pem \
--service-account-key-file=/home/k8s/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/home/k8s/etcd/ssl/ca.pem \
--etcd-certfile=/home/k8s/etcd/ssl/server.pem \
--etcd-keyfile=/home/k8s/etcd/ssl/server-key.pem"

创建kube-apiserver systemd unit文件

vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/home/k8s/kubernetes/cfg/kube-apiserver
ExecStart=/home/k8s/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动kube-apiserver服务

systemctl daemon-reload
systemctl enable kube-apiserver
systemctl restart kube-apiserver

# 查看apiserver是否运行
[root@docker-1 cfg]# ps  -ef | grep kube-apiserver 
root      16331      1  3 4月29 ?       06:43:45 /home/k8s/kubernetes/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://192.168.1.119:2379,https://192.168.1.118:2379,https://192.168.1.63:2379 --bind-address=192.168.1.119 --secure-port=6443 --advertise-address=192.168.1.119 --allow-privileged=true --service-cluster-ip-range=10.0.0.0/24 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --enable-bootstrap-token-auth --token-auth-file=/home/k8s/kubernetes/cfg/token.csv --service-node-port-range=30000-50000 --tls-cert-file=/home/k8s/kubernetes/ssl/server.pem --tls-private-key-file=/home/k8s/kubernetes/ssl/server-key.pem --client-ca-file=/home/k8s/kubernetes/ssl/ca.pem --service-account-key-file=/home/k8s/kubernetes/ssl/ca-key.pem --etcd-cafile=/home/k8s/etcd/ssl/ca.pem --etcd-certfile=/home/k8s/etcd/ssl/server.pem --etcd-keyfile=/home/k8s/etcd/ssl/server-key.pem
root     109163 104117  0 21:41 pts/0    00:00:00 grep --color=auto kube-apiserver

部署kube-scheduler

# 创建kube-scheduler配置文件
vim /home/k8s/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="--logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect"
  • -address:在127.0.0.1:10251 端口接收http/metrics请求；kube-scheduler目前还不支持接收https请求；
  • -kubeconfig:指定kubeconfig文件路径，kube-scheduler使用它连接和验证kube-apiserver;
  • -leader-elect=true:集群运行模式，启用选举功能；被选为leader的节点负责处理工作，其它节点为阻塞状态；

创建kube-scheduler systemd unit文件

vim /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/home/k8s/kubernetes/cfg/kube-scheduler
ExecStart=/home/k8s/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动服务

systemctl daemon-reload
systemctl enable kube-scheduler.service
systemctl .restart kube-scheduler.service

## 查看kube-scheduler是否运行和状态

[root@docker-1 cfg]#  ps -ef |grep kube-scheduler 
root      32101      1  1 4月29 ?       02:01:04 /home/k8s/kubernetes/bin/kube-scheduler --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect
root     110330 104117  0 21:50 pts/0    00:00:00 grep --color=auto kube-scheduler

[root@docker-1 cfg]# systemctl status kube-scheduler.service
● kube-scheduler.service - Kubernetes Scheduler
   Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2019-04-29 21:21:08 CST; 1 weeks 1 days ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 32101 (kube-scheduler)
   Memory: 26.1M
   CGroup: /system.slice/kube-scheduler.service
           └─32101 /home/k8s/kubernetes/bin/kube-scheduler --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect

5月 07 21:44:19 docker-1 kube-scheduler[32101]: I0507 21:44:19.379442   32101 reflector.go:357] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:232: Watch close - *v1.Pod total 3 items received
5月 07 21:44:44 docker-1 kube-scheduler[32101]: I0507 21:44:44.318917   32101 reflector.go:357] k8s.io/client-go/informers/factory.go:132: Watch close - *v1.Service total 0 items received
5月 07 21:45:09 docker-1 kube-scheduler[32101]: I0507 21:45:09.308219   32101 reflector.go:357] k8s.io/client-go/informers/factory.go:132: Watch close - *v1.PersistentVolumeClaim total 0 items received
5月 07 21:48:08 docker-1 kube-scheduler[32101]: I0507 21:48:08.319573   32101 reflector.go:357] k8s.io/client-go/informers/factory.go:132: Watch close - *v1.Node total 105 items received
5月 07 21:48:26 docker-1 kube-scheduler[32101]: I0507 21:48:26.301859   32101 reflector.go:357] k8s.io/client-go/informers/factory.go:132: Watch close - *v1.ReplicaSet total 4 items received
5月 07 21:48:34 docker-1 kube-scheduler[32101]: I0507 21:48:34.319324   32101 reflector.go:357] k8s.io/client-go/informers/factory.go:132: Watch close - *v1.ReplicationController total 0 items received
5月 07 21:49:20 docker-1 kube-scheduler[32101]: I0507 21:49:20.328175   32101 reflector.go:357] k8s.io/client-go/informers/factory.go:132: Watch close - *v1.PersistentVolume total 0 items received
5月 07 21:49:22 docker-1 kube-scheduler[32101]: I0507 21:49:22.380500   32101 reflector.go:357] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:232: Watch close - *v1.Pod total 3 items received
5月 07 21:49:33 docker-1 kube-scheduler[32101]: I0507 21:49:33.290127   32101 reflector.go:357] k8s.io/client-go/informers/factory.go:132: Watch close - *v1beta1.PodDisruptionBudget total 0 items received
5月 07 21:50:15 docker-1 kube-scheduler[32101]: I0507 21:50:15.277980   32101 reflector.go:357] k8s.io/client-go/informers/factory.go:132: Watch close - *v1.StorageClass total 0 items received

部署kube-controller-manager

# 创建kube-controller-manager配置文件
vim /home/k8s/kubernetes/cfg/kube-controller-manager 
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect=true \
--address=127.0.0.1 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/home/k8s/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/home/k8s/kubernetes/ssl/ca-key.pem  \
--root-ca-file=/home/k8s/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/home/k8s/kubernetes/ssl/ca-key.pem"

创建kube-controller-manager systemd unit文件

vim /usr/lib/systemd/system/kube-controller-manager.service 
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/home/k8s/kubernetes/cfg/kube-controller-manager
ExecStart=/home/k8s/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动kube-controller-manager服务

systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl restart kube-controller-manager

查看kube-controller-manager是否运行

[root@docker-1 cfg]# systemctl status kube-controller-manager
● kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2019-04-29 21:25:58 CST; 1 weeks 1 days ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 32544 (kube-controller)
   Memory: 76.2M
   CGroup: /system.slice/kube-controller-manager.service
           └─32544 /home/k8s/kubernetes/bin/kube-controller-manager --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect=true --address=127.0.0.1 --service-cluster-ip-range=10.0.0.0/24 --cluster-name=kubernetes --cluster-signing-cert-file=/home/k8s/kuberne...

5月 07 21:56:02 docker-1 kube-controller-manager[32544]: I0507 21:56:02.328943   32544 attach_detach_controller.go:634] processVolumesInUse for node "192.168.1.118"
5月 07 21:56:03 docker-1 kube-controller-manager[32544]: I0507 21:56:03.184628   32544 node_lifecycle_controller.go:929] Node 192.168.1.119 ReadyCondition updated. Updating timestamp.
5月 07 21:56:03 docker-1 kube-controller-manager[32544]: I0507 21:56:03.184673   32544 node_lifecycle_controller.go:929] Node 192.168.1.118 ReadyCondition updated. Updating timestamp.
5月 07 21:56:04 docker-1 kube-controller-manager[32544]: I0507 21:56:04.193087   32544 attach_detach_controller.go:634] processVolumesInUse for node "192.168.1.63"
5月 07 21:56:04 docker-1 kube-controller-manager[32544]: I0507 21:56:04.614869   32544 reflector.go:215] k8s.io/client-go/informers/factory.go:132: forcing resync
5月 07 21:56:04 docker-1 kube-controller-manager[32544]: I0507 21:56:04.614875   32544 reflector.go:215] k8s.io/client-go/informers/factory.go:132: forcing resync
5月 07 21:56:04 docker-1 kube-controller-manager[32544]: I0507 21:56:04.712730   32544 reflector.go:215] k8s.io/client-go/informers/factory.go:132: forcing resync
5月 07 21:56:08 docker-1 kube-controller-manager[32544]: I0507 21:56:08.184898   32544 node_lifecycle_controller.go:929] Node 192.168.1.63 ReadyCondition updated. Updating timestamp.
5月 07 21:56:08 docker-1 kube-controller-manager[32544]: I0507 21:56:08.782878   32544 reflector.go:215] k8s.io/client-go/informers/factory.go:132: forcing resync
5月 07 21:56:08 docker-1 kube-controller-manager[32544]: I0507 21:56:08.793132   32544 pv_controller_base.go:408] resyncing PV controller

[root@docker-1 cfg]# ps -ef |grep kube-controller-manager
root      32544      1  4 4月29 ?       09:30:31 /home/k8s/kubernetes/bin/kube-controller-manager --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect=true --address=127.0.0.1 --service-cluster-ip-range=10.0.0.0/24 --cluster-name=kubernetes --cluster-signing-cert-file=/home/k8s/kubernetes/ssl/ca.pem --cluster-signing-key-file=/home/k8s/kubernetes/ssl/ca-key.pem --root-ca-file=/home/k8s/kubernetes/ssl/ca.pem --service-account-private-key-file=/home/k8s/kubernetes/ssl/ca-key.pem
root     111998 104117  0 22:03 pts/0    00:00:00 grep --color=auto kube-controller-manager

将可执行文件路径/home/k8s/kubernetes/bin/添加到环境变量PATH中

cd /etc/profile.d/
vim k8s.sh
export PATH=/home/k8s/kubernetes/bin:$PATH

source k8s.sh

查看集群master状态

[root@docker-1 cfg]# kubectl get cs,nodes
NAME                                 STATUS    MESSAGE             ERROR
componentstatus/controller-manager   Healthy   ok                  
componentstatus/scheduler            Healthy   ok                  
componentstatus/etcd-0               Healthy   {"health":"true"}   
componentstatus/etcd-2               Healthy   {"health":"true"}   
componentstatus/etcd-1               Healthy   {"health":"true"}

六 部署node1和node2节点

kubernetes work节点运行如下组件：

  • docker ：开头我们已经部署了
  • kubelet
  • kube-proxy

部署kubelet组件

  • kubelet运行在每个work节点上，接收kube-apiserver发送的请求，管理Pod容器，执行交互式命令，如exec、run、logs等
  • kubelet启动时自动向kube-apiserver注册节点信息，内置的cadvisor统计和监控节点的资源使用情况
  • 为确保安全，本文档只开启接收https请求的安全端口，对请求进行认证和授权，拒绝未授权的访问(如apierver、heapster)。

将master节点上kubelet二进制文件拷贝到node1和node2节点上

[root@docker-1 bin]# pwd
/home/k8s/kubernetes/bin
[root@docker-1 bin]# cd /usr/local/src/kubernetes/server/bin 
#master也作为一个节点使用
[root@docker-1 /usr/local/src/kubernetes/server/bin] cp kubelet kube-proxy  /home/k8s/kubernetes/bin
scp kubelet kube-proxy 192.168.1.118:/home/k8s/kubernetes/bin/
scp kubelet kube-proxy 192.168.1.63:/home/k8s/kubernetes/bin

创建kubelet bootstrap kubeconfig文件

[root@docker-1 kubernetes]# pwd
/home/k8s/cfssl/kubernetes
[root@docker-1 kubernetes]#vim environment.sh 
# 创建kubelet bootstrapping kubeconfig 
BOOTSTRAP_TOKEN=b2d81324fb83862f435f66498b90e5bd
KUBE_APISERVER="https://192.168.1.119:6443"
# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=./ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
  --token=${BOOTSTRAP_TOKEN} \
  --kubeconfig=bootstrap.kubeconfig

# 设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

#----------------------

# 创建kube-proxy kubeconfig文件

kubectl config set-cluster kubernetes \
  --certificate-authority=./ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

将生成的bootstrap.kubeconfig 、kube-proxy.kubeconfig两个文件拷贝到所有nodes节点 

[root@docker-1 kubernetes] pwd
/home/k8s/cfssl/kubernetes
[root@docker-1 kubernetes] cp /home/k8s/cfssl/kubernetes/bootstrap.kubeconfig kube-proxy.kubeconfig /home/k8s/kubernetes/cfg/

[root@docker-1 kubernetes] cp /home/k8s/cfssl/kubernetes/bootstrap.kubeconfig kube-proxy.kubeconfig   192.168.1.118:/home/k8s/kubernetes/cfg/

[root@docker-1 kubernetes] cp /home/k8s/cfssl/kubernetes/bootstrap.kubeconfig kube-proxy.kubeconfig   192.168.1.63:/home/k8s/kubernetes/cfg/

继续在master主机上创建kubelet参数配置文件拷贝到所有nodes节点

# 创建kubelet参数配置模板文件
vim /home/k8s/kubernetes/cfg/kubelet.config
apiVersion: kubelet.config.k8s.io/v1beta1
address: 192.168.1.119
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.0.0.2"]
clusterDomain: cluster.local.
failSwapOn: false
authentication:
  anonymous:
    enabled: true

# 创建kubelet配置文件
vim /home/k8s/kubernetes/cfg/kubelet
KUBELET_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.1.119 \
--kubeconfig=/home/k8s/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/home/k8s/kubernetes/cfg/bootstrap.kubeconfig \
--config=/home/k8s/kubernetes/cfg/kubelet.config \
--cert-dir=/home/k8s/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"

# 创建kubelet systemd unit文件
vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=/home/k8s/kubernetes/cfg/kubelet
ExecStart=/home/k8s/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target

在master操作将kubelet-bootstrap用户绑定到系统集群角色

kubectl create clusterrolebinding kubelet-bootstrap \
  --clusterrole=system:node-bootstrapper \
  --user=kubelet-bootstrap

将master上的/home/k8s/kubernetes/cfg/kubelet文件和/usr/lib/systemd/system/kubelet.service拷贝到node1和node2上

[root@docker-1]scp /home/k8s/kubernetes/cfg/kubelet 192.168.1.118:/home/k8s/kubernetes/cfg/kubelet

&& scp /home/k8s/kubernetes/cfg/kubelet

[root@docker-1]scp /usr/lib/systemd/system/kubelet.service 192.168.1.118:/usr/lib/systemd/system/kubelet.service && scp /usr/lib/systemd/system/kubelet.service 192.168.1.63:/usr/lib/systemd/system/kubelet.service

其中/home/k8s/kubernetes/cfg/kubelet文件要修改为对应的IP,node1修改为–hostname-override=192.168.1.118、 node2修改为–hostname-override=192.168.1.63

其中usr/lib/systemd/system/kubelet.service 文件不需要修改

启动三台服务器的kubelet服务

systemctl daemon-reload
systemctl enable kubelet
systemctl restart kubelet

approve kubelet CSR请求

查看CSR列表

# kubectl get csr
NAME                                                   AGE    REQUESTOR           CONDITION
node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs   39m    kubelet-bootstrap   Pending
node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s   5m5s   kubelet-bootstrap   Pending

# kubectl certificate approve node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs
certificatesigningrequest.certificates.k8s.io/node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs 

# kubectl certificate approve node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s  
certificatesigningrequest.certificates.k8s.io/node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s approved
[
# kubectl get csr
NAME                                                   AGE     REQUESTOR           CONDITION
node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs   41m     kubelet-bootstrap   Approved,Issued
node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s   7m32s   kubelet-bootstrap   Approved,Issued
  • Requesting User: 请求CSR的用户，kube-apiserver对它进行认证和授权
  • Subject: 请求签名的证书信息
  • 证书的 CN 是system:node;kube-node2, Organization 是system:nodes,kube-apiserver的Node授权模式会授予该证书的相关权限；

在master上查看集群状态

[root@docker-1 kubernetes]# kubectl get nodes
NAME            STATUS   ROLES    AGE     VERSION
192.168.1.118   Ready    node     7d22h   v1.13.0
192.168.1.119   Ready    master   8d      v1.13.0
192.168.1.63    Ready    node     7d22h   v1.13.0

先在master上部署kube-proxy组件

kube-proxy运行在所有node节点上，它监听apiserver中 service 和Endpoint的变化情况，创建路由规则来进行服务负载均衡。

创建kube-proxy配置文件

[root@docker-1 kubernetes]#vim /home/k8s/kubernetes/cfg/kube-proxy
KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.1.119 \
--cluster-cidr=10.0.0.0/24 \
--kubeconfig=/home/k8s/kubernetes/cfg/kube-proxy.kubeconfig"
  • bindAddress：监听地址；
  • clientConnection.kubeconfig: 连接apiserver的kubeconfig文件；
  • clusterCIDR: kube-proxy根据-cluster-cidr判断集群内部和外部流量指定 –cluster-cidr 或 –masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT；
  • hostnameOverride: 参数值必须与 kubelet 的值一致，否则 kube-proxy 启动后会找不到该 Node，从而不会创建任何 ipvs 规则；
  • mode: 使用ipvs模式；

创建kube-proxy systemd unit文件

[root@docker-1 kubernetes]# vim /home/k8s/kubernetes/cfg/kube-proxy
KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.1.119 \
--cluster-cidr=10.0.0.0/24 \
--kubeconfig=/home/k8s/kubernetes/cfg/kube-proxy.kubeconfig"
[root@docker-1 kubernetes]# cat /usr/lib/systemd/system/kube-proxy.service 
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=-/home/k8s/kubernetes/cfg/kube-proxy
ExecStart=/home/k8s/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

在node1和node2上部署kube-proxy服务

把master上kube-proxy配置文件和kube-proxy systemd unit文件拷贝到node1和node2，其中/home/k8s/kubenetes/cfg/kube-proxy需要修改为相应的IP

最后启动服务

systemctl daemon-reload
systemctl enable kube-proxy
systemctl restart kube-proxy

[root@docker-1 kubernetes]# systemctl status kube-proxy
● kube-proxy.service - Kubernetes Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since 二 2019-04-30 00:46:30 CST; 1 weeks 0 days ago
 Main PID: 51295 (kube-proxy)
   Memory: 11.1M
   CGroup: /system.slice/kube-proxy.service
           ‣ 51295 /home/k8s/kubernetes/bin/kube-proxy --logtostderr=true --v=4 --hostname-override=192.168.1.119 --cluster-cidr=10.0.0.0/24 --kubeconfig=/home/k8s/kubernetes/cfg/kube-proxy.kubeconfig

5月 07 23:10:00 docker-1 kube-proxy[51295]: I0507 23:10:00.815793   51295 proxier.go:664] Syncing iptables rules
5月 07 23:10:00 docker-1 kube-proxy[51295]: I0507 23:10:00.834550   51295 iptables.go:327] running iptables-save [-t filter]
5月 07 23:10:00 docker-1 kube-proxy[51295]: I0507 23:10:00.836358   51295 iptables.go:327] running iptables-save [-t nat]
5月 07 23:10:00 docker-1 kube-proxy[51295]: I0507 23:10:00.839045   51295 proxier.go:1019] Port "nodePort for kube-system/kubernetes-dashboard:" (:49655/tcp) was open before and is still needed
5月 07 23:10:00 docker-1 kube-proxy[51295]: I0507 23:10:00.839157   51295 iptables.go:391] running iptables-restore [--noflush --counters]
5月 07 23:10:00 docker-1 kube-proxy[51295]: I0507 23:10:00.844415   51295 healthcheck.go:235] Not saving endpoints for unknown healthcheck "default/nginx"
5月 07 23:10:00 docker-1 kube-proxy[51295]: I0507 23:10:00.844438   51295 proxier.go:641] syncProxyRules took 28.670563ms
5月 07 23:10:00 docker-1 kube-proxy[51295]: I0507 23:10:00.844451   51295 bounded_frequency_runner.go:221] sync-runner: ran, next possible in 0s, periodic in 30s
5月 07 23:10:01 docker-1 kube-proxy[51295]: I0507 23:10:01.729871   51295 config.go:141] Calling handler.OnEndpointsUpdate
5月 07 23:10:01 docker-1 kube-proxy[51295]: I0507 23:10:01.730171   51295 config.go:141] Calling handler.OnEndpointsUpdate

集群状态

打node或master节点的标签

kubectl label node 192.168.1.119  node-role.kubernetes.io/master='master'
kubectl label node 192.168.1.118  node-role.kubernetes.io/node='node'
kubectl label node 192.168.1.63  node-role.kubernetes.io/node='node'

[root@docker-1 kubernetes]# kubectl get node,cs
NAME                 STATUS   ROLES    AGE     VERSION
node/192.168.1.118   Ready    node     7d22h   v1.13.0
node/192.168.1.119   Ready    master   8d      v1.13.0
node/192.168.1.63    Ready    node     7d22h   v1.13.0

NAME                                 STATUS    MESSAGE             ERROR
componentstatus/scheduler            Healthy   ok                  
componentstatus/controller-manager   Healthy   ok                  
componentstatus/etcd-0               Healthy   {"health":"true"}   
componentstatus/etcd-1               Healthy   {"health":"true"}   
componentstatus/etcd-2               Healthy   {"health":"true"}
文章作者: 阿培
版权声明: 本博客所有文章除特別声明外，均采用 CC BY 4.0 许可协议。转载请注明来源 阿培 !
kubernetes,docker
赏
 上一篇
python-TCP-UDP python-TCP-UDP
Socket通讯及其TCP/IP原语Socket通讯Socket提供了一个通信端口，应用程序在网络上传输/接收的信息都通过Socket接口来实现的。在应用开发中可以像使用文件句柄一样来对Socket句柄进行读写操作。Windows Sock
2019-05-18 python基础
python socket TCP UDP
下一篇 
csrf csrf
Django 的CSRF保护机制什么是CSRFCSRF(Cross Site Request Forgery),跨站点请求伪造。也被称为one-click-attack或者session riding,通常缩写为CSRF或者XSRF，是一种
2018-12-03 python基础
python CSRF
  目录