kubernetes node节点安装

发表于   |   更新于 | 分类于 | | 阅读次数: | 总览

More: master节点安装请参考

1、部署kubernetes node节点

Kubernetes node节点包含如下组件：

  • Flanneld: 之前单机节点安装没有配置TLS，现在需要在service配置文件中增加TLS配置
  • Docker: version 18.06.2-ce
  • kubelet
  • kube-proxy
1
2
3
4
5
# ls /etc/kubernetes/
bootstrap.kubeconfig  kubelet   kube-proxy.kubeconfig  proxy  ssl
# ls /etc/kubernetes/ssl
admin-key.pem  kube-apiserver-key.pem  kube-controller-manager-key.pem  kubelet-api-admin-key.pem   kube-proxy-key.pem  kubernetes-ca-key.pem  kube-scheduler-key.pem
admin.pem      kube-apiserver.pem      kube-controller-manager.pem      kubelet-api-admin.pem       kube-proxy.pem      kubernetes-ca.pem      kube-scheduler.pem

增加docker 源

1
2
3
# yum-config-manager \
  --add-repo \
  https://download.docker.com/linux/centos/docker-ce.repo

  • 根据实际查找当前版本 (可选)

    1
    # yum list docker-ce --showduplicates | sort -r

  • 如果确定了版本,直接安装,如果要装17。03直接修改下面数字即可

    1
    # yum -y install docker-ce-18.06.2.ce-3.el7  # 主意版本填写包名的格式.

  • 启docker服务,和开机启动

    1
    # systemctl start docker && systemctl enable docker

1.1、安装flanneld

1
2
3
# mv kubernetes  /etc/ && chown -R root: /etc/kubernetes
# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
# tar zxf flannel-v0.11.0-linux-amd64.tar.gz && mv flanneld mk-docker-opts.sh /usr/bin/ && rm -rf flannel-v0.11.0-linux-amd64.tar.gz

1.1.1、flanneld启动配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# cat /lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/flanneld
ExecStart=/usr/bin/flanneld -etcd-endpoints=${FLANNEL_ETCD} $FLANNEL_OPTIONS
ExecStartPost=/usr/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service

1.1.2、flanneld配置文件

flanneld 配置文件连接了etcd，而在配置etcd的时候需要证书，所以记的吧证书copy到node节点上去

1
2
3
4
5
6
7
8
9
# cat /etc/sysconfig/flanneld
# Flanneld configuration options
# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD="https://172.21.17.4:2379,https://172.21.16.230:2379,https://172.21.16.240:2379"
# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_PREFIX="/coreos.com/network"
# Any additional options that you want to pass
FLANNEL_OPTIONS="-etcd-cafile=/etc/etcd/ssl/etcd-ca.pem -etcd-certfile=/etc/etcd/ssl/etcd.pem -etcd-keyfile=/etc/etcd/ssl/etcd-key.pem"
  • 在启动flanneld之前，需要在etcd中添加一条网络配置记录，这个配置将用于flanneld分配给每个docker的虚拟ip地址段,
  • 在任意一台master执行
    1
    2
    3
    4
    # etcdctl set /coreos.com/network/config '{ "Network": "10.254.0.0/16" }'
    { "Network": "10.254.0.0/16" }
    # etcdctl get  /coreos.com/network/config 
    { "Network": "10.254.0.0/16" }

    在执行的时候我们需要创建一个配置文件，因为前面etcd是启用了https的，否则的话，会报Error: client: etcd cluster is unavailable or misconfigured; error #0: x509: certificate signed by unknown authority的错误。

1
2
3
4
5
# etcd.rc
export ETCDCTL_ENDPOINT=https://172.21.17.4:2379,https://172.21.16.230:2379,https://172.21.16.240:2379
export ETCDCTL_CERT_FILE=/etc/etcd/ssl/etcd.pem
export ETCDCTL_KEY_FILE=/etc/etcd/ssl/etcd-key.pem
export ETCDCTL_CA_FILE=/etc/etcd/ssl/etcd-ca.pem

1.1.3、启动flanneld

在启动flanneld之前，我们需要修改docker的配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# cat /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target
# systemctl daemon-reload
# systemctl enable flanneld &&systemctl start flanneld &&systemctl status flanneld

重启了docker和flanneld以后，我们在任意一台node节点上通过ip add s可以查看。flanneld 和docker 网络绑定的情况

1
# ip add s

2、安装和配置kubelet

    kubelet启动时向kube-apiserver发送tls bootstrapping请求，需要将bootstrap token文件中kube-bootsrap用户授予system:node-bootstrapper cluster角色（role），然后kubelet才能有权限创建认证请求（certificate signing requests）

2.1、安装kubelet

1
2
# wget https://dl.k8s.io/v1.13.3/kubernetes-server-linux-amd64.tar.gz
# tar -xzf kubernetes-server-linux-amd64.tar.gz &&cp -r ./kubernetes/server/bin/{kube-proxy,kubelet} /usr/bin/ && rm -rf ./kubernetes*

2.2、创建kubelet启动文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# cat /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/bin/kubelet \
           $KUBE_LOGTOSTDERR \
           $KUBE_LOG_LEVEL \
           $KUBELET_API_SERVER \
           $KUBELET_ADDRESS \
           $KUBELET_PORT \
           $KUBELET_HOSTNAME \
           $KUBE_ALLOW_PRIV \
           $KUBELET_ARGS
Restart=on-failure
[Install]
WantedBy=multi-user.target

2.3、kubelet配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
# cat /etc/kubernetes/kubelet
###
# kubernetes kubelet (minion) config

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--node-ip={node_ip}"

# The port for the info server to serve on
# KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override={node_ip}"

# location of the api-server
# KUBELET_API_SERVER=""

# Add your own!
KUBELET_ARGS="  --address=0.0.0.0 \
                --allow-privileged \
                --anonymous-auth=false \
                --authorization-mode=Webhook \
                --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \
                --client-ca-file=/etc/kubernetes/ssl/kubernetes-ca.pem \
                --cgroup-driver=cgroupfs \
                --cert-dir=/etc/kubernetes/ssl \
                --cluster-dns=10.254.0.2 \
                --cluster-domain=cluster.local \
                --eviction-soft=imagefs.available<15%,memory.available<512Mi,nodefs.available<15%,nodefs.inodesFree<10% \
                --eviction-soft-grace-period=imagefs.available=3m,memory.available=1m,nodefs.available=3m,nodefs.inodesFree=1m \
                --eviction-hard=imagefs.available<10%,memory.available<256Mi,nodefs.available<10%,nodefs.inodesFree<5% \
                --eviction-max-pod-grace-period=30 \
                --image-gc-high-threshold=80 \
                --image-gc-low-threshold=70 \
                --image-pull-progress-deadline=30s \
                --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
                --max-pods=100 \
                --minimum-image-ttl-duration=720h0m0s \
                --node-labels=node.kubernetes.io/k8s-node=true \
                --pod-infra-container-image=docker.io/kubernetes/pause:latest \
                --port=10250 \
                --read-only-port=0 \
                --rotate-certificates \
                --rotate-server-certificates \
                --fail-swap-on=false \
                --v=2"

2.4、启动kubelet

1
2
3
4
# mkdir /var/lib/kubelet -p
# systemctl daemon-reload
# systemctl enable kubelet &&systemctl start kubelet && systemctl status kubelet
# journalctl -fxeu kubelet

3、通过kubelet的tls请求

kubelet首次启动时像kube-apiserver发送证书签名请求，必须通过后kubernetes系统才会将该node加入集群：

3.1、查看未授权csr请求

  • 任意master节点均可
1
2
3
4
5
6
# kubectl get csr
NAME                                                   AGE   REQUESTOR                   CONDITION
csr-kxfql                                              78m   system:node:172.21.16.204   Pending
node-csr-QptfMgAu2y4GmUZX1Ph9B0XomA0Rg-fxcgs0Yzd-XRU   79m   system:bootstrap:ff90fd     Approved,Issued
# kubectl get nodes
No resources found.

3.2、通过csr请求

1
2
# kubectl certificate approve csr-kxfql
certificatesigningrequest.certificates.k8s.io/csr-kxfql approved
  • 自动生成kubelet kubeconfig文件和公私钥,新版本 kubelet server 的证书自动签发已经被关闭,所以对于 kubelet server 的证书仍需要手动签署

4、配置kube-proxy

4.1、kupe-proxy 启动文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# cat /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/proxy
ExecStart=/usr/bin/kube-proxy \
       $KUBE_LOGTOSTDERR \
       $KUBE_LOG_LEVEL \
       $KUBE_MASTER \
       $KUBE_PROXY_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

4.2、kube-proxy配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# cat /etc/kubernetes/proxy
###
# kubernetes proxy config
# default config should be adequate
# Add your own!
KUBE_PROXY_ARGS="   --bind-address=0.0.0.0 \
                    --cleanup-ipvs=true \
                    --cluster-cidr=10.254.0.0/16 \
                    --hostname-override=docker4.node \
                    --healthz-bind-address=0.0.0.0 \
                    --healthz-port=10256 \
                    --masquerade-all=true \
                    --proxy-mode=ipvs \
                    --ipvs-min-sync-period=5s \
                    --ipvs-sync-period=5s \
                    --ipvs-scheduler=wrr \
                    --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
                    --logtostderr=true \
                    --v=2"

4.3、启动kube-proxy

1
2
# systemctl daemon-reload
# systemctl enable kube-proxy && systemctl start kube-proxy && systemctl status kube-proxy

4.4 在kube-proxy和kubelet启动之前

由于 kubelet 组件是采用 TLS Bootstrap 启动，所以需要预先创建相关配置

  • 创建用于 tls bootstrap 的 token secret

    master节点操作

    1
    # kubectl create -f bootstrap.secret.yaml

为了能让 kubelet 实现自动更新证书，需要配置相关 clusterrolebinding

  • 允许 kubelet tls bootstrap 创建 csr 请求

    1
    2
    3
    kubectl create clusterrolebinding create-csrs-for-bootstrapping \
        --clusterrole=system:node-bootstrapper \
        --group=system:bootstrappers

  • 自动批准 system:bootstrappers 组用户 TLS bootstrapping 首次申请证书的 CSR 请求

    1
    2
    3
    kubectl create clusterrolebinding auto-approve-csrs-for-group \
        --clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient \
        --group=system:bootstrappers

  • 自动批准 system:nodes 组用户更新 kubelet 自身与 apiserver 通讯证书的 CSR 请求

    1
    2
    3
    kubectl create clusterrolebinding auto-approve-renewals-for-nodes \
        --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient \
        --group=system:nodes

  • 在 kubelet server 开启 api 认证的情况下，apiserver 反向访问 kubelet 10250 需要此授权(eg: kubectl logs)

    1
    2
    3
    kubectl create clusterrolebinding system:kubelet-api-admin \
        --clusterrole=system:kubelet-api-admin \
        --user=system:kubelet-api-admin
  • 问题:
    在启动kubelet的时候，node节点在master节点无法查看，查看kubelet的日志提示如下：
  • 查看kubelet的日志方式有两种
1
2
# journalctl -f -u kubelet
# systemctl  status kubelet -l
  • 查看nodes
    1
    2
    3
    # kubectl get nodes
    NAME            STATUS   ROLES    AGE   VERSION
    172.21.16.244   Ready    <none>   12m   v1.13.3

  • 验证测试集群
    创建一个nginx测试集群是否可用

    1
    2
    3
    4
    5
    # kubectl run nginx --image=docker.io/nginx:latest --replicas=2 --labels run=nginx
    kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
    deployment.apps/nginx created
    # kubectl expose deployment nginx --port=80 --type=NodePort
    service/nginx exposed

  • 查看pod情况

1
2
3
4
# kubectl get pod -o wide
NAME                     READY   STATUS              RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES
nginx-766994fc9f-gcv4n   0/1     ContainerCreating   0          55s   <none>        172.21.16.248   <none>           <none>
nginx-766994fc9f-w2j8p   1/1     Running             0          55s   10.254.45.2   172.21.16.83    <none>           <none>
  • 查看对外的服务
1
2
3
# kubectl get svc nginx
NAME    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
nginx   NodePort   10.254.11.147   <none>        80:48713/TCP   31s

部署完成后，通过任意node节点IP的地址加端口48713即可访问
http://node-ip:48713/

坚持原创技术分享，您的支持将鼓励我继续创作！
0%