k8s部署ingress

在kubernetes 集群中，一个服务安装以后怎么对外提供访问，外部用户怎么来访问我们容器中业务。

1、Ingress 介绍

Kubernetes 暴露服务的方式目前只有三种：LoadBlancer Service、NodePort Service、Ingress；本文主要通过Ingress来访问

2、Ingress 是什么

Ingress 就是能利用 Nginx、Haproxy 啥的负载均衡器暴露集群内服务的工具问题来了，集群内服务想要暴露出去面临着几个问题：

Pod 漂移问题
众所周知 Kubernetes 具有强大的副本控制能力，能保证在任意副本(Pod)挂掉时自动从其他机器启动一个新的，还可以动态扩容等，总之一句话，这个 Pod 可能在任何时刻出现在任何节点上，也可能在任何时刻死在任何节点上；那么自然随着 Pod 的创建和销毁，Pod IP 肯定会动态变化；那么如何把这个动态的 Pod IP 暴露出去？这里借助于 Kubernetes 的 Service 机制，Service 可以以标签的形式选定一组带有指定标签的 Pod，并监控和自动负载他们的 Pod IP，那么我们向外暴露只暴露 Service IP 就行了；这就是 NodePort 模式：即在每个节点上开起一个端口，然后转发到内部 Service IP 上，如下图所示：
端口管理问题
采用 NodePort 方式暴露服务面临一个坑爹的问题是，服务一旦多起来，NodePort 在每个节点上开启的端口会及其庞大，而且难以维护；这时候引出的思考问题是 “能不能使用 Nginx 啥的只监听一个端口，比如 80，然后按照域名向后转发？” 简单的实现就是使用 DaemonSet 在每个 node 上监听 80，然后写好规则，因为 Nginx 外面绑定了宿主机 80 端口(就像 NodePort)，本身又在集群内，那么向后直接转发到相应 Service IP 就行了，如下图所示
域名分配及动态更新问题
    从上面的思路，采用 Nginx 似乎已经解决了问题，但是其实这里面有一个很大缺陷：每次有新服务加入怎么改 Nginx 配置？总不能手动改或者来个 Rolling Update 前端 Nginx Pod 吧？这时候 “伟大而又正直勇敢的” Ingress 登场，如果不算上面的 Nginx，Ingress 只有两大组件：Ingress Controller 和 Ingress
    Ingress 简单的理解就是你原来要改 Nginx 配置，然后配置各种域名对应哪个 Service，现在把这个动作抽象出来，变成一个 Ingress 对象，你可以用 yml 创建，每次不要去改 Nginx 了，直接改 yml 然后创建/更新就行了；那么问题来了：”Nginx 咋整？”
    Ingress Controller 这东西就是解决 “Nginx 咋整” 的；Ingress Controoler 通过与 Kubernetes API 交互，动态的去感知集群中 Ingress 规则变化，然后读取他，按照他自己模板生成一段 Nginx 配置，再写到 Nginx Pod 里，最后 reload 一下，工作流程如下图:

当然在实际应用中，最新版本 Kubernetes 已经将 Nginx 与 Ingress Controller 合并为一个组件，所以 Nginx 无需单独部署，只需要部署 Ingress Controller 即可。

3、Nginx Ingress

3.1、下载官方文件

官方的mandatory.yaml文件里面包含了ingress RBAC，重要的组件 Nginx+Ingres Controller

# cat mandatory.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
     --default-ssl-certificate=$(POD_NAMESPACE)/ingress-secret
      --default-backend-service=$(POD_NAMESPACE)/default-http-backend
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
              hostPort: 80
            - name: https
              containerPort: 443
              hostPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
---

hostNetwork: true是增加的，官方的 Ingress Controller 有个坑，默认注释了hostNetwork 工作方式。以防止端口的在宿主机的冲突。没有绑定到宿主机 80 端口，也就是说前端 Nginx 没有监听宿主机 80 端口；所以需要把配置搞下来自己加一下 hostNetwork。

3.2、部署默认后端

我们知道前端的 Nginx 最终要负载到后端 service 上，那么如果访问不存在的域名咋整？官方给出的建议是部署一个默认后端，对于未知请求全部负载到这个默认后端上；这个后端啥也不干，就是返回 404，部署如下

# cat default-backend.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    k8s-app: default-http-backend
  namespace: ingress-nginx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: docker.io/xxlaila/defaultbackend:1.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
#      nodeSelector:
#        kubernetes.io/hostname: 172.21.16.231
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: ingress-nginx
  labels:
    k8s-app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    k8s-app: default-http-backend

3.3、执行创建,完成后可以看到

1 2	# kubectl create -f mandatory.yaml # kubectl create -f default-backend.yaml

# kubectl get pods -n ingress-nginx
NAME                                       READY   STATUS    RESTARTS   AGE
default-http-backend-66cdcb6c7d-pb9sp      1/1     Running   0          8h
nginx-ingress-controller-69585dbb4-m6fcm   1/1     Running   0          8h
# kubectl get svc -n ingress-nginx
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
default-http-backend   ClusterIP   10.102.60.154   <none>        80/TCP    8h

4、部署 Ingress

从上面可以知道 Ingress 就是个规则，指定哪个域名转发到哪个 Service，所以说首先我们得有个 Service，当然 Service 去哪找这里就不管了；这里默认为已经有了两个可用的 Service，以下以 jenkins、Dashboard 为例
先写一个 Ingress 文件，语法格式啥的请参考官方文档，由于我的 jenkins在kube-ops，Dashboard 在kube-system 这个命名空间，所以要指定 namespace.参考下面实例

4.1、部署jenkins实例

# cat jenkins-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins-ingress
  namespace: kube-ops
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: ci.xxlaila.io
    http:
      paths:
      - backend:
          serviceName: jenkins2
          servicePort: 8080
# kubectl create -f jenkins-ingress.yaml

执行域名解析到ip地址，访问jenkins

4.2、部署kubernetes-dashboard

# cat nginx-kubernetes-dashboard.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kube-system
  annotations:
    nginx.ingress.kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  tls:
  - hosts:
    - k8s.xxlaila.io
    secretName: ingress-secret
  rules:
    - host: dashboard.xxlaila.io
      http:
        paths:
        - path: /
          backend:
            serviceName: kubernetes-dashboard
            servicePort: 443

5、部署 Ingress TLS

上面已经做好了 Ingress，接下来配置TLS ；官方给出的样例很简单，大致步骤就两步：创建一个含有证书的 secret、在 Ingress 开启证书；但是官方的有坑，下面是操作步骤

5.1、创建证书

首先第一步当然要有个证书，由于我这个 Ingress 有两个服务域名，所以证书要支持两个域名；生成证书命令如下：

生成CA证书
1
# mkdir cert && cd cert
编辑 openssl 配置
1
# cp /etc/pki/tls/openssl.cnf .

修改主要配置

1
2
3

# vi openssl.cnf
	[req]
	req_extensions = v3_req # 这行默认注释关着的 把注释删掉

增加配置

# vi openssl.cnf
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = dashboard.mritd.me		#需要增加的域名
DNS.2 = kibana.mritd.me

生成证书

1
2
3

# openssl genrsa -out ingress-key.pem 2048
# openssl req -new -key ingress-key.pem -out ingress.csr -subj "/CN=kube-ingress" -config openssl.cnf
# openssl x509 -req -in ingress.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out ingress.pem -days 365 -extensions v3_req -extfile openssl.cnf

查看生成后的证书

1 2	# ls ca-key.pem ca.pem ca.srl ingress-key.pem ingress.csr ingress.pem openssl.cnf

5.2、创建 secret

创建好证书以后，需要将证书内容放到 secret 中，secret 中全部内容需要 base64 编码，然后注意去掉换行符(变成一行)；以下是我的 secret 样例(上一步中 ingress.pem 是证书crt，ingress-key.pem 是证书的 key)

# vim ingress-secret.yml
apiVersion: v1
data:
  tls.crt: MIIDAjCCAeqgAwIBAgIJAIUNBpFKFrg4MA0GCSqGSIb3DQEBCwUAMBIxEDAOBgNVBAMMB2t1YmUtY2EwHhcNMTkwMTI5MTAzMzQ3WhcNMjAwMTI5MTAzMzQ3WjAXMRUwEwYDVQQDDAxrdWJlLWluZ3Jlc3MwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC0rLcpYHdqVjN84vCJYF2l61F+LYuPRczPNWyo8Rba4XpT6MMMqoGqgmI164r4of2klBEMPZ0dm1mJaYnjb1Zq/qzVUlqaednxfXsr6u8Xm0a6l7ep+Yr+XcRISZC9AjgyqlFtgjzbNJauHkHTy0i+jqV2A4SkVUT2whBqF00WEKC6kQLhw4Ab1XBG5aOK2Jz4TZdP+Mw4n3AsihycHgjhFvhGNixKl4mpfHfLvFeKxmBa8ZoWT+3AGgkX186EXhdhsfdYqHeLT2TvwsqbUJI8E8F7nleo5VMifr9KGHxaCJ+ynr/WFX1c+0wYAGmSKsw4CnXh4EE+IvaeQjBz8O2HAgMBAAGjVjBUMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgXgMDoGA1UdEQQzMDGCFmNpLnp4Yy5raW5neHVubGlhbi5jb22CF2s4cy56eGMua2luZ3h1bmxpYW4uY29tMA0GCSqGSIb3DQEBCwUAA4IBAQCShThVXg3Fnkrm82sHowxCEc9UG9uzOY2LbxhVN7mcm8U0cXy3acAXdKWLHUwdZnOxNJytpaBBWb/6KFFKrIekaSK+tSD+oRISJf43c1tbt0QEpplUaDagQ35NANyQY2VHQntDdVK4/NNJULbHNEqsu19vDDvmDFi0aLHvqPFAvlGnEBPgO1Ac297pDR2thHyMCGBzRKTQOJy2q3HKexa1pItHjAVmv/k71HBeTJ1en2tFHLUlR0kEhYYPBeRclVZ1oYWn7THRaBW8NtugdM6mxVNFAWQTq5goP0VcW44lWYiUF6mX/UZa2c+5FwsGWdSwbTqmZj7ptA2QcveUcN2/
  tls.key: MIIEowIBAAKCAQEAtKy3KWB3alYzfOLwiWBdpetRfi2Lj0XMzzVsqPEW2uF6U+jDDKqBqoJiNeuK+KH9pJQRDD2dHZtZiWmJ429Wav6s1VJamnnZ8X17K+rvF5tGupe3qfmK/l3ESEmQvQI4MqpRbYI82zSWrh5B08tIvo6ldgOEpFVE9sIQahdNFhCgupEC4cOAG9VwRuWjitic+E2XT/jMOJ9wLIocnB4I4Rb4RjYsSpeJqXx3y7xXisZgWvGaFk/twBoJF9fOhF4XYbH3WKh3i09k78LKm1CSPBPBe55XqOVTIn6/Shh8Wgifsp6/1hV9XPtMGABpkirMOAp14eBBPiL2nkIwc/DthwIDAQABAoIBACLj97sV1fnDC85iRPFCmtMfzmz/fqP8ZsDdIE6/wBok0OrDWGdpxgCXjT+8bOn23nSZ43DptR2ykmfm6anyJk4jQF0xui16uovYH6ErjWCRq+b8xYsdlanpka4kBr95XkDqgy8Sp43taevWDABKkZG7Gljf9Q2HKfo9H85dEZXg7RKKz77wahcHpZofthNo0s5kkH2ckVpjwh5svM+M/gm8KhZkKayndr1ezEAmndT+K/P4EVuYbpPQyk5A6E1G7Dh9M2TczZZa6oiR5etloDk2sOIYxysIZqAblyMN2IrIO0hf4nGqSmfe0TZMCKzSXoVOV2Qe+ckJ+mSyOPcdbIECgYEA3EMWrrr4oc9R8o35mQ2FDCG0FUyXrCo9SN/CyhScdeUx1IRV00CBwL340H1CwzEpj5g4bj6LGGSeEw2g8qfPoPdzr1yxNnCv43dhaGrWRrSxGL6kGfKmIlbgHTPIuzRkAXGWiCgo5JnzPjQtjmpUTmwmxXwqDih8kpRsBSTsmx0CgYEA0f1ORGtmrj/bRBQU0NF+ztTVFJQbPV3lTHl21Qgd8QcEIcOHqzpI2fmkeGZdyYQXIfODA/X+PFng+Z6J5ubtvplN5jPzpQuQRtIZ47NRmtgn4DoH+GAqxIb2hFbwXpXuSR/LelFtzxb91nGiVKpdizvuwnitgOuAIbgGHYbgpfMCgYAvvA5fYb/eeWq+EUzFgauS3H8FmqrIMgNEFtJFL0BVQI2TC/b5qGI2XjVdIbhlSvNB3nBkXAOTDsM/R9XYoMubi+UzXPg+3x8PQeEHWxgDDMfQoAg6Y17j1EYPrhhTkeAWfAJukZ2DJWYU1gQFeD+7Gy8v31/R365XqfjbCIyKdQKBgD+/3tr2oB2WVUK9tfQPJag1BNtSe1KOBubImULjS/O4ZZC6g51//E3wc/X5Xc+nwj4UZ1n0fFJmFt6xOrxWryaF9BhG/VjFwe8+KY3vCn8v0CtKctD8oP842e4jVqXgbo7UkDl6LxQHrthDdzys2+lBMKLpcAMLe8LA01pzcA/xAoGBAK3KXGXz0AsM+5FBbUFVZFOROUQLcd+N8esHnSqD8i7/J0PJsxm9irutuLht0cYK7Fcq65fYmZ4lrJO7bpiBRzONR78lMgm/rcKfCCNr4o4n6almvuFxA+jGgEM2W7iYb9pW+FFOqIOjl0cSHy2hsN7seGv3JBcz5StRjtwuSWf6
kind: Secret
metadata:
  name: ingress-secret
  namespace: kube-system
type: Opaque

创建完成后 create

1 2	# kubectl create -f ingress-secret.yml secret/ingress-secret created

5.3、快速创建

5.2步骤可以简化创建，可以执行一条命令进行创建，

1	# kubectl create secret tls ingress-secret --key cert/ingress-key.pem --cert cert/ingress.pem

5.4、重新部署 Ingress

在tls生成完成后，需要重新部署Ingress，让Ingress能够家在tls。修改配置文件

5.4.1、jenkins tls

# vi jenkins-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins-ingress
  namespace: kube-ops
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - ci.xxlaila.io
    secretName: ingress-secret
  rules:
  - host: ci.xxlaila.io
    http:
      paths:
      - backend:
          serviceName: jenkins2
          servicePort: 8080
# kubect create -f jenkins-ingress.yaml

访问jenkins域名，这里输入http访问会强制跳转到https

5.4.2、kubernetes dashboard tls

# vim nginx-kubernetes-dashboard.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - k8s.xxlaila.io
    secretName: ingress-secret
  rules:
  - host: k8s.xxlaila.io
    http:
      paths:
      - backend:
          serviceName: kubernetes-dashboard
          servicePort: 80
# kubectl create -f nginx-kubernetes-dashboard.yaml

6、ingress 高级用法

lvs 反向代理到物理nginx。完成https拆包，继承nginx所有功能
nginx 反向代理到ingress-control。 ingress-control 有两种部署方式。
- ingress-control 使用nodePort 方式暴漏服务
- ingress-control 使用hostNetwork 方式暴漏服务

7、总结分析

ingress-control 在自己的所属的namespace=ingress, 是可以夸不同namespace提供反向代理服.
如果需要提供夸NS 访问ingress，先给 ingress-control创建RBAC
ingress-control 使用hostnetwork 模式性能比使用service nodePort 性能好很多。因为hostnetwork 是直接获取pod 的IP？