traefik ingress使用

Traefik介绍

简单的说，ingress就是从kubernetes集群外访问集群的入口，将用户的URL请求转发到不同的service上。Ingress相当于nginx、apache等负载均衡反向代理服务器，其中还包括规则定义，即URL的路由信息。

Traefik是一款开源的反向代理与负载均衡工具。它最大的优点是能够与常见的微服务系统直接整合，实现自动化动态配置。Traefik通过不断地跟 kubernetes API 打交道，实时的感知后端 service、pod 等变化，比如pod，service 增加与减少等；当得到这些变化信息后，Ingress自动更新配置并热重载，达到服务发现的作用。

traefix 整体架构图如下:

Traefik主要特性详解

自动熔断
- 在集群中，当某一个服务大量出现请求错误，或者请求响应时间过久，或者返回500+错误状态码时，我们希望可以主动剔除该服务，也就是不在将请求转发到该服务上，而这一个过程是自动完成，不需要人工执行。Traefik 通过配置很容易就能帮我们实现，Traefik 可以通过定义策略来主动熔断服务。
- NetworkErrorRatio() > 0.5：监测服务错误率达到50%时，熔断
- LatencyAtQuantileMS(50.0) > 50：监测延时大于50ms时，熔断
- ResponseCodeRatio(500, 600, 0, 600) > 0.5：监测返回状态码为[500-600]在[0-600]区间占比超过50%时，熔断
负载均衡策略
- Traefik 提供两种负载均衡策略支持。一种是 wrr（加权轮训调度算法），一种是 drr（动态加权循环调度算法）
- wrr是默认的负载均衡策略，新创建的 service 权重都是一样为1，这样的话，请求会平均分给每个服务，但是这样很多时候会出现资源分配不均衡的问题，比如由于集群中每个机器配置不一样，而且服务消耗不一样，假设 A 资源使用率已经很高，而 B 属于空闲状态，如果还是均摊到每个服务的话，会加重 A 的负荷，这时候因该有一种策略能够主动识别并分担更多流量到 B 才对
- drr 就更加智能，它是一种动态加权轮训调度方式，它会记录一段时间内转发到 A 的请求数，跟转发到 B 的请求数对比，转发数量多，说明处理速度快，响应时间快。如果 A 处理请求速度比 B 快，那么就会调整 A 的权重，接下来的一段时间，就会转发更多请求给 A，相应的 B 的转发就少一些。整个过程都在不断的调整权重，实现请求的合理分配，从而达到资源使用最大化

部署Traefik ingress

创建ingress-rbac.yaml，将用于service account验证。

# cat ingress-rbac.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress
  namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: ingress
subjects:
  - kind: ServiceAccount
    name: ingress
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

创建Depeloyment部署traefik，如文件名为deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: traefik-ingress-lb
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 2 # 增加行
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      terminationGracePeriodSeconds: 60
      hostNetwork: true
      restartPolicy: Always
      serviceAccountName: ingress
      containers:
      - image: traefik
        name: traefik-ingress-lb
        resources:
          limits:
            cpu: 1000m
            memory: 3000Mi
          requests:
            cpu: 500m
            memory: 2000Mi
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: admin
          containerPort: 8580
          hostPort: 8580
        args:
        - --web
        - --web.address=:8580
        - --kubernetes

注意: 我们这里用的是Deploy类型，没有限定该pod运行在哪个主机上。Traefik的端口是8580。

编写Traefik UI的ingress部署文件，如文件名为traefik-ui.yaml

apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system #增加行
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - name: web
    port: 80
    targetPort: 8580
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  rules:
  - host: traefik.xxlaila.io
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-web-ui
          servicePort: web

backend中要配置default namespace中启动的service名字。
path就是URL地址后的路径，如traefik.frontend.io/path，service将会接受path这个路径
host最好使用service-name.filed1.filed2.domain-name这种类似主机名称的命名方式，方便区分服务。

逼逼一下: 目前我所在的公司后端微服务100+，前端60+，如果用传统nginx的local来匹配，估计要写死人，而且对于运维自动化来也不是很好做，再则是出了问题也还要去看一下是哪个应用；我们目前是通过每个服务每一个域名，域名是根据服务名来自动生成，除了几个特定对外公开的是特制的域名，其他的均采用这种机制，当有问题的时候，一下就能判断出那里出问题，很好定位，有域名有特殊配置的时候，也可以单独的进行设置，但是截止目前两年多来，运维拒绝这种特殊需求(有还有，很少，只有那么两三个)

配置完成后就可以启动treafik ingress了

# kubectl apply -f ./
deployment.extensions/traefik-ingress-lb created
serviceaccount/ingress created
clusterrolebinding.rbac.authorization.k8s.io/ingress created
service/traefik-web-ui created
ingress.extensions/traefik-web-ui created

查看是否部署成功

# kubectl get pods -n kube-system | grep traefik

traefik-ingress-lb-5d7f658cfd-4vkjc     1/1     Running   0          29m
traefik-ingress-lb-5d7f658cfd-7sszp     1/1     Running   0          19m

# kubectl get ingress -o wide --all-namespaces 
NAMESPACE     NAME                HOSTS                ADDRESS   PORTS   AGE
kube-system   traefik-web-ui      traefik.xxlaila.io             80      29m

在浏览器绑定hosts域名解析，node的ip地址，在浏览器输入traefik.xxlaila.io即可访问了

左侧蓝色部分列出的是所有的前端(frontends)，右侧绿色部分是所有的后端(backend)。

测试

下面模拟部署一个程序，以Nginx 为例，并使用drr动态轮训加权策略。

nginx-deployment.yaml

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx-pod
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx-pod
    spec:
      containers:
      - name: nginx
        image: nginx:1.15.5
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  annotations:
    traefik.ingress.kubernetes.io/load-balancer-method: drr
spec:
  template:
    metadata:
      labels:
        name: nginx-service
        namespace: default
spec:
  selector:
    app: nginx-pod
  ports:
  - port: 80
    targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: k8s.nginx.com
    http:
      paths:
      - backend:
          serviceName: nginx-service
          servicePort: 80

创建nginx

# kubectl apply -f nginx-deployment.yaml 
deployment.apps/nginx-pod created
service/nginx-service created
ingress.extensions/nginx-ingress created

# kubectl get pods

所有访问这些地址的流量都会发送给172.16.0.180这台主机，就是我们启动traefik的主机。Traefik会解析http请求header里的Host参数将流量转发给Ingress配置里的相应service。

客户端绑定host，浏览器进行访问: http://k8s.nginx.com

在K8s集群节点上访问测试

# curl -x 172.21.16.204 http://k8s.nginx.com
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
[root@k8s ~]# curl -x 172.21.16.204:80 http://k8s.nginx.com
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

ingress配置同域名不同路径代理web应用

很多时候我们不想配置太多的域名来区别应用，使用同域名分路径的方式来区别应用就简洁方便很多。ingress也提供了相关的配置。

假设两个应用tomcat-test1和tomcat-test2。这里可配置域名tomcat.xxlaila.io，通过路径test1、test2来分别代理两个tomcat应用。其中，分路径配置需添加配置：traefik.frontend.rule.type: PathPrefixStrip,首先，我先创建tomcat-test1和tomcat-test2的pod和service，其中8080为tomcat的http端口，8443为tomcat的https端口，本例中仅使用http端口测试。

tomcat-test1.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-test1
  labels: 
    app: tomcat-test1
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: tomcat-test1
  template:
    metadata:
      labels:
        app: tomcat-test1
    spec:
      containers:
      - name: tomcat-test1
        image: tomcat
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8443
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: tomcat-test1
  labels:
    name: tomcat-test1
spec:
  ports:
  - port: 8443
    targetPort: 8443
  selector:
    app: tomcat-test1
  ports:
  - port: 8080 
    targetPort: 8080
  selector:
    app: tomcat-test1

tomcat-test2.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-test2
  labels: 
    app: tomcat-test2
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: tomcat-test2
  template:
    metadata:
      labels:
        app: tomcat-test2
    spec:
      containers:
      - name: tomcat-test2
        image: manjeetchauhan211/tomcat_test2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8443
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: tomcat-test2
  labels:
    name: tomcat-test2
spec:
  ports:
  - port: 8443
    targetPort: 8443
  ports:
  - port: 8080 
    targetPort: 8080
  selector:
    app: tomcat-test2

执行创建

# kubectl apply -f ./

$ kubectl get deployment
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
nginx-pod      2/2     2            2           17h
tomcat-test1   1/1     1            1           42m
tomcat-test2   1/1     1            1           42m

$ kubectl get svc
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
kubernetes       ClusterIP   10.254.0.1       <none>        443/TCP    5d23h
nginx-service    ClusterIP   10.254.149.69    <none>        80/TCP     17h
tomcat-test1     ClusterIP   10.254.195.108   <none>        8080/TCP   42m
tomcat-test2     ClusterIP   10.254.6.88      <none>        8080/TCP   42m
traefik-web-ui   ClusterIP   10.254.22.102    <none>        80/TCP     17h

创建test1的ingress，来发布tomcat-test1服务

ingress-tomcat1.yam

$ cat ingress-tomcat1.yaml 
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tomcat-test1-web
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: tomcat.xxlaila.io
    http:
      paths:
      - path: /
        backend:
          serviceName: tomcat-test1
          servicePort: 8080

# kubectl create -f ingress-tomcat.yaml

在traefix-ui界面上，可以看到已经有了一个tomcat.xxlaila.io的域名规则.

在hosts文件添加tomcat.xxlaila.io绑定来进行访问

ingress配置同域名对应location

ingress-tomcat.yaml

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tomcat-test-web
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.frontend.rule.type: PathPrefixStrip
spec:
  rules:
  - host: tomcat.xxlaila.io
    http:
      paths:
      - path: /test1
        backend:
          serviceName: tomcat-test1
          servicePort: 8080
      - path: /
        backend:
          serviceName: tomcat-test2
          servicePort: 8080

创建并查看

$ kubectl create -f ingress-tomcat.yaml 

$ kubectl describe ingress tomcat-test-web
Name:             tomcat-test-web
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host               Path  Backends
  ----               ----  --------
  tomcat.xxlaila.io  
                     /test1/   tomcat-test1:8080 (<none>)
                     /test2/   tomcat-test2:8080 (<none>)
Annotations:
  traefik.frontend.rule.type:                        PathPrefixStrip
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"traefik","traefik.frontend.rule.type":"PathPrefixStrip"},"name":"tomcat-test-web","namespace":"default"},"spec":{"rules":[{"host":"tomcat.xxlaila.io","http":{"paths":[{"backend":{"serviceName":"tomcat-test1","servicePort":8080},"path":"/test1/"},{"backend":{"serviceName":"tomcat-test2","servicePort":8080},"path":"/test2/"}]}}]}}

  kubernetes.io/ingress.class:  traefik
Events:                         <none>

给节点设置label

由于是 Kubernetes DeamonSet 这种方式部署 Traefik，所以需要提前给节点设置 Label，这样当程序部署时 Pod 会自动调度到设置 Label 的点上。

节点设置 Label 标签

格式：kubectl label nodes [节点名] [key=value]

 kubectl get nodes
NAME            STATUS   ROLES    AGE    VERSION
172.21.16.204   Ready    <none>   7d5h   v1.13.3
172.21.16.240   Ready    <none>   7d2h   v1.13.3
172.21.16.87    Ready    <none>   7d2h   v1.13.3

# kubectl label nodes 172.21.16.204 IngressProxy=true
node/172.21.16.204 labeled

# 查看节点label设置是否成功
# kubectl get nodes --show-labels
NAME            STATUS   ROLES    AGE    VERSION   LABELS
172.21.16.204   Ready    <none>   7d5h   v1.13.3   IngressProxy=true,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=172.21.16.204,node.kubernetes.io/k8s-node=true
172.21.16.240   Ready    <none>   7d2h   v1.13.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=172.21.16.240,node.kubernetes.io/k8s-node=true
172.21.16.87    Ready    <none>   7d2h   v1.13.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=172.21.16.87,node.kubernetes.io/k8s-node=true

修改Traefix部署文件

# cat deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: traefik-ingress-lb
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 2 # 增加行
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      terminationGracePeriodSeconds: 60
      volumes:
      - name: ssl
        secret:
          secretName: traefik-cert
      - name: config
        configMap:
          name: traefik-conf
      hostNetwork: true
      restartPolicy: Always
      serviceAccountName: ingress
      containers:
      - image: traefik
        name: traefik-ingress-lb
        volumeMounts:
        - mountPath: "/etc/kubernetes/certs"
          name: "ssl"
        - mountPath: "/etc/kubernetes/conf"
          name: "config"
        resources:
          limits:
            cpu: 1000m
            memory: 3000Mi
          requests:
            cpu: 500m
            memory: 2000Mi
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: admin
          containerPort: 8580
          hostPort: 8580
        args:
        - --configFile=/etc/kubernetes/conf/traefik.toml
        - --web
        - --web.address=:8580
        - --kubernetes
      nodeSelector:
        IngressProxy: "true"

# 执行部署即可

traefix-ui界面上可以看到

从describe信息和ui界面上可以看到，tomcat.test.k8s分别有了/test1/和/test2/的域名代理以及相对应的后端

参考文献2
参考文献1