Error message here!

Hide Error message here!

忘记密码?

Error message here!

请输入正确邮箱

Hide Error message here!

密码丢失?请输入您的电子邮件地址。您将收到一个重设密码链接。

Error message here!

返回登录

Close

使用minukube部署kubernetes admission webhook实现etcd pod安全删除

charlieroro 2019-02-25 21:54:00 阅读数:314 评论数:0 点赞数:0 收藏数:0

本需求来自于一道面试题(本环境使用centos 7)

最好使用阿里云ec2服务器安装minikube,若使用本地pc的vmware可能会出现网络方面的问题。

使用如下命令安装minikube,参见install minikube

# curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube
# sudo cp minikube /usr/local/bin && rm minikube

启动minikube可能会遇到docker版本过旧导致启动失败的问题。默认centos下面yum安装的docker版本比较旧,需要安装最新docker,更新docker参见Get Docker CE for CentOS

如下命令移除已经安装的docker

yum remove docker \
docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-engine

如下命令安装存储驱动和设置docker stable版本的库

yum install -y yum-utils \
device-mapper-persistent-data \ lvm2 yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo

使用如下命令即可更新为最新的docker

yum install docker-ce docker-ce-cli containerd.io

 

管理kubernetes上的服务最好使用helm(本次未用到,如无需要可忽略本节),helm安装如下:

使用如下方式获取helm的二进制版本

Download your desired version
Unpack it (tar -zxvf helm-v2.0.0-linux-amd64.tgz) Find the helm binary in the unpacked directory, and move it to its desired destination (mv linux-amd64/helm /usr/local/bin/helm)

 tiller安装给出了在不同scope下面安装tiller的方法,最简单的是在cluster-admin下面安装即可(生成环境下建议参考helm安装文档将权限最小化)

# cat rbac-config.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: tiller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: tiller namespace: kube-system

使用如下命令创建即可

kubectl create -f rbac-config.yaml
serviceaccount "tiller" created clusterrolebinding "tiller" created $ helm init --service-account tiller

admission webhook原理

kubernetes的认证和授权是对客户端进行认证以及对资源进行授权,但在资源的使用处理上不够细化。admission webhook是在一种在改变资源的持久化之前(比如某些资源的创建或删除,修改等之前)的机制。参见官方资料

如下图,在部署了admission webhook之后,apiserver会发送一个AdmissionReview的json数据到webhook,其中主要包含一个AdmissionRequest的请求,AdmissionRequest.RawExtension.Raw包含了需要处理的kubernetes组件(如pod,deployment等)的详细信息。webhook在接收到该请求之后会根据自定义逻辑进行处理,并返回处理结果AdmissionResponse。admission webhook有两种处理方式:

MutatingAdmissionWebhook:可以修改自定义的策略,MutatingAdmissionWebhook的处理一般优先于MutatingAdmissionWebhook,这样前者修改的内容就可以由后者进行校验

ValidatingAdmissionWebhook: 允许或拒绝客户自定义的策略

type AdmissionReview struct {
metav1.TypeMeta `json:",inline"` // Request describes the attributes for the admission request. // +optional Request *AdmissionRequest `json:"request,omitempty" protobuf:"bytes,1,opt,name=request"` // Response describes the attributes for the admission response. // +optional Response *AdmissionResponse `json:"response,omitempty" protobuf:"bytes,2,opt,name=response"` }

 

 处理流程图如下,可以看到MutatingAdmissionWebhook的处理优先于MutatingAdmissionWebhook

 

使用如下命令启动minikube

minikube start --vm-driver=none --extra-config=apiserver.enable-admission-plugins="NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota"

 使用kubectl api-versions查看是否支持admissionregistration.k8s.io/v1alpha1 API

In-depth introduction to Kubernetes admission webhooks是个很好的例子,里面详细记录了创建admission webhook的方式。这里参照In-depth introduction to Kubernetes admission webhooks实现了根据etcd pod角色(leader和非leader)来删除pod。

首先使用webhook-create-signed-cert.sh文件来生成自签证书,后续ValidatingWebhookConfiguration中会用到

# ./deployment/webhook-create-signed-cert.sh

部署admission webhook

# kubectl create -f deployment/deployment.yaml # kubectl create -f deployment/service.yaml

生成待CA bundle的config文件以及validatingwebhook.yaml原始文件如下

# cat ./deployment/validatingwebhook.yaml | ./deployment/webhook-patch-ca-bundle.sh > ./deployment/validatingwebhook-ca-bundle.yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
name: validation-webhook-example-cfg labels: app: admission-webhook-example webhooks: - name: required-labels.banzaicloud.com clientConfig: service: name: admission-webhook-example-svc #service名称 namespace: default path: "/validate" #访问的后缀路径 caBundle: ${CA_BUNDLE} #上述命令生成的认证字段 rules: - operations: [ "DELETE" ] #操作的动作 apiGroups: ["apps", ""] #api groups apiVersions: ["v1"] #api version resources: ["pods"] #操作的资源 namespaceSelector: matchLabels: admission-webhook-example: enabled # 限制default的命名空间

给default namespace打上标签

$ kubectl label namespace default admission-webhook-example=enabled

创建ValidatingWebhookConfiguration

$ kubectl create -f deployment/validatingwebhook-ca-bundle.yaml

admission-webhook-example/build用于编译和生成容器镜像

etcd的部署直接使用admission-webhook-example\deployment\etcd中的配置文件即可,这样在删除etcd的leader时会显示如下内容

admission webhook的实现只有2个文件,main.go和webhook.go。main.go中通过mux.HandleFunc("/validate", whsvr.serve)来启动一个http服务处理路径/validate(对应validatingwebhook.yaml的webhooks.clientConfig.service.path)的请求。主要逻辑是现在函数func (whsvr *WebhookServer) validate(ar *v1beta1.AdmissionReview) *v1beta1.AdmissionRespons中

 

FAQ:

  • 执行yum install -y socat可解决如下问题:
an error occurred forwarding 40546 -> 44134: error forwarding port 44134 to pod 3ea221f842e1446a5fd9da9fc29e7e415a1ecb6dd6d07c56f64fb4788f2c3915, uid : unable to do port forwarding: socat not found.

 

TIPS:

  •  使用如下方式可以安装etcdctl命令行工具
# choose either URL
GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/coreos/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL} rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1 rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz /tmp/etcd-download-test/etcd --version ETCDCTL_API=3 /tmp/etcd-download-test/etcdctl version # start a local etcd server /tmp/etcd-download-test/etcd # write,read to etcd ETCDCTL_API=3 /tmp/etcd-download-test/etcdctl --endpoints=localhost:2379 put foo bar ETCDCTL_API=3 /tmp/etcd-download-test/etcdctl --endpoints=localhost:2379 get foo
  • go get -u github.com/golang/dep/cmd/dep 安装dep
  • etcdv3的API使用可以参考package clientv3

 

参考:

https://github.com/helm/helm/blob/master/docs/install.md

https://container-solutions.com/some-admission-webhook-basics/

https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/

https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/statefulset/etcd

https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/medium-article.md

版权声明
本文为[charlieroro]所创,转载请带上原文链接,感谢
https://www.cnblogs.com/charlieroro/p/10434138.html

编程之旅,人生之路,不止于编程,还有诗和远方。
阅代码原理,看框架知识,学企业实践;
赏诗词,读日记,踏人生之路,观世界之行;

支付宝红包,每日可领