# kubernetes安装(containerd) ## 概念介绍 - cri (Container runtime interface) - - cri is a [containerd](https://links.jianshu.com/go?to=https%3A%2F%2Fcontainerd.io%2F) plugin implementation of Kubernetes [container runtime interface (CRI)](https://links.jianshu.com/go?to=https%3A%2F%2Fgithub.com%2Fkubernetes%2Fkubernetes%2Fblob%2Fmaster%2Fpkg%2Fkubelet%2Fapis%2Fcri%2Fruntime%2Fv1alpha2%2Fapi.proto). - cri是 kubernetes的容器运行时接口的容器插件实现。 - containerd - - containerd is an industry-standard container runtime with an emphasis on simplicity, robustness and portability. - containerd完全支持运行容器的的CRI运行时规范。 - cri在containerd1.1以上的版本的原生插件。它内置于containerd并默认启用。 - cri-o - - OCI-based implementation of Kubernetes Container Runtime Interface. - kubernetes为了兼容cri和oci孵化了项目cri-o。为了架设在cri和oci之间的一座桥梁。由此cri-o既兼容cri插件实现又兼容oci的容器运行时标准。 - oci (Open Container Initiative) - - [https://www.opencontainers.org/about/oci-scope-table](https://links.jianshu.com/go?to=https%3A%2F%2Fwww.opencontainers.org%2Fabout%2Foci-scope-table) - oci是由多家公司成立的项目,并由linux基金会进行管理,致力于container runtime 的标准的制定和runc的开发等工作。 - runc - - runc is a CLI tool for spawning and running containers according to the OCI specification. - runc，是对于OCI标准的一个参考实现，是一个可以用于创建和运行容器的CLI(command-line interface)工具。 直接通过调用containerd来管理容器的α新特性。 ![img](/images/posts/706130851746.png) 从上图中可以明显的看出来绕过docker，直接通过containerd管理容器，可以节省一个操作步骤，那么这样会带来很多好处： 1、 由于减少了一个操作步骤，提高了系统效率，并且系统更易于维护。 2、 取消了DockerDaemon，整个系统占用的资源更少了，可以给业务使用的资源变多了。 3、 不依赖DockerDaemon，整个系统更加开放。 4、 由于kubelet并没有变化，那么不影响以前用户使用。 ## 系统初始化 请参考dockershim运行时安装的初始化文章 ## kubernetes部署安装 ### kube-proxy开启ipvs的前置条件 ``` modprobe br_netfilter cat > /etc/sysconfig/modules/ipvs.modules < /etc/containerd/config.toml [root@drbd2 ~]# grep image /etc/containerd/config.toml sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.1" 注意将sandbox镜像改为阿里云的地址 ``` ### 配置私有仓库 配置文件参考信息： https://github.com/containerd/cri/blob/release/1.4/docs/registry.md ``` /etc/containerd/config.toml [plugins."io.containerd.grpc.v1.cri".registry] [plugins."io.containerd.grpc.v1.cri".registry.mirrors] #默认仓库为docker.io [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] endpoint = ["https://registry-1.docker.io"] #新增的私有仓库，注意需要在hosts添加解析地址 [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.gtmsh.com"] endpoint = ["https://harbor.gtmsh.com"] # 私有仓库配置 [plugins."io.containerd.grpc.v1.cri".registry.configs] # harbor.gtmsh.com仓库的tls配置，insecure_skip_verify = true 为跳过tls配置 [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.gtmsh.com".tls] insecure_skip_verify = true #私有仓库认证信息 [plugins."io.containerd.grpc.v1.cri".registry.auths] #harbor.gtmsh.com私有仓库认证信息，注意这只是在crictl pull的时候会使用到的认证，k8s需要配置secret [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.gtmsh.com".auth] username = "admin" password = "Harbor12345" systemctl restart containerd ``` ### 安装kubeadm等软件 ``` cat < /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF yum -y install kubeadm-1.15.1 kubelet-1.15.1 kubectl-1.15.1 systemctl enable kubelet ``` ### 使用crictl连接containerd 下一步我们使用crictl连接containerd。 - 修改crictl的配置文件,在 /etc/crictl.yaml 写入以下内容： ``` runtime-endpoint: unix:///run/containerd/containerd.sock image-endpoint: unix:///run/containerd/containerd.sock timeout: 10 debug: false ``` 这里注意runtime-endpoint 和image-endpoint 必须与/etc/containerd/config.toml中配置保持一致。 - 验证一下cri插件是否可用 ``` 可以使用私有仓库 crictl pull nginx:alpine crictl rmi nginx:alpine crictl images 使用ctr命令导入镜像。不能使用私有仓库 ctr image import app.tar #导入本地镜像 ctr images list|grep app #查看导入的镜像 crictl images list|grep app #此命令也可查看 cri导入镜像命令(cri导入镜像)，可能已经废弃： ctr cri load images.tar containerd导入镜像命令(containerd导入镜像)： ctr images import images.tar ``` 其中 crictl images 会列出所有的cri容器镜像。 到此我们的cri + containerd已经完成整合了。 ### 修改kubeadm配置进行部署 ``` kubeadm config print init-defaults > kubeadm-config.yaml [root@drbd2 ~]# cat kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.116.101 #这里需要修改 bindPort: 6443 nodeRegistration: criSocket: /run/containerd/containerd.sock #这里需要修改 name: drbd2 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: registry.aliyuncs.com/google_containers #这里需要修改 kind: ClusterConfiguration kubernetesVersion: v1.15.1 networking: dnsDomain: cluster.local podSubnet: 10.244.0.0/16 #这里需要添加 serviceSubnet: 10.96.0.0/12 --- #下面都需要添加 apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration featureGates: SupportIPVSProxyMode: true mode: ipvs ``` #### 修改kubelet配置和kubeadm安装时配置 在 kubelet配置文件 10-kubeadm.conf 的[Service] 结点加入以下配置： ``` [root@drbd2 ~]# cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf # Note: This dropin only works with kubeadm and kubelet v1.11+ [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" #下面一行需要添加 Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock" # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file. EnvironmentFile=-/etc/sysconfig/kubelet ExecStart= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS ``` ``` 由于安装containerd会安装3.10内核，这里重选选择下内核 grub2-set-default "CentOS Linux (4.4.238-1.el7.elrepo.x86_64) 7 (Core)" && reboot ``` ### 初始化集群 ``` kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs | tee kubeadm-init.log ``` ### 部署网络 ``` kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs | tee kubeadm-init.log ``` ### 部署后注意事项 参考dockershim运行时安装的文章 ### 测试 自定制python + flask + uwsgi镜像python:v2，以及数据镜像 test:v2,上传到私有仓库 的公共仓库（不需要账号密码的） #### 创建deploy资源 ``` [root@con-mas ~]# cat hello-dp.yaml apiVersion: apps/v1 kind: Deployment metadata: labels: app: helloworld name: helloworld spec: replicas: 2 selector: matchLabels: app: helloworld template: metadata: labels: app: helloworld spec: initContainers: - name: code image: harbor.gtmsh.com/library/test:v2 volumeMounts: - name: main mountPath: /data/main command: ['sh','-c','cp /data/helloworld/* /data/main/ '] imagePullPolicy: IfNotPresent containers: - image: harbor.gtmsh.com/library/python:v2 imagePullPolicy: Never name: helloworld ports: - containerPort: 5000 protocol: TCP volumeMounts: - name: main mountPath: /data/main imagePullPolicy: IfNotPresent volumes: - name: main emptyDir: {} kubectl apply -f hello-dp.yaml [root@con-mas ~]# kubectl get pod NAME READY STATUS RESTARTS AGE helloworld-55cbdf644d-dg62t 1/1 Running 0 13m helloworld-55cbdf644d-tszg8 1/1 Running 0 13m ```