| 知乎專欄 ｜ 多維度架構 | 微信號 netkiller-ebook | QQ群：128659835 請註明“讀者” |
目錄
執行下面命令檢查伺服器是否開啟虛擬化技術
egrep --color 'vmx|svm' /proc/cpuinfo
如果沒有任何輸出，請重啟伺服器進入 BIOS 啟用 VT-X 或 AMD-v
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \ && install minikube-linux-amd64 /usr/local/bin/minikube
嘗試運行 minikube 如果輸出幫助信息表示安裝成功
[root@localhost ~]# minikube version minikube version: v1.13.0 commit: 0c5e9de4ca6f9c55147ae7f90af97eff5befef5f-dirty
echo "1" > /proc/sys/net/bridge/bridge-nf-call-iptables
dnf 安裝 kubectl
cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF
[root@localhost ~]# dnf install kubectl
二進制安裝 kubectl
curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl" \ && install kubectl /usr/local/bin/kubectl
如果你不想安裝虛擬機
adduser docker su - docker sudo usermod -aG docker $USER && newgrp docker
[docker@localhost ~]$ minikube start --driver=docker
* minikube v1.13.0 on Centos 8.2.2004
* Using the docker driver based on user configuration
X Requested memory allocation (1694MB) is less than the recommended minimum 2000MB. Deployments may fail.
X The requested memory allocation of 1694MiB does not leave room for system overhead (total system memory: 1694MiB). You may face stability issues.
* Suggestion: Start minikube with less memory allocated: 'minikube start --memory=1694mb'
* Starting control plane node minikube in cluster minikube
* Pulling base image ...
* Downloading Kubernetes v1.19.0 preload ...
> preloaded-images-k8s-v6-v1.19.0-docker-overlay2-amd64.tar.lz4: 486.28 MiB
檢查硬件是否支持虛擬化
iMac:Linux neo$ sysctl -a | grep -E --color 'machdep.cpu.features|VMX' machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX SMX EST TM2 SSSE3 CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC POPCNT AES PCID XSAVE OSXSAVE TSCTMR AVX1.0
$ brew install minikube $ brew install kubectl $ brew install kubernetes-helm
neo@MacBook-Pro-Neo ~ % minikube start 😄 minikube v1.13.1 on Darwin 11.0 🆕 Kubernetes 1.19.2 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.19.2 ✨ Using the hyperkit driver based on existing profile 👍 Starting control plane node minikube in cluster minikube 🔄 Restarting existing hyperkit VM for "minikube" ... ❗ This VM is having trouble accessing https://k8s.gcr.io 💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/ 🐳 Preparing Kubernetes v1.19.0 on Docker 19.03.12 ... 🔎 Verifying Kubernetes components... 🌟 Enabled addons: dashboard, default-storageclass, storage-provisioner 🏄 Done! kubectl is now configured to use "minikube" by default
有些老系統可能不支持 hyperkit，需要virtualbox。
$ brew cask install virtualbox $ minikube start —vm-driver=virtualbox $ minikube dashboard
啟動
minikube start
運行一個 echoserver 鏡像
kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 kubectl expose deployment hello-minikube --type=NodePort minikube service hello-minikube
查詢 echoserver 訪問地址
minikube service hello-minikube --url
在瀏覽器中訪問查詢到的網址
停止並刪除鏡像
minikube stop minikube delete
例 164.1. minikube 操作演示
快速開始使用 minikube 運行一個鏡像
[root@localhost ~]# kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. deployment.apps/hello-minikube created [root@localhost ~]# kubectl expose deployment hello-minikube --type=NodePort service/hello-minikube exposed [root@localhost ~]# minikube service hello-minikube Opening kubernetes service default/hello-minikube in default browser... [root@localhost ~]# kubectl get pod NAME READY STATUS RESTARTS AGE hello-minikube-5c856cbf98-6vfvp 1/1 Running 0 6m59s [root@localhost ~]# minikube service hello-minikube --url http://172.16.0.121:30436 [root@localhost ~]# curl http://172.16.0.121:30436 CLIENT VALUES: client_address=172.17.0.1 command=GET real path=/ query=nil request_version=1.1 request_uri=http://172.16.0.121:8080/ SERVER VALUES: server_version=nginx: 1.10.0 - lua: 10001 HEADERS RECEIVED: accept=*/* host=172.16.0.121:30436 user-agent=curl/7.29.0 BODY: -no body in request-
[root@localhost ~]# minikube
Minikube is a CLI tool that provisions and manages single-node Kubernetes clusters optimized for development workflows.
Usage:
minikube [command]
Available Commands:
addons Modify minikube's kubernetes addons
cache Add or delete an image from the local cache.
completion Outputs minikube shell completion for the given shell (bash or zsh)
config Modify minikube config
dashboard Access the kubernetes dashboard running within the minikube cluster
delete Deletes a local kubernetes cluster
docker-env Sets up docker env variables; similar to '$(docker-machine env)'
help Help about any command
ip Retrieves the IP address of the running cluster
logs Gets the logs of the running instance, used for debugging minikube, not user code
mount Mounts the specified directory into minikube
profile Profile sets the current minikube profile
service Gets the kubernetes URL(s) for the specified service in your local cluster
ssh Log into or run a command on a machine with SSH; similar to 'docker-machine ssh'
ssh-key Retrieve the ssh identity key path of the specified cluster
start Starts a local kubernetes cluster
status Gets the status of a local kubernetes cluster
stop Stops a running local kubernetes cluster
tunnel tunnel makes services of type LoadBalancer accessible on localhost
update-check Print current and latest version number
update-context Verify the IP address of the running cluster in kubeconfig.
version Print the version of minikube
Flags:
--alsologtostderr log to standard error as well as files
-b, --bootstrapper string The name of the cluster bootstrapper that will set up the kubernetes cluster. (default "kubeadm")
-h, --help help for minikube
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--logtostderr log to standard error instead of files
-p, --profile string The name of the minikube VM being used.
This can be modified to allow for multiple minikube instances to be run independently (default "minikube")
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
-v, --v Level log level for V logs
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
Use "minikube [command] --help" for more information about a command.
minikube start --registry-mirror=https://registry.docker-cn.com minikube start --image-mirror-country=cn --registry-mirror="https://docker.mirrors.ustc.edu.cn" --insecure-registry="127.0.0.1:5000" minikube start --image-mirror-country=cn --registry-mirror="https://docker.mirrors.ustc.edu.cn" --insecure-registry="192.168.0.0/24"
minikube start --image-mirror-country=cn --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers
# 從阿里雲下載 virtualbox 鏡像
minikube start --vm-driver='virtualbox' --image-mirror-country cn \
--iso-url=https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/iso/minikube-v1.9.0.iso \
--registry-mirror=https://docker.mirrors.ustc.edu.cn
minikube start --vm-driver=virtualbox \
--image-mirror-country cn \
--registry-mirror=https://docker.mirrors.ustc.edu.cn \
--image-repository=registry.aliyuncs.com/google_containers \
--insecure-registry=192.168.0.10:5000 //訪問宿主機的私有docker倉庫
[root@localhost ~]# minikube stop Stopping local Kubernetes cluster... Machine stopped.
neo@MacBook-Pro-Neo ~ % minikube docker-env export DOCKER_TLS_VERIFY="1" export DOCKER_HOST="tcp://192.168.64.3:2376" export DOCKER_CERT_PATH="/Users/neo/.minikube/certs" export MINIKUBE_ACTIVE_DOCKERD="minikube" # To point your shell to minikube's docker-daemon, run: # eval $(minikube -p minikube docker-env)
設置環境變數
# eval $(minikube docker-env) # eval $(minikube -p minikube docker-env)
neo@MacBook-Pro-Neo ~ % minikube ssh
_ _
_ _ ( ) ( )
___ ___ (_) ___ (_)| |/') _ _ | |_ __
/' _ ` _ `\| |/' _ `\| || , < ( ) ( )| '_`\ /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )( ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)
$
# cache a image into $HOME/.minikube/cache/images $ minikube cache add ubuntu:16.04 $ minikube cache add redis:3 # list cached images $ minikube cache list redis:3 ubuntu:16.04 # delete cached images $ minikube cache delete ubuntu:16.04 $ minikube cache delete $(minikube cache list)
Dashboard是基于Web的Kubernetes管理界面。使用下面的命令啟動：
minikube dashboard
查詢控製麵板訪問地址
$ minikube dashboard --url http://192.168.3.14:30000
[root@localhost ~]# minikube service hello-minikube --url http://172.16.0.121:30436
minikube profile demo minikube start -p demo --memory=8192 --cpus=6 --disk-size=50g
minikube addons enable heapster minikube addons enable ingress
啟用 WebUI
[root@localhost ~]# minikube addons enable dashboard dashboard was successfully enabled [root@localhost ~]# minikube addons list | grep dashboard - dashboard: enabled
[root@localhost ~]# minikube addons list - addon-manager: enabled - dashboard: enabled - default-storageclass: enabled - efk: disabled - freshpod: disabled - gvisor: disabled - heapster: disabled - ingress: disabled - kube-dns: disabled - metrics-server: disabled - nvidia-driver-installer: disabled - nvidia-gpu-device-plugin: disabled - registry: disabled - registry-creds: disabled - storage-provisioner: enabled - storage-provisioner-gluster: disabled
--vm-driver=none 不支持 ssh
[root@localhost ~]# minikube ssh 'none' driver does not support 'minikube ssh' command
iMac:registry neo$ minikube addons list |-----------------------------|----------|--------------| | ADDON NAME | PROFILE | STATUS | |-----------------------------|----------|--------------| | ambassador | minikube | disabled | | dashboard | minikube | enabled ✅ | | default-storageclass | minikube | enabled ✅ | | efk | minikube | disabled | | freshpod | minikube | disabled | | gcp-auth | minikube | disabled | | gvisor | minikube | disabled | | helm-tiller | minikube | disabled | | ingress | minikube | disabled | | ingress-dns | minikube | disabled | | istio | minikube | disabled | | istio-provisioner | minikube | disabled | | kubevirt | minikube | disabled | | logviewer | minikube | disabled | | metallb | minikube | disabled | | metrics-server | minikube | disabled | | nvidia-driver-installer | minikube | disabled | | nvidia-gpu-device-plugin | minikube | disabled | | olm | minikube | disabled | | pod-security-policy | minikube | disabled | | registry | minikube | disabled | | registry-aliases | minikube | disabled | | registry-creds | minikube | disabled | | storage-provisioner | minikube | enabled ✅ | | storage-provisioner-gluster | minikube | disabled | |-----------------------------|----------|--------------|
# enable the registry addon
$ minikube addons enable registry
$ minikube start
# use the minikube docker daemon from the host
$ eval $(minikube docker-env)
# get the ip of the registry endpoint
$ kubectl -n kube-system get svc registry -o jsonpath="{.spec.clusterIP}"
10.0.0.240
{
"insecure-registries" : ["10.0.0.240"]
}
$ minikube ssh $ docker pull busybox $ docker tag busybox 10.0.0.240/busybox # or # build and push to insecure registry $ docker build -t 10.0.0.240/busybox . $ docker push 10.0.0.240/busybox
檢查一下 BIOS 是否開啟 VT-X/AMD-v
如果在虛擬機安裝 Minikube 也會遇到這個問題。 可以使用 --vm-driver=none 參數啟動。
neo@ubuntu:~$ sudo minikube start --vm-driver=none
解決方法
echo "1" > /proc/sys/net/bridge/bridge-nf-call-iptables
然後在 minikube start
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: 3.1: Pulling from pause Get https://k8s.gcr.io/v2/pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610: net/http: TLS handshake timeout
更換鏡像再重試
[root@localhost ~]# minikube start --vm-driver=none --registry-mirror=https://registry.docker-cn.com
啟動提示如下錯誤，一般出現這種錯誤是因為 minikube stop, minikube delete 後再重啟 minikube start
error execution phase kubeconfig/admin: a kubeconfig file "/etc/kubernetes/admin.conf" exists already but has got the wrong CA cert error execution phase kubeconfig/kubelet: a kubeconfig file "/etc/kubernetes/kubelet.conf" exists already but has got the wrong CA cert error execution phase kubeconfig/controller-manager: a kubeconfig file "/etc/kubernetes/controller-manager.conf" exists already but has got the wrong CA cert error execution phase kubeconfig/scheduler: a kubeconfig file "/etc/kubernetes/scheduler.conf" exists already but has got the wrong CA cert
解決方法
[root@localhost ~]# mv /etc/kubernetes/admin.conf /etc/kubernetes/admin.conf.backup [root@localhost ~]# mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.backup [root@localhost ~]# mv /etc/kubernetes/controller-manager.conf /etc/kubernetes/controller-manager.conf.backup [root@localhost ~]# mv /etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.backup
現在啟動 minikube start 不會再出錯
[root@localhost ~]# minikube start --vm-driver=none Starting local Kubernetes v1.13.2 cluster... Starting VM... Getting VM IP address... Moving files into cluster... Setting up certs... Connecting to cluster... Setting up kubeconfig... Stopping extra container runtimes... Starting cluster components... Verifying kubelet health ... Verifying apiserver health ... Kubectl is now configured to use the cluster. =================== WARNING: IT IS RECOMMENDED NOT TO RUN THE NONE DRIVER ON PERSONAL WORKSTATIONS The 'none' driver will run an insecure kubernetes apiserver as root that may leave the host vulnerable to CSRF attacks When using the none driver, the kubectl config and credentials generated will be root owned and will appear in the root home directory. You will need to move the files to the appropriate location and then set the correct permissions. An example of this is below: sudo mv /root/.kube $HOME/.kube # this will write over any previous configuration sudo chown -R $USER $HOME/.kube sudo chgrp -R $USER $HOME/.kube sudo mv /root/.minikube $HOME/.minikube # this will write over any previous configuration sudo chown -R $USER $HOME/.minikube sudo chgrp -R $USER $HOME/.minikube This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true Loading cached images from config file. Everything looks great. Please enjoy minikube!
問題原因，使用私有 registry 由於沒有 HTTPS 導致 kubectl 使用 https 去訪問私有 registry.
Failed to pull image "192.168.3.85:5000/netkiller/config:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://192.168.3.85:5000/v2/: http: server gave HTTP response to HTTPS client
minikube 並不會使用 docker 配置檔案中的 insecure-registry 配置項
解決辦法
minikube start --insecure-registry=127.0.0.1:5000
或指定網段
minikube start --insecure-registry "10.0.0.0/24"
iMac:kubernetes neo$ kubectl create -f redis/redis.yml configmap/redis-config created deployment.apps/redis created The Service "redis" is invalid: spec.ports[0].nodePort: Invalid value: 6379: provided port is not in the valid range. The range of valid ports is 30000-32767
編輯kube-apiserver.yaml檔案
$ minikube ssh $ sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
增加kube-apiserver的啟動配置項
--service-node-port-range=1024-65535
$ sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.64.5:8443
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
- --advertise-address=192.168.64.5
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/var/lib/minikube/certs/ca.crt
- --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
- --enable-bootstrap-token-auth=true
- --etcd-cafile=/var/lib/minikube/certs/etcd/ca.crt
- --etcd-certfile=/var/lib/minikube/certs/apiserver-etcd-client.crt
- --etcd-keyfile=/var/lib/minikube/certs/apiserver-etcd-client.key
- --etcd-servers=https://127.0.0.1:2379
- --insecure-port=0
- --kubelet-client-certificate=/var/lib/minikube/certs/apiserver-kubelet-client.crt
- --kubelet-client-key=/var/lib/minikube/certs/apiserver-kubelet-client.key
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --proxy-client-cert-file=/var/lib/minikube/certs/front-proxy-client.crt
- --proxy-client-key-file=/var/lib/minikube/certs/front-proxy-client.key
- --requestheader-allowed-names=front-proxy-client
- --requestheader-client-ca-file=/var/lib/minikube/certs/front-proxy-ca.crt
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --requestheader-group-headers=X-Remote-Group
- --requestheader-username-headers=X-Remote-User
- --secure-port=8443
- --service-account-key-file=/var/lib/minikube/certs/sa.pub
- --service-cluster-ip-range=10.10.0.0/24
- --service-node-port-range=1024-65535
- --tls-cert-file=/var/lib/minikube/certs/apiserver.crt
- --tls-private-key-file=/var/lib/minikube/certs/apiserver.key
image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.2
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 192.168.64.5
path: /livez
port: 8443
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
name: kube-apiserver
readinessProbe:
failureThreshold: 3
httpGet:
host: 192.168.64.5
path: /readyz
port: 8443
scheme: HTTPS
periodSeconds: 1
timeoutSeconds: 15
resources:
requests:
cpu: 250m
startupProbe:
failureThreshold: 24
httpGet:
host: 192.168.64.5
path: /livez
port: 8443
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /var/lib/minikube/certs
name: k8s-certs
readOnly: true
- mountPath: /usr/share/ca-certificates
name: usr-share-ca-certificates
readOnly: true
hostNetwork: true
priorityClassName: system-node-critical
volumes:
- hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
name: ca-certs
- hostPath:
path: /var/lib/minikube/certs
type: DirectoryOrCreate
name: k8s-certs
- hostPath:
path: /usr/share/ca-certificates
type: DirectoryOrCreate
name: usr-share-ca-certificates
status: {}
sudo systemctl restart kubelet
iMac:~ neo$ minikube addons enable registry 🔎 Verifying registry addon... ❌ Exiting due to MK_ENABLE: run callbacks: running callbacks: [verifying registry addon pods : timed out waiting for the condition: timed out waiting for the condition] 😿 If the above advice does not help, please let us know: 👉 https://github.com/kubernetes/minikube/issues/new/choose