synopsis

preflight                    Run pre-flight checks
certs                        Certificate generation
  /ca                          Generate the self-signed Kubernetes CA to provision identities for other Kubernetes components
  /apiserver                   Generate the certificate for serving the Kubernetes API
  /apiserver-kubelet-client    Generate the certificate for the API server to connect to kubelet
  /front-proxy-ca              Generate the self-signed CA to provision identities for front proxy
  /front-proxy-client          Generate the certificate for the front proxy client
  /etcd-ca                     Generate the self-signed CA to provision identities for etcd
  /etcd-server                 Generate the certificate for serving etcd
  /etcd-peer                   Generate the certificate for etcd nodes to communicate with each other
  /etcd-healthcheck-client     Generate the certificate for liveness probes to healthcheck etcd
  /apiserver-etcd-client       Generate the certificate the apiserver uses to access etcd
  /sa                          Generate a private key for signing service account tokens along with its public key
kubeconfig                   Generate all kubeconfig files necessary to establish the control plane and the admin kubeconfig file
  /admin                       Generate a kubeconfig file for the admin to use and for kubeadm itself
  /kubelet                     Generate a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes
  /controller-manager          Generate a kubeconfig file for the controller manager to use
  /scheduler                   Generate a kubeconfig file for the scheduler to use
kubelet-start                Write kubelet settings and (re)start the kubelet
control-plane                Generate all static Pod manifest files necessary to establish the control plane
  /apiserver                   Generates the kube-apiserver static Pod manifest
  /controller-manager          Generates the kube-controller-manager static Pod manifest
  /scheduler                   Generates the kube-scheduler static Pod manifest
etcd                         Generate static Pod manifest file for local etcd
  /local                       Generate the static Pod manifest file for a local, single-node local etcd instance
upload-config                Upload the kubeadm and kubelet configuration to a ConfigMap
  /kubeadm                     Upload the kubeadm ClusterConfiguration to a ConfigMap
  /kubelet                     Upload the kubelet component config to a ConfigMap
upload-certs                 Upload certificates to kubeadm-certs
mark-control-plane           Mark a node as a control-plane
bootstrap-token              Generates bootstrap tokens used to join a node to a cluster
kubelet-finalize             Updates settings relevant to the kubelet after TLS bootstrap
  /experimental-cert-rotation  Enable kubelet client certificate rotation
addon                        Install required addons for passing conformance tests
  /coredns                     Install the CoreDNS addon to a Kubernetes cluster
  /kube-proxy                  Install the kube-proxy addon to a Kubernetes cluster

options

• --apiserver-advertise-address string
• --apiserver-bind-port int32 Default: 6443
• --apiserver-cert-extra-sans strings
• --cert-dir string Default: "/etc/kubernetes/pki"
• --certificate-key string
• --config string
• --control-plane-endpoint string
• --cri-socket string
• --dry-run
• --feature-gates string : A set of key=value pairs that describe feature gates for various features
  • PublicKeysECDSA=true|false (ALPHA - default=false)
  • RootlessControlPlane=true|false (ALPHA - default=false)
  • UnversionedKubeletConfigMap=true|false (BETA - default=true)
• -h, --help
• --ignore-preflight-errors strings
• --image-repository string Default: "k8s.gcr.io"
• --kubernetes-version string Default: "stable-1"
• --node-name string
• --patches string
• --pod-network-cidr string
• --service-cidr string Default: "10.96.0.0/12"
• --service-dns-domain string Default: "cluster.local"
• --skip-certificate-key-print
• --skip-phases strings
• --skip-token-print
• --token string
• --token-ttl duration Default: 24h0m0s
• --upload-certs
• --rootfs string

init workflow

1. preflight checks
2. generate the necessary certificates
3. generate kubeconfig files for control plane components

4. generate static pod manifests for control plane components

   • api server
   • controller-manager
   • scheduler

   [!TIP] kubeadm writes static Pod manifest files for control plane components to /etc/kubernetes/manifests static pod manifest generation for control plane components can be invoked individually with the kubeadm init phase control-plane all command

   references:

   • using custom images

5. generate static pod manifest for local etcd

6. wait for the control plane to come up

   [!TIP] kubeadm waits (upto 4m0s) until localhost:6443/healthz (kube-apiserver liveness) returns ok. However in order to detect deadlock conditions, kubeadm fails fast if localhost:10255/healthz (kubelet liveness) or localhost:10255/healthz/syncloop (kubelet readiness) don't return ok within 40s and 60s respectively.

7. save the kubeadm clusterconfiguration in a configmap for later reference

8. mark the node as control-plane

   [!TIP] Please note that:

   • The node-role.kubernetes.io/master taint is deprecated and will be removed in kubeadm version 1.25
   • Mark control-plane phase phase can be invoked individually with the kubeadm init phase mark-control-plane command

9. configure tls-bootstrapping for node joining

   • create a bootstrap token
   • allow joining nodes to call csr api
   • Setup auto approval for new bootstrap tokens
   • setup nodes certificate rotation with auto approval
   • create the public cluster-info configmap
10. install addons
    • proxy
    • dns

init steps

[!NOTE|label:references:]

• How to Setup Kubernetes(k8s) Cluster in HA with Kubeadm
• Creating Highly Available Clusters with kubeadm
• Multi-Master Kubernetes Cluster Setup with CRI-O and vSphere Storage on CentOS 8
• cri-o/tutorials
  • Running CRI-O on a Kubernetes cluster
• 在Kubernetes中使用CRI-O运行时
• 使用 Kubeadm 和 CRI-O 在 Rocky Linux 8 上安装 Kubernetes 集群
• kubernetes cri-o浅尝
• AlmaLinux基于cri-o+Calico用kubeadm搭建1.24版本多master高可用Kubernetes集群
• etcd-io/etcd
• 详解 K8S 高可用部署
• others:
  • Kubernetes 1.28 震撼发布，Sidecar Containers 迎面而来
  • Manual installation on Kubernetes from scratch with kubectl

1. environment setup
2. install container runtime
   • cri-o
   • docker
3. High Availability
   • keepalived
   • haproxy
   • extenal etcd
4. kuberentes packates
5. init first control plane
6. join peer control planes
7. join work nodes
8. install network plugin
   • calico
   • flannel
9. install ingress
   • ingress-nginx
10. addons
    • monitoring
      • kubernetes-dashboard
      • grafana
      • metrics-server
11. setup tls

troubleshooting

• curl/curl-container

  $ kubectl run curl-deploy --image=quay.io/curl/curl:latest -i --tty -- sh
  $ curl <clusterIP>:<svcPort>
  
  # resume
  $ kubectl attach curl-deploy -c curl-deploy -i -t