• core design principles
  • constants and well-known values and paths
    • /etc/kubernetes/manifests
    • /etc/kubernetes
    • names of certificates and key files
  • API server
    • static pod manifest
    • other api server flags
  • controller manager
    • static Pod manifest
    • other flags
  • flow
    • pod creation
    • ingress traffic
• ports and protocols
  • contol plane
  • worker node(s)
• architecture
  • control pannel
    • kube-apiserver
    • etcd
    • kube-scheduler
    • controller manager
    • ccm : cloud controller manager
  • work node
    • kubelet
    • kube proxy
    • cri-o : container runtime
• jsonpath
• options
  • explain
• kubectl alias
  • __start_kubectl
  • _complete_alias
  • kubecolor
• token
  • check token
  • generate token
• tear down
• references

[!TIP] kubernetes.io add /_print as suffix in the url, it will show pages into one page i.e.:

• https://kubernetes.io/docs/setup/best-practices/
• https://kubernetes.io/docs/setup/best-practices/_print/
• * ramitsurana/awesome-kubernetes | Awesome Kubernetes
• ibm-cloud-architecture/refarch-cloudnative-devops-kubernetes
• Kubernetes Architecture Explanation | Part 1
• Load Balancers vs API Gateways

core design principles

constants and well-known values and paths

/etc/kubernetes/manifests

[!TIP] /etc/kubernetes/manifests as the path where kubelet should look for static Pod manifests. Names of static Pod manifests are:

• etcd.yaml
• kube-apiserver.yaml
• kube-controller-manager.yaml
• kube-scheduler.yaml

/etc/kubernetes

[!TIP]

• important kubernetes cluster configurations /etc/kubernetes/ as the path where kubeconfig files with identities for control plane components are stored. Names of kubeconfig files are:
• kubelet.conf (bootstrap-kubelet.conf during TLS bootstrap)
• controller-manager.conf
• scheduler.conf
• admin.conf for the cluster admin and kubeadm itself

names of certificates and key files

[!TIP]

• ca.crt, ca.key for the Kubernetes certificate authority
• apiserver.crt, apiserver.key for the API server certificate
• apiserver-kubelet-client.crt, apiserver-kubelet-client.key for the client certificate used by the API server to connect to the kubelets securely
• sa.pub, sa.key for the key used by the controller manager when signing ServiceAccount
• front-proxy-ca.crt, front-proxy-ca.key for the front proxy certificate authority
• front-proxy-client.crt, front-proxy-client.key for the front proxy client

API server

static pod manifest

[!TIP]

• apiserver-advertise-address and apiserver-bind-port to bind to; if not provided, those value defaults to the IP address of the default network interface on the machine and port 6443
• service-cluster-ip-range to use for services
• If an external etcd server is specified, the etcd-servers address and related TLS settings (etcd-cafile, etcd-certfile, etcd-keyfile);
  • if an external etcd server is not be provided, a local etcd will be used ( via host network )
• If a cloud provider is specified, the corresponding --cloud-provider is configured, together with the --cloud-config path if such file exists (this is experimental, alpha and will be removed in a future version)

other api server flags

• --insecure-port=0 to avoid insecure connections to the api server
• --enable-bootstrap-token-auth=true to enable the BootstrapTokenAuthenticator authentication module. See TLS Bootstrapping for more details
• --allow-privileged to true (required e.g. by kube proxy)
• --requestheader-client-ca-file to front-proxy-ca.crt
• --enable-admission-plugins to:
  • NamespaceLifecycle e.g. to avoid deletion of system reserved namespaces
  • LimitRanger and ResourceQuota to enforce limits on namespaces
  • ServiceAccount to enforce service account automation
  • PersistentVolumeLabel attaches region or zone labels to PersistentVolumes as defined by the cloud provider (This admission controller is deprecated and will be removed in a future version. It is not deployed by kubeadm by default with v1.9 onwards when not explicitly opting into using gce or aws as cloud providers)
  • DefaultStorageClass to enforce default storage class on PersistentVolumeClaim objects
  • DefaultTolerationSeconds
  • NodeRestriction to limit what a kubelet can modify (e.g. only pods on this node)
• --kubelet-preferred-address-types to InternalIP,ExternalIP,Hostname; this makes kubectl logs and other API server-kubelet communication work in environments where the hostnames of the nodes aren't resolvable
• Flags for using certificates generated in previous steps:
  • --client-ca-file to ca.crt
  • --tls-cert-file to apiserver.crt
  • --tls-private-key-file to apiserver.key
  • --kubelet-client-certificate to apiserver-kubelet-client.crt
  • --kubelet-client-key to apiserver-kubelet-client.key
  • --service-account-key-file to sa.pub
  • --requestheader-client-ca-file to front-proxy-ca.crt
  • --proxy-client-cert-file to front-proxy-client.crt
  • --proxy-client-key-file to front-proxy-client.key
• Other flags for securing the front proxy (API Aggregation) communications:
  • --requestheader-username-headers=X-Remote-User
  • --requestheader-group-headers=X-Remote-Group
  • --requestheader-extra-headers-prefix=X-Remote-Extra-
  • --requestheader-allowed-names=front-proxy-client

controller manager

static Pod manifest

[!TIP]

• If kubeadm is invoked specifying a --pod-network-cidr, the subnet manager feature required for some CNI network plugins is enabled by setting:
  • --allocate-node-cidrs=true
  • --cluster-cidr and --node-cidr-mask-size flags according to the given CIDR
• If a cloud provider is specified, the corresponding --cloud-provider is specified, together with the --cloud-config path if such configuration file exists (this is experimental, alpha and will be removed in a future version)

other flags

• --controllers enabling all the default controllers plus BootstrapSigner and TokenCleaner controllers for TLS bootstrap. See TLS Bootstrapping for more details
• --use-service-account-credentials to true
• Flags for using certificates generated in previous steps:
  • --root-ca-file to ca.crt
  • --cluster-signing-cert-file to ca.crt, if External CA mode is disabled, otherwise to ""
  • --cluster-signing-key-file to ca.key, if External CA mode is disabled, otherwise to ""
  • --service-account-private-key-file to sa.key

flow

pod creation

ingress traffic

ports and protocols

contol plane

worker node(s)

architecture

control pannel

kube-apiserver

etcd

kube-scheduler

controller manager

ccm : cloud controller manager

work node

[!NOTE]

• supported endpoints
• linux

RUNTIME	PATH TO UNIX DOMAIN SOCKET
containerd	unix:///var/run/containerd/containerd.sock
CRI-O	unix:///var/run/crio/crio.sock
Docker Engine (using cri-dockerd)	unix:///var/run/cri-dockerd.sock

• windows

RUNTIME	PATH TO UNIX DOMAIN SOCKET
containerd	npipe:////./pipe/containerd-containerd
Docker Engine (using cri-dockerd)	npipe:////./pipe/cri-dockerd

kubelet

kube proxy

cri-o : container runtime

jsonpath

[!NOTE|label:references:]

• JSONPath Support
• how to make kubectl jsonpath output on separate lines
• JSONPath Reference
• Basic JSONPath Rules
• Playing with kubectl output

options

explain

$ kubectl explain hpa
KIND:     HorizontalPodAutoscaler
VERSION:  autoscaling/v1

DESCRIPTION:
     configuration of a horizontal pod autoscaler.

FIELDS:
   apiVersion <string>
   ...

• or

  $ kubectl explain configmap
  KIND:     ConfigMap
  VERSION:  v1
  
  DESCRIPTION:
       ConfigMap holds configuration data for pods to consume.
  
  FIELDS:
     apiVersion <string>
       APIVersion defines the versioned schema of this representation of an
       object. Servers should convert recognized schemas to the latest internal
       value, and may reject unrecognized values. More info:
       https://git.k8s.io/community/contributors/devel/api-conventions.md#resources
       ...
  

kubectl alias

__start_kubectl

$ echo 'source <(kubectl completion bash)' >> ~/.bashrc
$ cat >> ~/.bashrc <<EOF
alias k='kubectl'
alias kc='kubectl -n kube-system'
alias ki='kubectl -n ingress-ngxin'
alias kk='kubectl -n kubernetes-dashboard'
for _i in k kc ki kk; do complete -F __start_kubectl "${_i}"; done
EOF
$ source ~/.bashrc

_complete_alias

$ sudo dnf install -y bash-completion

# download bash_completion.sh for kubectl
$ curl -fsSL https://github.com/cykerway/complete-alias/raw/master/complete_alias -o ~/.bash_completion.sh
# or rhel/centos
$ sudo curl -fsSL https://github.com/marslo/dotfiles/raw/main/.marslo/.completion/complete_alias -o /etc/profile.d/complete_alias.sh
$ sudo chmod +x !$

$ cat >> ~/.bashrc << EOF
command -v kubectl >/dev/null && source <(kubectl completion bash)
test -f ~/.bash_completion.sh && source ~/.bash_completion.sh
# or
# test -f /etc/profile.d/complete_alias.sh && source /etc/profile.d/complete_alias.sh

alias k='kubectl'
alias kc='kubectl -n kube-system'
alias ki='kubectl -n ingress-ngxin'
alias kk='kubectl -n kubernetes-dashboard'
alias km='kubectl -n monitoring'

complete -o default -F __start_kubectl kubecolor
complete -o nosort -o bashdefault -o default -F _complete_alias $(alias | sed -rn 's/^alias ([^=]+)=.+kubec.+$/\1/p' | xargs)
EOF

$ source ~/.bashrc

kubecolor

$ [[ -d /tmp/kubecolor ]] && sudo mkdir -p /tmp/kubecolor
$ curl -fsSL https://github.com/hidetatz/kubecolor/releases/download/v0.0.25/kubecolor_0.0.25_Linux_x86_64.tar.gz | tar xzf - -C /tmp/kubecolor
$ sudo mv /tmp/kubecolor/kubecolor /usr/local/bin/
$ sudo chmod +x /usr/local/bin/kubecolor

token

check token

$ sudo kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION   EXTRA GROUPS
bop765.brol9nsrw820gmbi   <forever>   <never>                     authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token
khhfwa.jvkvrpiknx4o6ffy   19h         2018-07-13T11:37:43+08:00   authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token

generate token

[!NOET|label:see also:]

• * iMarslo: get join command
• * iMarslo: retrive join command

$ sudo kubeadm token create --print-join-command
kubeadm join 192.168.1.100:6443 --token lhb1ln.oj0fqwgd1yl7l9xp --discovery-token-ca-cert-hash sha256:cba8df87dcb70c83c19af72c02e4886fcc7b0cf05319084751e6ece688443bde

$ sudo kubeadm token create --print-join-command --ttl=0
kubeadm join 192.168.1.100:6443 --token bop765.brol9nsrw820gmbi --discovery-token-ca-cert-hash sha256:c8650c56faf72b8bf71c576f0d13f44c93bea2d21d4329c64bb97cba439af5c3

tear down

[!TIP]

• How to completely uninstall kubernetes
• * iMarslo: kubeadm reset and teardown

• ubuntu

  $ kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
  $ kubectl delete node <node name>
  
  $ sudo kubeadm  reset
  [preflight] Running pre-flight checks.
  [reset] Stopping the kubelet service.
  [reset] Unmounting mounted directories in "/var/lib/kubelet"
  [reset] Removing kubernetes-managed containers.
  [reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes /var/lib/etcd]
  [reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
  [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
  
  $ systemctl stop kubelet
  $ docker system prune -a -f
  $ systemctl stop docker
  
  $ sudo rm -rf /etc/kubernetes/
  $ sudo rm -rf /var/lib/cni/
  $ sudo rm -rf /var/lib/kubelet/*
  $ sudo rm -rf /etc/cni/
  $ sudo ifconfig cni0 down
  $ sudo ifconfig flannel.1 down
  
  $ rm -rf ~/.kube/
  
  $ sudo apt purge kubeadm kubectl kubelet kubernetes-cni kube*
  $ sudo apt autoremove
  

• CentOS/RHEL

  $ kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
  $ kubectl delete node <node name>
  
  $ sudo kubeadm reset -f --v=5
  $ docker system prune -a -f
  
  # stop and disable services
  $ systemctl stop kubelet
  $ systemctl disable kubelet
  $ systemctl stop docker
  $ systemctl disable docker
  $ systemctl stop crio
  # or
  $ systemctl disable crio
  $ sudo rm -rf /etc/systemd/system/multi-user.target.wants/kubelet.service
  $ sudo rm -rf /etc/systemd/system/multi-user.target.wants/docker.service
  $ sudo rm -rf /usr/lib/systemd/system/docker.service
  $ sudo rm -rf /usr/lib/systemd/system/kubelet.service.d/
  
  # network interface
  $ sudo ifconfig cni0 down
  $ sudo ip link delete cni0
  $ sudo ifconfig flannel.1 down
  $ sudo ip link delete flannel.1
  $ sudo ifconfig docker0 down
  $ sudo ip link delete docker0
  $ sudo ifconfig vxlan.calico down
  $ sudo ip link delete vxlan.calico
  
  $ sudo yum versionlock delete docker-ce
  $ sudo yum versionlock delete docker-ce-cli
  $ sudo yum versionlock delete kubeadm
  $ sudo yum versionlock delete kubelet
  $ sudo yum versionlock delete kubectl
  $ sudo yum versionlock delete kubernetes-cni
  # or
  $ sudo yum versionlock clear
  $ sudo yum remove -y docker-ce docker-ce-cli containerd.io kubectl kubeadm kubelet kubernetes-cni
  $ sudo yum autormeove
  
  $ sudo rm -rf /etc/cni /etc/kubernetes /etc/docker $HOME/.kube
  $ sudo rm -rf /usr/libexec/docker /usr/libexec/kubernetes
  
  $ sudo rm -rf /var/lib/etcd/               # optional
  $ sudo rm -rf /var/lib/kubelet/ /var/lib/dockershim /var/lib/yum/repos/x86_64/7/kubernetes /var/log/pods /var/log/containers
  $ sudo rm -rf /var/run/docker.sock
  $ sudo rm -rf /var/cache/yum/x86_64/7/kubernetes
  
  $ sudo yum clean all
  $ sudo rm -rf /var/cache/yum
  $ sudo yum makecache
  $ sudo yum check-update
  

references