• error code
• debug services
  • get podIp
• check log
  • system logs
  • pod logs

[!TIP]

• * coredns 被误删了，可以通过重新应用 coredns 的 Deployment 或 DaemonSet 配置文件来恢复
• * k8s 中如何修改 pod-network-cidr 地址范围
• Troubleshooting Applications
  • Debug Pods
  • Debug Service
  • Debug a StatefulSet
  • Determine the Reason for Pod Failure
  • Debug Init Containers
  • Debug Running Pods
  • Get a Shell to a Running Container
• Troubleshooting Clusters
  • Resource metrics pipeline
  • Tools for Monitoring Resources
  • Monitor Node Health
  • Debugging Kubernetes nodes with crictl
  • Debugging Kubernetes Nodes With Kubectl
  • Developing and debugging services locally using telepresence
  • Auditing
  • Windows debugging tips
• Communicate Between Containers in the Same Pod Using a Shared Volume
• Translate a Docker Compose File to Kubernetes Resources
• pod资源限制和QoS探索
• 【K8S系列】Pod重启策略及重启可能原因
• POD的内存限制 和 OOM Killer
• 记一次k8s pod频繁重启的优化之旅

error code

debug services

[!NOTE|label:reference:]

• How to Debug a Kubernetes Service Effectively
• Debug Services
• Access Services Running on Clusters
• DNS for Services and Pods
• svc in cluster can be visit via
  • CLUSTER-IP
  • <svc-name>.<namespace>.svc.cluster.local

• create pod from cmd
• svc status

  $ kubectl get svc
  NAME      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
  jenkins   ClusterIP   10.111.230.13   <none>        8080/TCP,50017/TCP   17h
  

# create new pod
$ kubectl run ubuntu-marslo \
              --image=ubuntu:18.04 \
              --overrides='{"spec": { "nodeSelector": {"kubernetes.io/hostname": "k8s-node-01"}}}' \
              -it \
              --rm

# check DNS
<ubuntu-marslo> $ cat /etc/resolv.conf
nameserver 10.96.0.10
search devops.svc.cluster.local svc.cluster.local cluster.local company.com
options ndots:5

# debug
$ nc -zv jenkins.devops.svc.cluster.local 30338
$ nc -zv 10.111.230.13                    30338
$ ssh -l marslo -p 30338 -i ~/.ssh/id_rsa jenkins.devops.svc.cluster.local list-plugins
$ ssh -l marslo -p 30338 -i ~/.ssh/id_rsa 10.111.230.13                    list-plugins

get podIp

$ kubectl get pods \
              -l app=hostnames \
              -o go-template='{{range .items}}{{.status.podIP}}{{"\n"}}{{end}}'

check log

[!NOTE|label:references:]

• Kubernetes Logging Tutorial For Beginners

system logs

$ journalctl -u <service> -f

# or
$ journalctl -u kubelet -o cat

# or
$ sudo systemctl status <service> -l --no-pager

pod logs

$ kubectl logs pod <pod_name> --all-containers