前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >Kubernetes集群中,Node异常时Pod状态分析

Kubernetes集群中,Node异常时Pod状态分析

原创
作者头像
Walton
发布2019-02-28 23:59:33
5.2K0
发布2019-02-28 23:59:33
举报
文章被收录于专栏:KubernetesKubernetes

摘要:Kubernetes集群中Node NotReady是经常遇到的现象,我们需要了解各种Workload Type对应的Pod此时的行为。文中只给出现象总结,并没有写出对应的逻辑分析,因为这主要是Node Controller的行为,我对Node Controller写过四篇系列博客,大家可以参考。

Kubelet进程异常,Pod状态变化

一个节点上运行着pod前提下,这个时候把kubelet进程停掉。里面的pod会被干掉吗?会在其他节点recreate吗?

结论:

(1)Node状态变为NotReady

(2)Pod 5分钟之内状态无变化,5分钟之后的状态变化:Daemonset的Pod状态变为Nodelost,Deployment、Statefulset和Static Pod的状态先变为NodeLost,然后马上变为Unknown。Deployment的pod会recreate,但是Deployment如果是node selector停掉kubelet的node,则recreate的pod会一直处于Pending的状态。Static Pod和Statefulset的Pod会一直处于Unknown状态。

Kubelet恢复,Pod行为

如果kubelet 10分钟后又起来了,node和pod会怎样?

结论:

(1)Node状态变为Ready。

(2)Daemonset的pod不会recreate,旧pod状态直接变为Running。

(3)Deployment的则是将kubelet进程停止的Node删除(原因可能是因为旧Pod状态在集群中有变化,但是Pod状态在变化时发现集群中Deployment的Pod实例数已经够了,所以对旧Pod做了删除处理)

(4)Statefulset的Pod会重新recreate。

(5)Staic Pod没有重启,但是Pod的运行时间会在kubelet起来的时候置为0。

在kubelet停止后,statefulset的pod会变成nodelost,接着就变成unknown,但是不会重启,然后等kubelet起来后,statefulset的pod才会recreate。

还有一个就是Static Pod在kubelet重启以后应该没有重启,但是集群中查询Static Pod的状态时,Static Pod的运行时间变了

StatefulSet Pod为何在Node异常时没有Recreate

Node down后,StatefulSet Pods並沒有重建,為什麼?

我们在node controller中发现,除了daemonset pods外,都会调用delete pod api删除pod。

但并不是调用了delete pod api就会从apiserver/etcd中删除pod object,仅仅是设置pod 的deletionTimestamp,标记该pod要被删除。真正删除Pod的行为是kubelet,kubelet grace terminate该pod后去真正删除pod object。这个时候statefulset controller 发现某个replica缺失就会去recreate这个pod。

但此时由于kubelet挂了,无法与master通信,导致Pod Object一直无法从etcd中删除。如果能成功删除Pod Object,就可以在其他Node重建Pod。

另外,要注意,statefulset只会针对isFailed Pod,(但现在Pods是Unkown状态)才会去delete Pod。

代码语言:txt
复制
// delete and recreate failed pods
		if isFailed(replicas[I]) {
			ssc.recorder.Eventf(set, v1.EventTypeWarning, "RecreatingFailedPod",
				"StatefulSetPlus %s/%s is recreating failed Pod %s",
				set.Namespace,
				set.Name,
				replicas[I].Name)
			if err := ssc.podControl.DeleteStatefulPlusPod(set, replicas[I]); err != nil {
				return &status, err
			}
			if getPodRevision(replicas[I]) == currentRevision.Name {
				status.CurrentReplicas—
			}
			if getPodRevision(replicas[I]) == updateRevision.Name {
				status.UpdatedReplicas—
			}
			status.Replicas—
			replicas[I] = newVersionedStatefulSetPlusPod(
				currentSet,
				updateSet,
				currentRevision.Name,
				updateRevision.Name,
				i)
		}

优化StatefulSet Pod的行为

所以针对node异常的情况,有状态应用(Non-Quorum)的保障,应该补充以下行为:

  • 监测node的网络、kubelet进程、操作系统等是否异常,区别对待。
  • 比如,如果是网络异常,Pod无法正常提供服务,那么需要kubectl delete pod -f —grace-period=0进行强制从etcd中删除该pod。
  • 强制删除后,statefulset controller就会自动触发在其他Node上recreate pod。

亦或者,更粗暴的方法,就是放弃GracePeriodSeconds,StatefulSet Pod GracePeriodSeconds为nil或者0,则就会直接从etcd中删除该object。

代码语言:txt
复制
// BeforeDelete tests whether the object can be gracefully deleted.
// If graceful is set, the object should be gracefully deleted.  If gracefulPending
// is set, the object has already been gracefully deleted (and the provided grace
// period is longer than the time to deletion). An error is returned if the
// condition cannot be checked or the gracePeriodSeconds is invalid. The options
// argument may be updated with default values if graceful is true. Second place
// where we set deletionTimestamp is pkg/registry/generic/registry/store.go.
// This function is responsible for setting deletionTimestamp during gracefulDeletion,
// other one for cascading deletions.
func BeforeDelete(strategy RESTDeleteStrategy, ctx context.Context, obj runtime.Object, options *metav1.DeleteOptions) (graceful, gracefulPending bool, err error) {
	objectMeta, gvk, kerr := objectMetaAndKind(strategy, obj)
	if kerr != nil {
		return false, false, kerr
	}
	if errs := validation.ValidateDeleteOptions(options); len(errs) > 0 {
		return false, false, errors.NewInvalid(schema.GroupKind{Group: metav1.GroupName, Kind: "DeleteOptions"}, "", errs)
	}
	// Checking the Preconditions here to fail early. They'll be enforced later on when we actually do the deletion, too.
	if options.Preconditions != nil && options.Preconditions.UID != nil && *options.Preconditions.UID != objectMeta.GetUID() {
		return false, false, errors.NewConflict(schema.GroupResource{Group: gvk.Group, Resource: gvk.Kind}, objectMeta.GetName(), fmt.Errorf("the UID in the precondition (%s) does not match the UID in record (%s). The object might have been deleted and then recreated", *options.Preconditions.UID, objectMeta.GetUID()))
	}
	gracefulStrategy, ok := strategy.(RESTGracefulDeleteStrategy)
	if !ok {
		// If we're not deleting gracefully there's no point in updating Generation, as we won't update
		// the obcject before deleting it.
		return false, false, nil
	}
	// if the object is already being deleted, no need to update generation.
	if objectMeta.GetDeletionTimestamp() != nil {
		// if we are already being deleted, we may only shorten the deletion grace period
		// this means the object was gracefully deleted previously but deletionGracePeriodSeconds was not set,
		// so we force deletion immediately
		// IMPORTANT:
		// The deletion operation happens in two phases.
		// 1. Update to set DeletionGracePeriodSeconds and DeletionTimestamp
		// 2. Delete the object from storage.
		// If the update succeeds, but the delete fails (network error, internal storage error, etc.),
		// a resource was previously left in a state that was non-recoverable.  We
		// check if the existing stored resource has a grace period as 0 and if so
		// attempt to delete immediately in order to recover from this scenario.
		if objectMeta.GetDeletionGracePeriodSeconds() == nil || *objectMeta.GetDeletionGracePeriodSeconds() == 0 {
			return false, false, nil
		}
		...
	}
	...
}	

原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。

如有侵权,请联系 cloudcommunity@tencent.com 删除。

原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。

如有侵权,请联系 cloudcommunity@tencent.com 删除。

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
目录
  • Kubelet进程异常,Pod状态变化
  • Kubelet恢复,Pod行为
  • StatefulSet Pod为何在Node异常时没有Recreate
  • 优化StatefulSet Pod的行为
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档