前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >intel userspace cni 适配 Kubevirt

intel userspace cni 适配 Kubevirt

作者头像
后端云
发布2022-11-25 15:58:07
3170
发布2022-11-25 15:58:07
举报
文章被收录于专栏:后端云后端云

虽然KubeVirt还没官方支持DPDK,但intel userspace cni已经为KubeVirt做了一些适配。

有以下3点适配:

  • vhost user client&server
  • emptyDir
  • ovs&qemu privilege

vhost user client&server

kubevirt 使用DPDK需要用到intel的网络插件userspace cni,该插件使得ovs工作在client模式,kubevirt使得qemu工作在server模式。

关于 ovs&kubevirt vhost user client&server 参考 https://www.backendcloud.cn/2022/06/20/kubevirt-with-dpdk/#For-vhost-user-client-ports-Open-vSwitch-acts-as-the-client-and-QEMU-the-server

本篇是关于intel userspace cni,所以本篇只涉及ovs工作在client模式,qemu工作在server模式参考 https://www.backendcloud.cn/2022/06/20/kubevirt-with-dpdk/

cniovs/ovsctrl.go

代码语言:javascript
复制
func createVhostPort(sock_dir string, sock_name string, client bool, bridge_name string) (string, error) {
	var err error
	type_str := "type=dpdkvhostuser"
	if client {
		type_str = "type=dpdkvhostuserclient"
	}
	// COMMAND: ovs-vsctl add-port <bridge_name> <sock_name> -- set Interface <sock_name> type=<dpdkvhostuser|dpdkvhostuserclient>
	cmd := "ovs-vsctl"
	args := []string{"add-port", bridge_name, sock_name, "--", "set", "Interface", sock_name, type_str}

	if client == true {
		socketarg := "options:vhost-server-path=" + sock_dir		if sock_dir[len(sock_dir)-1] != '/' {
			socketarg += "/"
		}
		socketarg += sock_name
		logging.Debugf("Additional string: %s", socketarg)

		args = append(args, socketarg)
	}
	if _, err = execCommand(cmd, args); err != nil {
		return "", err	}
	if client == false {
		// Determine the location OvS uses for Sockets. Default location can be
		// overwritten with environmental variable: OVS_SOCKDIR
		ovs_socket_dir, ok := os.LookupEnv("OVS_SOCKDIR")
		if ok == false {
			ovs_socket_dir = defaultOvSSocketDir		}
		// Move socket to defined dir for easier mounting
		err = os.Rename(ovs_socket_dir+sock_name, sock_dir+sock_name)
		if err != nil {
			logging.Errorf("Rename ERROR: %v", err)
			err = nil
			//deleteVhostPort(sock_name, bridge_name)
		}
	}
	return sock_name, err}language-go复制代码

上面代码只需要关注client == true的部分,另一种client == false已经废弃,execCommand(cmd, args)方法执行了ovs-vsctl add-port COMMAND: ovs-vsctl add-port – set Interface type=<dpdkvhostuser|dpdkvhostuserclient>

emptyDir

volumeMount.HostPath.Path基础上增加volumeMount.EmptyDir提供给libevirt容器podvolumemount使用,emptyDir用于创建vhostuser socket

pkg/annotations/annotations.go

代码语言:javascript
复制
const (
    ...
	volMntKeySharedDir      = "shared-dir"
	DefaultHostkubeletPodBaseDir  = "/var/lib/kubelet/pods/"
	DefaultHostEmptyDirVolumeName = "volumes/kubernetes.io~empty-dir/")func GetPodVolumeMountHostSharedDir(pod *v1.Pod) (string, error) {
	var hostSharedDir string
	logging.Verbosef("GetPodVolumeMountSharedDir: type=%T Volumes=%v", pod.Spec.Volumes, pod.Spec.Volumes)
	if len(pod.Spec.Volumes) == 0 {
		return hostSharedDir, &NoSharedDirProvidedError{"Error: No Volumes. Need \"shared-dir\" in podSpec \"Volumes\""}
	}

	for _, volumeMount := range pod.Spec.Volumes {
		if volumeMount.Name == volMntKeySharedDir {
			if volumeMount.HostPath != nil {
				hostSharedDir = volumeMount.HostPath.Path			} else if volumeMount.EmptyDir != nil {
				hostSharedDir = DefaultHostkubeletPodBaseDir + string(pod.UID) + "/" + DefaultHostEmptyDirVolumeName + volMntKeySharedDir			} else {
				return hostSharedDir, &NoSharedDirProvidedError{"Error: Volume is invalid"}
			}
			break
		}
	}
	if len(hostSharedDir) == 0 {
		return hostSharedDir, &NoSharedDirProvidedError{"Error: No shared-dir. Need \"shared-dir\" in podSpec \"Volumes\""}
	}
	return hostSharedDir, nil}language-go复制代码

host提供给VolumeMount的host volume增加emptyDir,命名看上去很长 DefaultHostkubeletPodBaseDir + string(pod.UID) + “/“ + DefaultHostEmptyDirVolumeName + volMntKeySharedDir unix socket有108长度的限制,可以通过下面一段代码mount的一个短的path来缩短host user socket path长度。

cniovs/cniovs.go

代码语言:javascript
复制
const (
	defaultBridge               = "br0"
	DefaultHostVhostuserBaseDir = "/var/lib/vhost_sockets/")func getShortSharedDir(sharedDir string) string {
	// sun_path for unix domain socket has a array size of 108
	// When the sharedDir path length greater than 89 (108 - 19)
	// 19 is the possible vhostuser socke file name length "/abcdefghijkl-net99" (1 + 12 + 1 + 3 + 2)
	if len(sharedDir) >= 89 && strings.Contains(sharedDir, "empty-dir") {
		// Format - /var/lib/kubelet/pods/<podID>/volumes/kubernetes.io~empty-dir/shared-dir
		parts := strings.Split(sharedDir, "/")
		podID := parts[5]
		newSharedDir := DefaultHostVhostuserBaseDir + podID
		logging.Infof("getShortSharedDir: Short shared directory: %s", newSharedDir)
		return newSharedDir	}
	return sharedDir}func createSharedDir(sharedDir, oldSharedDir string) error {
	var err error

	_, err = os.Stat(sharedDir)
	if os.IsNotExist(err) {
		err = os.MkdirAll(sharedDir, 0750)
		if err != nil {
			logging.Errorf("createSharedDir: Failed to create dir (%s): %v", sharedDir, err)
			return err		}

		if strings.Contains(sharedDir, DefaultHostVhostuserBaseDir) {
			logging.Debugf("createSharedDir: Mount from %s to %s", oldSharedDir, sharedDir)
			err = unix.Mount(oldSharedDir, sharedDir, "", unix.MS_BIND, "")
			if err != nil {
				logging.Errorf("createSharedDir: Failed to bind mout: %s", err)
				return err			}
		}
		return nil

	}
	return err}language-go复制代码

Why is the maximal path length allowed for unix-sockets on linux 108? 参考 https://www4.cs.fau.de/Services/Doc/C/libc.html#TOC189 章节 Sockets - The File Namespace - Details of File Namespace

ovs&qemu privilege

conf.HostConf.VhostConf.Group 配置一个”group”名称,默认配置为”hugetlbfs”,用于vhostuser socket的权限设定。让ovs和qemu用户在同一个用户组中,使得vhostuser socket在他们中可以共享,避免出现低权限访问不了高权限的情况。

代码语言:javascript
复制
func setSharedDirGroup(sharedDir string, group string) error {
	groupInfo, err := user.LookupGroup(group)
	if err != nil {
		return err	}

	logging.Debugf("setSharedDirGroup: group %s's gid is %s", group, groupInfo.Gid)
	gid, err := strconv.Atoi(groupInfo.Gid)
	if err != nil {
		return err	}

	err = os.Chown(DefaultHostVhostuserBaseDir, -1, gid)
	if err != nil {
		return err	}

	err = os.Chown(sharedDir, -1, gid)
	if err != nil {
		return err	}
	return nil}func addLocalDeviceVhost(conf *types.NetConf, args *skel.CmdArgs, actualSharedDir string, data *OvsSavedData) error {
	...
	group := conf.HostConf.VhostConf.Group	if group != "" {
		err = setSharedDirGroup(sharedDir, group)
		if err != nil {
			logging.Errorf("addLocalDeviceVhost: Failed to set shared dir group: %v", err)
			return err		}
	}
	...}language-go
本文参与 腾讯云自媒体同步曝光计划,分享自微信公众号。
原始发表:2022-06-25,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 后端云 微信公众号,前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体同步曝光计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
目录
  • vhost user client&server
  • emptyDir
  • ovs&qemu privilege
相关产品与服务
容器服务
腾讯云容器服务(Tencent Kubernetes Engine, TKE)基于原生 kubernetes 提供以容器为核心的、高度可扩展的高性能容器管理服务,覆盖 Serverless、边缘计算、分布式云等多种业务部署场景,业内首创单个集群兼容多种计算节点的容器资源管理模式。同时产品作为云原生 Finops 领先布道者,主导开源项目Crane,全面助力客户实现资源优化、成本控制。
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档