【k8s】多pod间通信之共享内存

原创

于顾而言SASE

发布于 2024-03-21 10:33:45

1870

发布于 2024-03-21 10:33:45

进程间通信一直是工程实现经常遇到的场景，比如说数据处理的进程将结果发送给日志进程记录；亦或者多个数据处理进程，在处理数据的的时候，会有一部分配置需要共享，这样以来这些配置文件一个node只存一份即可，并且在内存中查询肯定比redis快的；再或者就是处理大型数据集的时候，需要一个控制进程和多个数据线程，控制线程进行数据的分片和整合，数据进程进行无脑计算，这个时候控制线程和数据线程是需要进行通信的。

进程间通信方式也有很多，这次我们介绍一种最快的方式--共享内存，快的原因简单说就是这些进程间数据传递不再涉及到内核，即不再通过执行进入内核的系统调用来传递彼此的数据。

共享内存实现原理

裸金属共享内存通信

// 1.查看共享内存段
ipcs -m
ipcrm -m shmid

// 2.创建共享内存
/* shmget() returns the identifier of the shared memory segment associated with the value of the argument key. */
/* size equal to the value of size rounded up to a multiple of PAGE_SIZE  */
/* PC_CREAT | IPC_EXCL  */
int shmget(key_t key, size_t size, int shmflg);

// 3.挂接共享内存
/* shmat() attaches the shared memory segment identified by shmid to the address space of the calling process.  */
/* If shmaddr is NULL, the system chooses a suitable (unused) address at which to attach the segment. */
/* If shmaddr isn’t NULL and SHM_RND is specified in shmflg, the attach occurs at the address equal 
   to shmaddr rounded down to the nearest multiple of SHMLBA. Otherwise shmaddr must be a page-aligned 
   address at which the attach occurs. */
void *shmat(int shmid, const void *shmaddr, int shmflg);

// 4.去关联共享内存
int shmdt(const void *shmaddr);

// 5.以cmd操纵共享内存
/* shmctl() performs the control operation specified by cmd on the shared memory segment whose 
   identifier is given in shmid. */
/* cmd: 
   IPC_RMID: Mark the segment to be destroyed.
   IPC_SET:  Write the values of some members of the shmid_ds structure pointed to by arg.buf 
             to the kernel data structure associated with this shared memory segment
   IPC_STAT: Copy information from the kernel data structure associated with shmid 
             into the shmid_ds structure pointed to by buf.
   */
int shmctl(int shmid, int cmd, struct shmid_ds *buf);

简单写一个demo，分为两个进程，一个是reader进程，一个writer进程。writer进程负责初始化创建一段共享内存，reader进程启动后不断轮询共享内存是否创建好，如果创建好就可以将数据读取出来。

//Makefile:

.PHONY:all
all:reader writer
 
writer:writer.c 
	gcc -o $@ $^
reader:reader.c 
	gcc -o $@ $^
 
.PHONY:clean
clean:
	rm -f reader writer

//common.h

#ifndef _COMMON_H__
#define _COMMON_H__
 
#include<stdio.h>
#include<sys/types.h>
#include<sys/ipc.h>
#include<sys/shm.h>
 
#define PATHNAME "zhihushm"
#define PROJ_ID  0x6666

#endif

//writer.c

#include "writer.h"
static char *shm_addr = NULL;
int create_shm()
{
	key_t key = ftok(PATHNAME, PROJ_ID);
	if(key < 0)
	{
		perror("ftok");
		return -1;
	}
	int shmid = shmget(key, 4096, IPC_CREAT | IPC_EXCL | 0666);
	if(shmid < 0)
	{
		perror("shmget");
		return -2;
	}
	shm_addr = shmat(shmid, NULL, 0);
	return 0;
}

void write_something()
{
	int i = 0;
	while (i < 4096)
	{
		shm_addr[0] = i % 128;
		i++;
		sleep(1);
	}	
}

int main()
{
	int ret = create_shm();
	if (ret < 0)
	{
		return -1;
	}
	write_something();
	return 0;
}

//reader.c

#include "writer.h"
static char *shm_addr = NULL;
int init()
{
	key_t key = ftok(PATHNAME, PROJ_ID);
	if(key < 0)
	{
		perror("ftok");
		return -1;
	}

	int shmid = shmget(key, 4096, IPC_CREAT);
	if(shmid < 0)
	{
		perror("shmget");
		return -2;
	}
	shm_addr = shmat(shmid, NULL, 0);
	return 0;
}

void read_something()
{
	int i = 0;
	while (i < 4096)
	{
		printf("we get %c\n", shm_addr[0]);
		sleep(1);
	}	
}

int main()
{
	int ret = init();
	while (ret < 0)
	{
		ret = init();
	}
	read_something();
	return 0;

}

k8s中pod间共享内存通信

好啦，现在请出这章的主角---多pod共享内存通信，而实现的关键就在各pod同时挂载/dev/shm即可，详细方案如下图：

这样，我们简单把上面的reader和writer两个进程分别容器化，再由k8s调度起来，并起多个reader pod和一个writer pod，来看看是否多个reader pod可以读到writer写的数据，以达到每一个Node上数据只有一份，多pod共享的目的。

先写两个Dockerfile:

//reader.Dockerfile

FROM centos

ADD  .  /home/

RUN chmod 777 /home/reader

ENTRYPOINT ["/home/reader"]  

//writer.Dockerfile

FROM centos

ADD  .  /home/

RUN chmod 777 /home/writer

ENTRYPOINT ["/home/writer"]

再build出镜像并运行：

docker build -t reader:v1 . -f reader.Dockerfile

docker build -t writer:v1 -f writer.Dockerfile

docker run -ti --name writer -v /dev/shm:/dev/shm --net=host --ipc=host writer:v1 bash

docker run -ti --name reader -v /dev/shm:/dev/shm --net=host --ipc=host reader:v1 bash

既然起两个docker看来是可以通信的了，那我们再写两个yaml文件并用k8s部署出来：

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: huhuxx
  name: huhuxx
spec:
  selector:
    matchLabels:
      app: huhuxx
  template:
    metadata:
      labels:
        app: huhuxx
    spec:
      hostNetwork: true
      hostIPC: true
      containers:
      - name: huhuxx
        image: reader:v4
        imagePullPolicy: Never
        volumeMounts:
        - mountPath: /dev/shm
          name: shm
      volumes:
      - name: shm
        hostPath:
          path: /dev/shm
          type: Directory

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: huhuxx
  name: huhuxx
spec:
  selector:
    matchLabels:
      app: huhuxx
  template:
    metadata:
      labels:
        app: huhuxx
    spec:
      hostNetwork: true
      hostIPC: true
      containers:
      - name: huhuxx
        image: writer:v4
        imagePullPolicy: Never
        volumeMounts:
        - mountPath: /dev/shm
          name: shm
      volumes:
      - name: shm
        hostPath:
          path: /dev/shm
          type: Directory

查看同一node上的，writer pod和其中两个reader pod: