【分享】在MPSoC ZCU106单板上运行Docker

hankfu

发布于 2020-11-13 15:05:15

1.6K0

发布于 2020-11-13 15:05:15

文章被收录于专栏：hankhank

在MPSoC单板上运行Docker

作者

Hank FU 付汉杰 hankf@xilinx.com

测试环境

Xilinx ZCU106 单板 Xilinx VCU TRD2020.1

Linux 内核配置

根据文档Docker on Zynq Ultrascale+ (Xilinx Yocto Flow)，在PetaLinux工程的文件project-spec/meta-user/recipes-kernel/linux/linux-xlnx/user.cfg里添加下列配置项。

CONFIG_NAMESPACES=y
CONFIG_NET_NS=y
CONFIG_PID_NS=y
CONFIG_IPC_NS=y
CONFIG_UTS_NS=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_SCHED=y
CONFIG_CPUSETS=y
CONFIG_MEMCG=y
CONFIG_VETH=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
CONFIG_NETFILTER_XT_MATCH_IPVS=y
CONFIG_IP_NF_NAT=y
 
CONFIG_USER_NS=y
CONFIG_SECCOMP=y
CONFIG_CGROUP_PIDS=y
CONFIG_MEMCG_SWAP=y
CONFIG_MEMCG_SWAP_ENABLED=y
CONFIG_BLK_CGROUP=y
CONFIG_BLK_DEV_THROTTLING=y
CONFIG_CFQ_GROUP_IOSCHED=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_NET_CLS_CGROUP=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_IP_NF_TARGET_REDIRECT=y
CONFIG_IP_VS=y
CONFIG_IP_VS_NFCT=y
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_RR=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_VXLAN=y
CONFIG_INET_ESP=y
CONFIG_IPVLAN=y
CONFIG_MACVLAN=y
CONFIG_DUMMY=y
CONFIG_NF_NAT_FTP=y
CONFIG_NF_CONNTRACK_FTP=y
CONFIG_NF_NAT_TFTP=y
CONFIG_NF_CONNTRACK_TFTP=y
CONFIG_AUFS_FS=y
CONFIG_BTRFS_FS_POSIX_ACL=y
CONFIG_BLK_DEV_DM=y
CONFIG_DM_THIN_PROVISIONING=y
CONFIG_OVERLAY_FS=y
 
CONFIG_MD=y
CONFIG_NET_SCHED=y
CONFIG_NET_L3_MASTER_DEV=y
CONFIG_BRIDGE=m
 
# CONFIG_EXT2_FS is not set
CONFIG_EXT4_USE_FOR_EXT2=y

Linux 文件系统配置

在PetaLinux工程里选择Packagegroup-petalinux-ocicontainers。

启动命令

使用命令“dockerd &”可以启动Docker。运行之前，要先导出变量DOCKER_RAMDISK。

export DOCKER_RAMDISK=true

测试命令

使用命令“docker run --rm hello-world”可以运行一个简单容器。

常见问题

网络权限

单板要有访问网络权限，要能下载Docker的镜像。

root@vcu_trd:~# docker run --rm hello-world
Unable to find image 'hello-world:latest' locally
WARN[2020-11-11T09:47:14.257031091Z] Could not get operating system name: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory
WARN[2020-11-11T09:47:14.286697565Z] Error getting v2 registry: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:41143->[::1]:53: read: connection refused
INFO[2020-11-11T09:47:14.286851881Z] Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:41143->[::1]:53: read: connection refused
ERRO[2020-11-11T09:47:14.287032029Z] Handler for POST /v1.40/images/create returned error: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:41143->[::1]:53: read: connection refused
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:41143->[::1]:53: read: connection refused.
See 'docker run --help'.

存储空间

单板上可能缺乏足够存储空间。可以使用dockerd的选项data-root指定一个有足够空间的目录给dockerd使用。在ZCU106单板上，使用ramdisk时，最好进入/run/目录运行docker.

#ERROR Download failed: write /var/lib/docker/tmp/GetImageBlob091922966: no space left on device

docker启动失败

另外，docker启动过程中，可能没能启动containerd，出现下列错误。

root@vcu_trd:/run/test# docker run --rm hello-world
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.
root@vcu_trd:/run/test# docker-compose pull
ERROR: Couldn't connect to Docker daemon at http+docker://localhost - is it running?
If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

这时候可以杀死dockerd进程，再次启动dockerd。可以借助下列命令来快速杀死dockerd进程和启动dockerd。

export DOCKER_RAMDISK=true
echo -e "\nDOCKER_RAMDISK flag: $DOCKER_RAMDISK\n"   | tee -a  $docker_log_file

CONTAINERD_PID=$(ps -A | grep -v grep | grep containerd | grep -v .sh |awk '{printf $1}') 
DOCKERD_PID=$(ps -A | grep -v grep | grep dockerd | grep -v .sh |awk '{printf $1}') 
echo -e "containerd PID: $CONTAINERD_PID\n"   | tee -a  $docker_log_file
echo -e "dockerd PID: $DOCKERD_PID\n"   | tee -a  $docker_log_file

if [ ! "$CONTAINERD_PID" = "" ]; then
	echo -e "kill containerd PID: $CONTAINERD_PID\n"   | tee -a  $docker_log_file
	kill $CONTAINERD_PID
fi

if [ ! "$DOCKERD_PID" = "" ]; then
	echo -e "kill dockerd PID: $DOCKERD_PID\n"   | tee -a  $docker_log_file
	kill $DOCKERD_PID
	# Error starting daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid
	rm -f /var/run/docker.pid   | tee -a  $docker_log_file
fi

sleep 1
echo -e "\nStart Docker...\n"   | tee -a  $docker_log_file
ls -l /run/docker  | tee -a  $docker_log_file
mkdir -p /run/docker/data  | tee -a  $docker_log_file
dockerd --data-root /run/docker/data  | tee -a  $docker_log_file & 
sleep 12
ps -A | grep docker      | tee -a  $docker_log_file
ps -A | grep containerd   | tee -a  $docker_log_file

ls -l /var/run/docker.sock  | tee -a  $docker_log_file

文件/var/run/docker.pid已经存在

启动启动dockerd时，可能报错，说文件/var/run/docker.pid已经存在。重启动dockerd时，需要先删除文件/var/run/docker.pid。

# Error starting daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid

docker: Error response from daemon: OCI runtime create failed

PetaLinux编译的Linux启动后，已经启动dockerd。直接运行docker的hello-world ，出现下列错误。

root@vcu_trd:~# ps -A | grep -v grep | grep containerd
  937 ?        00:00:00 containerd
root@vcu_trd:~# ps -A | grep -v grep | grep dockerd
  926 ?        00:00:00 dockerd
root@vcu_trd:/run/test# docker run --rm hello-world
docker: Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:109: jailing process inside rootfs caused \\\"pivot_root invalid argument\\\"\"": unknown.
ERRO[0001] error waiting for container: context canceled

借助上面描述的脚本设置DOCKER_RAMDISK为true，并重启动dockerd，运行docker的hello-world成功。运行记录如下：

Run Docker hello-world...

Unable to find image 'hello-world:latest' locally
WARN[2020-11-12T01:47:00.103866590Z] Could not get operating system name: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory
latest: Pulling from library/hello-world
256ab8fe8778: Pulling fs layer
256ab8fe8778: Download complete
256ab8fe8778: Pull complete
Digest: sha256:8c5aeeb6a5f3ba4883347d3747a7249f491766ca1caa47e5da5dfcf6b9b717c0
Status: Downloaded newer image for hello-world:latest
INFO[2020-11-12T01:47:10.600246198Z] shim containerd-shim started                  address="/containerd-shim/moby/b506434ba15b4ba4896d909c1b0bd24fa9b788c6a75f26ea53da7b9679694fd8/shim.sock" debug=false pid=1618

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

INFO[2020-11-12T01:47:11.352948396Z] shim reaped                                   id=b506434ba15b4ba4896d909c1b0bd24fa9b788c6a75f26ea53da7b9679694fd8
INFO[2020-11-12T01:47:11.362202321Z] ignoring event                                module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
WARN[2020-11-12T01:47:11.453657929Z] b506434ba15b4ba4896d909c1b0bd24fa9b788c6a75f26ea53da7b9679694fd8 cleanup: failed to unmount IPC: umount /run/docker/data/containers/b506434ba15b4ba4896d909c1b0bd24fa9b788c6a75f26ea53da7b9679694fd8/mounts/shm, flags: 0x2: no such file or directory

pivot_root invalid argument: unknown

PetaLinux编译的Linux启动后，运行docker的hello-world ，出现下列错误“pivot_root invalid argument: unknown”。借助上面描述的脚本设置DOCKER_RAMDISK为true，并重启动dockerd，运行docker的hello-world成功。

root@vcu_trd:/run/test# export DOCKER_RAMDISK=true
root@vcu_trd:/run/test# echo -e "\nDOCKER_RAMDISK flag: $DOCKER_RAMDISK\n"   | tee -a  $docker_log_file
DOCKER_RAMDISK flag: true

root@vcu_trd:/run/test# docker run --rm hello-world
Unable to find image 'hello-world:latest' locally

latest: Pulling from library/hello-world
256ab8fe8778: Pull complete
Digest: sha256:8c5aeeb6a5f3ba4883347d3747a7249f491766ca1caa47e5da5dfcf6b9b717c0
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:109: jailing process inside rootfs caused \\\"pivot_root invalid argument\\\"\"": unknown.