[TOC]
Docker Runtime kubernetes 云原生 CNCF containerd.io containerd 容器运行时 WeiyiGeek 容器客户端工具
描述: containerd 是一个在任何系统上运行的简单守护程序。它提供了一个带有旋钮的最小配置,用于配置守护程序以及在必要时使用哪些插件。
在当前最新的 Containerd v1.6.2版本中其/etc/containerd/config.toml
配置文件项存在变化, 下面的文档中提供了常用配置以及CRI插件配置的说明。
官方参考地址: https://github.com/containerd/containerd/blob/main/docs/cri/config.md
$ cat /lib/systemd/system/containerd.service
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target
[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd
Type=notify
# 允许 containerd 及其运行时管理它所创建的容器的 cgroups
Delegate=yes
# 确保在 containerd 被关闭时仅终止 containerd 守护程序,而不终止任何子进程,如填充程序和容器
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
# Comment TasksMax if your systemd version does not supports it.Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target
描述:在 containerd 配置文件中,您将找到持久性和运行时存储位置的设置,以及各种 API 的 grpc、调试和指标地址。
containerd 在主机系统上还有两个不同的存储位置, 一个用于持久性数据,另一个用于运行时状态
。
root将用于存储容器的任何类型的持久性数据。快照、内容、容器和映像的元数据以及任何插件数据都将保留在此位置。根目录也为容器加载的插件提供了命名空间。每个插件都有自己的目录来存储数据。containerd本身实际上没有任何需要存储的持久性数据,其功能来自加载的插件。
/var/lib/containerd/
├── io.containerd.content.v1.content
│ ├── blobs
│ └── ingest
├── io.containerd.metadata.v1.bolt
│ └── meta.db
├── io.containerd.runtime.v2.task
│ ├── default
│ └── example
├── io.containerd.snapshotter.v1.btrfs
└── io.containerd.snapshotter.v1.overlayfs
├── metadata.db
└── snapshots
state将用于存储任何类型的临时数据, 套接字、pids、运行时状态、挂载点和其他在重新启动之间不得保留的插件数据存储在此位置。
/run/containerd
├── containerd.sock
├── debug.sock
├── io.containerd.runtime.v2.task
│ └── default
│ └── redis
│ ├── config.json
│ ├── init.pid
│ ├── log.json
│ └── rootfs
│ ├── bin
│ ├── data
│ ├── dev
│ ├── etc
│ ├── home
│ ├── lib
│ ├── media
│ ├── mnt
│ ├── proc
│ ├── root
│ ├── run
│ ├── sbin
│ ├── srv
│ ├── sys
│ ├── tmp
│ ├── usr
│ └── var
└── runc
└── default
└── redis
└── state.json
$ mkdir -vp /etc/containerd/ && containerd config default > /etc/containerd/config.toml
描述: 从 v1.4 版本开始 containerd 便支持懒拉取, Stargz Snapshotter
是使 containerd 能够处理 eStargz的插件,在 Kubernetes 上我们可以通过如下配置/etc/containerd/config.toml
启用了此功能。
Stargz Snapshotter插件(containerd-stargz-grpc二进制文件)也需要作为单独的进程运行, 其启动示例文件为:
# etc/systemd/system/stargz-snapshotter.service
[Unit]
Description=stargz snapshotter (Rootless)
PartOf=containerd.service
[Service]
Environment=PATH=/usr/local/bin:/sbin:/usr/sbin:/usr/local/elasticsearch-7.15.0/jdk/bin:/usr/local/elasticsearch-7.15.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
Environment=IPFS_PATH=/home/weiyigeek/.local/share/ipfs
ExecStart="/usr/local/bin/containerd-rootless-setuptool.sh" nsenter -- containerd-stargz-grpc -address "/run/user/1000/containerd-stargz-grpc/containerd-stargz-grpc.sock" -root "/home/weiyigeek/.local/share/containerd-stargz-grpc" -config "/home/weiyigeek/.config/containerd-stargz-grpc/config.toml"
ExecReload=/bin/kill -s HUP $MAINPID
RestartSec=2
Restart=always
Type=simple
KillMode=mixed
[Install]
WantedBy=default.target
温馨提示: 官方systemd示例地址(https://github.com/containerd/stargz-snapshotter/blob/v0.6.0/script/config/etc/systemd/system/stargz-snapshotter.service)
修改 /etc/containerd/config.toml 配置文件示例:
version = 2
# Plug stargz snapshotter into containerd Containerd recognizes stargz snapshotter through specified socket address.
[proxy_plugins]
[proxy_plugins.stargz]
type = "snapshot"
address = "/run/containerd-stargz-grpc/containerd-stargz-grpc.sock"
# Use stargz snapshotter through CRI
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "stargz"
disable_snapshot_annotations = false
描述: 在k8s的1.20版本发布之后,对外宣称在1.23.x不再使用dokershim作为默认的底层容器运行时,而是通过Container Runtime Interface(CRI)
使用containerd来作为容器运行时, 因此原来在docker中配置的个人仓库环境不再起作用,导致k8s配置pods时拉取镜像失败, 本节将进行演示如何在 containerd
配置从私有仓库拉取镜像。
Step 1.kubernetes 使用 containerd 拉取harbor仓库中镜像配置说明, 项目地址介绍: https://github.com/containerd/cri/blob/master/docs/registry.md
Step 2.containerd 的配置文件 (相当于docker 的 daemon.json)
$ vim /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://xlx9erfu.mirror.aliyuncs.com"] # 使用阿里镜像源到此为配置文件默认生成,之后为需要添加的内容
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.weiyigeek.top"] # 内部私有仓库配置
endpoint = ["https://harbor.weiyigeek.top"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.weiyigeek.to"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.weiyigeek.top".auth] # harbor 认证的账号密码 配置
username = "admin"
password = "Harbor12345"
auth = ""
identitytoken = ""
[plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.weiyigeek.top".tls] # harbor 证书认证配置
insecure_skip_verify = false # 是否跳过证书认证
ca_file = "/etc/containerd/harbor/ca.crt" # CA 证书
cert_file = "/etc/containerd/harbor/harbor.crt" # harbor 证书
key_file = "/etc/containerd/harbor/harbor.key" # harbor 私钥
Step 3.重载 systemd 的 daemon守护进程并重启containerd.service服务, 然后k8s集群便可正常从 harbor.weiyigeek.top 拉取镜像了。
systemctl daemon-reload && systemctl restart containerd.service
Step 4.虽然上面的方式可以使k8s直接拉取镜像,但是在利用 ctl命令 进行手动拉取镜像此时会报如下错误(巨坑-经过无数次失败测试,原本以为是CA证书签发的harbor证书问题),即使你在config.toml中配置insecure_skip_verify为true也是不行的。
# 错误信息
$ ctr -n k8s.io i pull harbor.weiyigeek.top/devops/jenkins-jnlp:3.13.8-alpine
INFO[0000] trying next host error="failed to do request: Head \"https://harbor.weiyigeek.top/v2/devops/jenkins-jnlp/manifests/3.13.8-alpine\": x509: certificate signed by unknown authority" host=harbor.weiyigeek.top
ctr: failed to resolve reference "harbor.weiyigeek.top/devops/jenkins-jnlp:3.13.8-alpine": failed to do request: Head "https://harbor.weiyigeek.top/v2/devops/jenkins-jnlp/manifests/3.13.8-alpine": x509: certificate signed by unknown authority
# 解决办法1.指定 -k 参数跳过证书校验。
$ ctr -n k8s.io i pull -k harbor.weiyigeek.top/devops/jenkins-jnlp:3.13.8-alpine
# 解决办法2.指定CA证书、Harbor相关证书文件路径。
$ ctr -n k8s.io i pull --tlscacert ca.crt --tlscert harbor.crt --tlskey harbor.key harbor.weiyigeek.top/devops/jenkins-jnlp:3.13.6-alpine
自定义配置 containerd 持久化目录和 containerd 运行时状态信息进行。
cat /etc/containerd/config.toml
version = 2
# containerd 持久化目录
root = "/var/lib/containerd"
# containerd 运行时状态信息
state = "/run/containerd"
oom_score = 0
$ vim /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"
# 或者
$ sed -i "s#k8s.gcr.io/pause#registry.cn-hangzhou.aliyuncs.com/google_containers/pause#g" /etc/containerd/config.toml
$ vim /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://xlx9erfu.mirror.aliyuncs.com"]
# 或者执行
sed -i '/registry.mirrors]/a\ \ \ \ \ \ \ \ [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]' /etc/containerd/config.toml
sed -i '/registry.mirrors."docker.io"]/a\ \ \ \ \ \ \ \ \ \ endpoint = ["https://xlx9erfu.mirror.aliyuncs.com"]' /etc/containerd/config.toml
$ vim /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
# 或者
$ sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml
于此同时, 我们需要在K8S的KubeletConfiguration资源清单中, 还必须将 cgroup 驱动程序配置为使用 “systemd”。
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: "systemd"
$ vim /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "overlayfs"
# 或者
sed -ri 's#snapshotter = "\w*"#snapshotter = "overlayfs"#' /etc/containerd/config.toml
设置运行时类, 下面的示例将自定义运行时注册到 containerd 中:
# 默认配置
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
base_runtime_spec = ""
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
runtime_engine = ""
runtime_root = ""
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
BinaryName = ""
CriuImagePath = ""
CriuPath = ""
CriuWorkPath = ""
IoGid = 0
IoUid = 0
NoNewKeyring = false
NoPivotRoot = false
Root = ""
ShimCgroup = ""
SystemdCgroup = true
# 自定义别名为crun的运行时
$ vim /etc/containerd/config.toml
version = 2
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "crun" # 注意别名
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
# crun: https://github.com/containers/crun
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.crun]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.crun.options]
BinaryName = "/usr/local/bin/crun"
# gVisor: https://gvisor.dev/
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.gvisor]
runtime_type = "io.containerd.runsc.v1"
# Kata Containers: https://katacontainers.io/
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
runtime_type = "io.containerd.kata.v2"
此外,还必须将以下RuntimeClass资源安装到具有该角色的群集中 cluster-admin
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: crun
handler: crun
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: gvisor
handler: gvisor
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: kata
handler: kata
要将运行时类应用于 pod,请设置.spec.runtimeClassName
apiVersion: v1
kind: Pod
spec:
runtimeClassName: crun
# 例如, 公共的 docker.io 镜像仓库为例
$ tree /etc/containerd/certs.d
/etc/containerd/certs.d
└── docker.io
└── hosts.toml
$ cat /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://docker.io"
[host."https://registry-1.docker.io"]
capabilities = ["pull", "resolve"]
# 例如, 私有的 hub.weiyigeek.top 镜像仓库配置
$ mkdir -vp /etc/containerd/certs.d/hub.weiyigeek.top/
$ touch /etc/containerd/certs.d/hub.weiyigeek.top/hosts.toml
$ vim /etc/containerd/certs.d/hub.weiyigeek.top/hosts.toml
server = "https://hub.weiyigeek.top"
[host."https://hub.weiyigeek.top"]
ca = "/path/to/ca.crt"
默认值为“共享”。虽然这在很大程度上是最需要的策略,但可以通过以下配置更改为“隔离”模式:
$ vim /etc/containerd/config.toml
version = 2
[plugins."io.containerd.metadata.v1.bolt"]
content_sharing_policy = "isolated"
补充说明: 在“隔离”模式下,还可以通过将标签添加到特定命名空间来仅共享该命名空间的内容。这将使其 Blob 在所有其他命名空间中都可用,即使内容共享策略设置为“隔离”也是如此。如果标签值设置为 除 以外的任何内容,则不会共享命名空间内容。
描述: Containerd 是一个工业级标准的容器运行时,它强调简单性、健壮性和可移植性。Containerd 可以在宿主机中管理完整的容器生命周期:容器镜像的传输和存储、容器的执行和管理、存储和网络等
学过Docker的童鞋都知道, 我们利用docker-cli与docker-compose工具能帮助快速的进行镜像和容器的相关操作。自从 Containerd 发布 1.5 以后,Kubernetes 的CRI接口使用 Containerd 来替代 dockershim 时, 我们便可以使用 nerdctl 工具(替代 docker cli) 配合 Containerd 的情况下基本已经可以替换掉 Docker 和 Docker Compose;
什么是nerdctl? nerdctl 是 与 Docker 兼容的CLI for Containerd,其支持Compose、Rootless、eStargz、OCIcrypt和IPFS,与docker命令行语法类似,所以其上手使用非常简单😊。
nerdctl 官方发布包包含两个安装版本:
Minimal: 仅包含 nerdctl 二进制文件以及 rootless 模式下的辅助安装脚本 Full: 看名字就能知道是个全量包,其包含了 Containerd、CNI、runc、BuildKit 等完整组件
项目地址: GitHub - containerd/nerdctl: Docker-compatible CLI for containerd, with support for Compose ( https://github.com/containerd/nerdctl )
最新nerdctl版本releases下载地址: https://github.com/containerd/nerdctl/releases
Step 1.从Github Release页面中下载适用于你系统的 Full 包,当前最新版本【2022年4月25日 11:22:59】为v.0.19.0。
# 断点与后台下载
# 全量包
wget -c -b https://github.com/containerd/nerdctl/releases/download/v0.19.0/nerdctl-full-0.19.0-linux-amd64.tar.gz
# 仅包含 nerdctl 二进制
wget -c -b https://github.com/containerd/nerdctl/releases/download/v0.19.0/nerdctl-0.19.0-linux-amd64.tar.gz
Step 2.下载完成后解压然后进行相应的安装操作。
# 1.安装仅包含 nerdctl 二进制文件包 (Minimal - 不推荐)
tar Cxzvvf /usr/local/bin nerdctl-0.19.0-linux-amd64.tar.gz
# -rwxr-xr-x root/root 27578368 2022-04-22 12:03 nerdctl
# -rwxr-xr-x root/root 21562 2022-04-22 12:02 containerd-rootless-setuptool.sh
# -rwxr-xr-x root/root 7032 2022-04-22 12:02 containerd-rootless.sh
cd /usr/local/bin && ./containerd-rootless-setuptool.sh install # 注意不能以root用户运行
# 2.安装 nerdctl 全量包 (Full - 推荐)
tar Cxzvvf /usr/local nerdctl-full-0.19.0-linux-amd64.tar.gz
# drwxr-xr-x 0/0 0 2022-04-22 12:16 bin/
# -rwxr-xr-x 0/0 25371420 2015-10-21 08:00 bin/buildctl
# -rwxr-xr-x 0/0 39651613 2015-10-21 08:00 bin/buildkitd
# drwxr-xr-x 0/0 0 2022-04-22 12:16 share/doc/nerdctl-full/
# -rw-r--r-- 0/0 1135 2022-04-22 12:16 share/doc/nerdctl-full/README.md
# -rw-r--r-- 0/0 5425 2022-04-22 12:16 share/doc/nerdctl-full/SHA256SUMS
# 3.初始化安装以全量包为例,如出现如下命令表示安装成功。
/usr/local/bin$ ./containerd-rootless-setuptool.sh install
# + systemctl --user enable containerd.service
# Created symlink /home/weiyigeek/.config/systemd/user/default.target.wants/containerd.service → /home/weiyigeek/.config/systemd/user/containerd.service.
# [INFO] Installed "containerd.service" successfully.
# [INFO] To control "containerd.service", run: `systemctl --user (start|stop|restart) containerd.service`
# [INFO] To run "containerd.service" on system startup automatically, run: `sudo loginctl enable-linger weiyigeek`
# [INFO] ------------------------------------------------------------------------------------------
# [INFO] Use `nerdctl` to connect to the rootless containerd.
# [INFO] You do NOT need to specify $CONTAINERD_ADDRESS explicitly.
Step 3.启动 containerd 和 buildkitd 服务。
# 启动 containerd && 验证服务状态
systemctl enable --now containerd && systemctl restart containerd && systemctl status containerd
# 启用 BuildKit 为了构建 Dockerfile,您需要使用以下命令启用 BuildKit。
/usr/local/bin$ CONTAINERD_NAMESPACE=default containerd-rootless-setuptool.sh install-buildkit-containerd
# [INFO] Creating "/home/weiyigeek/.config/systemd/user/default-buildkit.service"
# [INFO] Starting systemd unit "default-buildkit.service"
# + systemctl --user start default-buildkit.service
# + sleep 3
# + systemctl --user --no-pager --full status default-buildkit.service
# ● default-buildkit.service - BuildKit (Rootless)
# Loaded: loaded (/home/weiyigeek/.config/systemd/user/default-buildkit.service; disabled; vendor preset: enabled)
# Active: active (running) since Mon 2022-04-25 13:05:25 CST; 3s ago
# Main PID: 329750 (buildkitd)
# CGroup: /user.slice/user-1000.slice/user@1000.service/default-buildkit.service
# └─329750 buildkitd --addr=unix:///run/user/1000/buildkit-default/buildkitd.sock --root=/home/weiyigeek/.local/share/buildkit-default --containerd-worker-namespace=default
# Apr 25 13:05:25 node-2 systemd[327840]: Started BuildKit (Rootless).
# Apr 25 13:05:26 node-2 containerd-rootless-setuptool.sh[329750]: time="2022-04-25T13:05:26+08:00" level=info msg="found worker \"ffl4pd8j6x7fh6t9o85a70l11\", labels=map[org.mobyproject.buildkit.worker.containerd.namespace:default org.mobyproject.buildkit.worker.containerd.uuid:4f029882-9edc-4c20-ab49-0363abe0d40e org.mobyproject.buildkit.worker.executor:containerd org.mobyproject.buildkit.worker.hostname:node-2 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/386]"
# Apr 25 13:05:26 node-2 containerd-rootless-setuptool.sh[329750]: time="2022-04-25T13:05:26+08:00" level=info msg="found 1 workers, default=\"ffl4pd8j6x7fh6t9o85a70l11\""
# Apr 25 13:05:26 node-2 containerd-rootless-setuptool.sh[329750]: time="2022-04-25T13:05:26+08:00" level=warning msg="currently, only the default worker can be used."
# Apr 25 13:05:26 node-2 containerd-rootless-setuptool.sh[329750]: time="2022-04-25T13:05:26+08:00" level=info msg="running server on /run/user/1000/buildkit-default/buildkitd.sock"
# + systemctl --user enable default-buildkit.service
# Created symlink /home/weiyigeek/.config/systemd/user/default.target.wants/default-buildkit.service → /home/weiyigeek/.config/systemd/user/default-buildkit.service.
# [INFO] Installed "default-buildkit.service" successfully.
# [INFO] To control "default-buildkit.service", run: `systemctl --user (start|stop|restart) default-buildkit.service`
Step 4.查看nerdctl工具执行结果及其版本,至此安装完毕
~$ nerdctl --version
nerdctl version 0.19.0
描述: 在某一个用户执行时nerdctl命令时,我们可以其家目录中创建一个如下路径文件/home/weiyigeek/.config/nerdctl/nerdctl.toml
,该文件可配置包含nerdctl相关配置项目。
简述尝试
Step 1.镜像拉取与查看
$ nerdctl pull hello-world:latest
# docker.io/library/hello-world:latest: resolved |++++++++++++++++++++++++++++++++++++++|
# index-sha256:10d7d58d5ebd2a652f4d93fdd86da8f265f5318c6a73cc5b6a9798ff6d2b2e67: done |++++++++++++++++++++++++++++++++++++++|
# manifest-sha256:f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4: done |++++++++++++++++++++++++++++++++++++++|
# config-sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412: done |++++++++++++++++++++++++++++++++++++++|
# layer-sha256:2db29710123e3e53a794f2694094b9b4338aa9ee5c40b930cb8063a1be392c54: done |++++++++++++++++++++++++++++++++++++++|
# elapsed: 34.3s total: 6.9 Ki (205.0 B/s)
$ nerdctl image ls
# REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
# hello-world latest 10d7d58d5ebd 31 seconds ago linux/amd64 20.0 KiB 6.9 KiB
Step 2.运行拉取的 hello-world 镜像
# 创建并后台运行一个名为 hello-containerd 的容器
$ nerdctl run -d --privileged --name hello-containerd hello-world:latest
# 17cc212e69b2387b0aa1e4c4e679e41072eeeb14f909fff2acdb24d3c1033c0d
# 查看 hello-containerd 容器日志
$ nerdctl logs hello-containerd
# Hello from Docker!
# This message shows that your installation appears to be working correctly.
# .......
# Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/
# For more examples and ideas, visit:https://docs.docker.com/get-started/
Step 3.查看创建的容器相关信息以及删除容器。
# 容器查看
$ nerdctl ps -a
# CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
# 17cc212e69b2 docker.io/library/hello-world:latest "/hello" 2 minutes ago Exited (0) 2 minutes ago hello-containerd
# 容器详细信息
$ nerdctl inspect 17cc212e69b2
[
{
"Id": "17cc212e69b2387b0aa1e4c4e679e41072eeeb14f909fff2acdb24d3c1033c0d",
"Created": "2022-04-25T05:13:16.690822727Z",
"Path": "/hello",
"Args": null,
"State": { "Status": "exited", "Running": false, "Paused": false,"Pid": 330226, "ExitCode": 0,"FinishedAt": "2022-04-25T05:13:17.251972898Z"},
"Image": "docker.io/library/hello-world:latest",
"ResolvConfPath": "/home/weiyigeek/.local/share/nerdctl/1935db59/containers/default/17cc212e69b2387b0aa1e4c4e679e41072eeeb14f909fff2acdb24d3c1033c0d/resolv.conf",
"HostnamePath": "/home/weiyigeek/.local/share/nerdctl/1935db59/containers/default/17cc212e69b2387b0aa1e4c4e679e41072eeeb14f909fff2acdb24d3c1033c0d/hostname",
"LogPath": "/home/weiyigeek/.local/share/nerdctl/1935db59/containers/default/17cc212e69b2387b0aa1e4c4e679e41072eeeb14f909fff2acdb24d3c1033c0d/17cc212e69b2387b0aa1e4c4e679e41072eeeb14f909fff2acdb24d3c1033c0d-json.log",
"Name": "hello-containerd",
"Driver": "overlayfs",
"Platform": "linux",
"AppArmorProfile": "",
"Mounts": null,
"NetworkSettings": null
}
]
# 移除 hello-containerd 容器
$ nerdctl rm hello-containerd
hello-containerd
Step 4.删除指定 hello-world 镜像
$ nerdctl rmi hello-world
Untagged: docker.io/library/hello-world:latest@sha256:10d7d58d5ebd2a652f4d93fdd86da8f265f5318c6a73cc5b6a9798ff6d2b2e67
Deleted: sha256:e07ee1baac5fae6a26f30cabfe54a36d3402f96afda318fe0a96cec4ca393359
$ sudo nerdctl run -d --name nginx -p 80:80 nginx:alpine
# docker.io/library/nginx:alpine: resolved |++++++++++++++++++++++++++++++++++++++|
# index-sha256:5a0df7fb7c8c03e4158ae9974bfbd6a15da2bdfdeded4fb694367ec812325d31: done |++++++++++++++++++++++++++++++++++++++|
# manifest-sha256:efc09388b15fb423c402f0b8b28ca70c7fd20fe31f8d7531ae1896bbb4944999: done |++++++++++++++++++++++++++++++++++++++|
# config-sha256:51696c87e77e4ff7a53af9be837f35d4eacdb47b4ca83ba5fd5e4b5101d98502: done |++++++++++++++++++++++++++++++++++++++|
# layer-sha256:4071be97c256d6f5ab0e05ebdebcfec3d0779a5e199ad0d71a5fccba4b3e2ce4: done |++++++++++++++++++++++++++++++++++++++|
# layer-sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139: done |++++++++++++++++++++++++++++++++++++++|
# layer-sha256:5867cba5fcbd3ae827c5801e76d20e7dc91cbb626ac5c871ec6c4d04eb818b16: done |++++++++++++++++++++++++++++++++++++++|
# layer-sha256:4b639e65cb3ba47e77db93f93c6625a62ba1b9eec99160b254db380115ae009d: done |++++++++++++++++++++++++++++++++++++++|
# layer-sha256:061ed9e2b9762825b9869a899a696ce8b56e7e0ec1e1892b980969bf7bcda56a: done |++++++++++++++++++++++++++++++++++++++|
# layer-sha256:bc19f3e8eeb1bb75268787f8689edec9a42deda5cdecdf2f95b3c6df8eb57a48: done |++++++++++++++++++++++++++++++++++++++|
# elapsed: 33.7s total: 9.7 Mi (294.8 KiB/s)
# 98eb2f1d4639b173dc21c30e40be5f3e2c410e4ca325b6bd6bafcaab46ab6c11
# 查看创建的nginx容器
$ sudo nerdctl ps
# CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
# 98eb2f1d4639 docker.io/library/nginx:alpine "/docker-entrypoint.…" About an hour ago Up 0.0.0.0:80->80/tcp nginx
# 进入nginx容器内部
$ nerdctl exec -it nginx -- sh
/ # whoami
root
/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0@if4: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether ee:e6:a6:9a:07:48 brd ff:ff:ff:ff:ff:ff
inet 10.4.0.2/24 brd 10.4.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::ece6:a6ff:fe9a:748/64 scope link
valid_lft forever preferred_lft forever
/usr/share/nginx/html # hostname > index.html
/usr/share/nginx/html # ip addr >> index.html
# 访问创建的nginx容器 或者 浏览器访问 10.10.107.227 宿主机
$ curl -I localhost
Step 6.nginx 容器详细信息查看与容器停止、删除。
$ nerdctl inspect --format "{{ .Name }} {{ .Id }}" nginx
# nginx 98eb2f1d4639b173dc21c30e40be5f3e2c410e4ca325b6bd6bafcaab46ab6c11
$ nerdctl stop nginx
# nginx
$ nerdctl ps -a
# CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
# 98eb2f1d4639 docker.io/library/nginx:alpine "/docker-entrypoint.…" 2 hours ago Exited (0) 4 seconds ago 0.0.0.0:80->80/tcp nginx
$ nerdctl rm nginx
nginx
WeiyiGeek.nerdctl-containerd
温馨提示: 不知读者是否发现, 其子命令及其使用方法与docker客户端工具基本是类似的,所以前面说到其学习成本较低, 熟悉docker的朋友可以快速上手
描述 : nerdctl 是 containerd 的命令行界面的工具。
命令参数:
# Usage:
nerdctl [flags]
nerdctl [command]
# Management commands:
apparmor Manage AppArmor profiles
builder Manage builds
container Manage containers
image Manage images
ipfs Distributing images on IPFS
namespace Manage containerd namespaces
network Manage networks
system Manage containerd
volume Manage volumes
Commands:
build Build an image from a Dockerfile. Needs buildkitd to be running.
commit Create a new image from a container changes
completion Generate the autocompletion script for the specified shell
compose Compose
cp Copy files/folders between a running container and the local filesystem.
create Create a new container. Optionally specify "ipfs://" or "ipns://" scheme to pull image from IPFS.
events Get real time events from the server
exec Run a command in a running container
help Help about any command
history Show the history of an image
images List images
info Display system-wide information
inspect Return low-level information on objects.
kill Kill one or more running containers
load Load an image from a tar archive or STDIN
login Log in to a Docker registry
logout Log out from a Docker registry
logs Fetch the logs of a container. Currently, only containers created with `nerdctl run -d` are supported.
pause Pause all processes within one or more containers
port List port mappings or a specific mapping for the container
ps List containers
pull Pull an image from a registry. Optionally specify "ipfs://" or "ipns://" scheme to pull image from IPFS.
push Push an image or a repository to a registry. Optionally specify "ipfs://" or "ipns://" scheme to push image to IPFS.
restart Restart one or more running containers
rm Remove one or more containers
rmi Remove one or more images
run Run a command in a new container. Optionally specify "ipfs://" or "ipns://" scheme to pull image from IPFS.
save Save one or more images to a tar archive (streamed to STDOUT by default)
start Start one or more running containers
stats Display a live stream of container(s) resource usage statistics.
stop Stop one or more running containers
tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
top Display the running processes of a container
unpause Unpause all processes within one or more containers
update Update one or more running containers
version Show the nerdctl version information
wait Block until one or more containers stop, then print their exit codes.
# Flags:
-H, --H string Alias of --address (default "/run/containerd/containerd.sock")
-a, --a string Alias of --address (default "/run/containerd/containerd.sock")
--address string containerd address, optionally with "unix://" prefix [$CONTAINERD_ADDRESS] (default "/run/containerd/containerd.sock")
--cgroup-manager string Cgroup manager to use ("cgroupfs"|"systemd") (default "none")
--cni-netconfpath string cni config directory [$NETCONFPATH] (default "/home/weiyigeek/.config/cni/net.d")
--cni-path string cni plugins binary directory [$CNI_PATH] (default "/usr/local/libexec/cni")
--data-root string Root directory of persistent nerdctl state (managed by nerdctl, not by containerd) (default "/home/weiyigeek/.local/share/nerdctl")
--debug debug mode
--debug-full debug mode (with full output)
-h, --help help for nerdctl
--host string Alias of --address (default "/run/containerd/containerd.sock")
--hosts-dir strings A directory that contains <HOST:PORT>/hosts.toml (containerd style) or <HOST:PORT>/{ca.cert, cert.pem, key.pem} (docker style) (default [/home/weiyigeek/.config/containerd/certs.d,/home/weiyigeek/.config/docker/certs.d])
--insecure-registry skips verifying HTTPS certs, and allows falling back to plain HTTP
-n, --n string Alias of --namespace (default "default") , 支持名称空间。
--namespace string containerd namespace, such as "moby" for Docker, "k8s.io" for Kubernetes [$CONTAINERD_NAMESPACE] (default "default")
--snapshotter string containerd snapshotter [$CONTAINERD_SNAPSHOTTER] (default "overlayfs")
--storage-driver string Alias of --snapshotter (default "overlayfs")
-v, --version version for nerdctl
nerdctl 使用示例
$ nerdctl login -u weiyigeek index.docker.io
# Enter Password:
# WARNING: Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See
# https://docs.docker.com/engine/reference/commandline/login/#credentials-store
# Login Succeeded
$ nerdctl logout
# Removing login credentials for https://index.docker.io/v1/
# 拉取
nerdctl pull docker.io/library/nginx:alpine
nerdctl -n k8s.io pull docker.io/library/nginx:alpine
# 查看
nerdctl -n default image ls
nerdctl -n default inspect nginx:alpine
# 删除
nerdctl -n default rmi nginx:alpine
容器操作
# 创建运行容器
sudo nerdctl -n default run -d --name nginx -p 80:80 nginx:alpine
sudo nerdctl -n k8s.io run -d --privileged --restart=always --name rancher -p 80:80 -p 443:443 docker.io/cnrancher/rancher:v2.4.17-ent
# 查看容器
sudo nerdctl -n k8s.io ps -a
sudo nerdctl -n k8s.io inpsect rancher
# 进入容器内部
sudo nerdctl -n k8s.io exec rancher -- sh
# 停止容器
sudo nerdctl -n k8s.io stop rancher
# 删除容器
sudo nerdctl -n k8s.io rm rancher
描述: 我在验证环境中使用时发现特别吸引人的最新功能一个是图像快速拉取(eStargz), 另外一个是高速 rootless 模式。
描述: 在过去的一年中,可以在容器相关的工具(包括Kubernetes,contained,nerdctl,CRI-O,Podman,BuildKit,Kaniko
等)上使用eStargz镜像的懒拉取功能, 在2021年懒拉取将成为越来越普遍的镜像分发技术,并且正在把eStargz增加到OCI容器镜像规范当中。
从 1.4 版开始 Containered 支持懒拉取, Stargz Snapshotter 是使 containerd 能够处理eStargz的插件。 这是一种用于懒拉取的镜像分发技术。这使容器运行时无需等待整个镜像内容即可启动容器。取而代之的是,按需获取必要的内容块(例如单个文件)。这样可以将容器启动延迟从数十秒缩短到最佳的几秒。
WeiyiGeek.eStargz加速拉取
配置参考地址: https://github.com/containerd/nerdctl/blob/master/docs/rootless.md#stargz-snapshotter
我们将继续执行如下步骤, 以启用图像拉取的加速(延迟拉取), 即加速镜像拉取。
实践操作
1.安装与启用 Stargz 快照程序。
/usr/local/bin$ containerd-rootless-setuptool.sh install-stargz
[INFO] Creating "/home/weiyigeek/.config/systemd/user/stargz-snapshotter.service"
[INFO] Starting systemd unit "stargz-snapshotter.service"
+ systemctl --user start stargz-snapshotter.service
+ sleep 3
+ systemctl --user --no-pager --full status stargz-snapshotter.service
...........
[INFO] Installed "stargz-snapshotter.service" successfully.
[INFO] To control "stargz-snapshotter.service", run: `systemctl --user (start|stop|restart) stargz-snapshotter.service`
[INFO] Add the following lines to "/home/weiyigeek/.config/containerd/config.toml" manually, and then run `systemctl --user restart containerd.service`:
### BEGIN ###
[proxy_plugins]
[proxy_plugins."stargz"]
type = "snapshot"
address = "/run/user/1000/containerd-stargz-grpc/containerd-stargz-grpc.sock"
### END ###
[INFO] Set `export CONTAINERD_SNAPSHOTTER="stargz"` to use the stargz snapshotter.
2.准备一个包含以下 containerd 内容的配置, 为了安全我们可以指定一个用户, 例如使用 nerdctl 的用户的 UID 为 1000 的配置放置位置为/run/user/1000
。
# 确认方法当前用户UID
$ echo $UID
1000
# 在 containerd 配置文件中启用该插件
tee ~/.config/containerd/config.toml <<'EOF'
# Plug stargz snapshotter into containerd
# Containerd recognizes stargz snapshotter through specified socket address.
[proxy_plugins]
[proxy_plugins."stargz"]
type = "snapshot"
# NOTE: replace "1000" with your actual UID
address = "/run/user/1000/containerd-stargz-grpc/containerd-stargz-grpc.sock"
# Use stargz snapshotter through CRI
EOF
3.重新启动 restargz-snapshotter 与 containerd 以使配置更改生效。
systemctl daemon-reload
systemctl --user start restargz-snapshotter.service && systemctl --user restart containerd.service
systemctl status stargz-snapshotter.service && systemctl status containerd.service
● stargz-snapshotter.service - stargz snapshotter (Rootless)
Loaded: loaded (/home/weiyigeek/.config/systemd/user/stargz-snapshotter.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-04-25 16:48:56 CST; 7s ago
Main PID: 360664 (containerd-star)
CGroup: /user.slice/user-1000.slice/user@1000.service/stargz-snapshotter.service
└─360664 containerd-stargz-grpc -address /run/user/1000/containerd-stargz-grpc/containerd-stargz-grpc.sock -root /home/weiyigeek/.local/share/containerd-stargz-grpc -config /home/weiyigeek/.config/container>
Apr 25 16:48:56 node-2 systemd[358295]: Started stargz snapshotter (Rootless).
4.您可以选择执行以下命令,并指定使用--snapshotter=stargz
进行快速图像拉取。
export CONTAINERD_SNAPSHOTTER="stargz"
nerdctl run --snapshotter=stargz ghcr.io/stargz-containers/python:3.10-esgz python -c "print('hello')"
描述: Rootless模式允许你在没有特权的情况下运行容器,具有安全优势,但它的缺点是容器外的通信速度比特权 Rootful 模式慢。
但是我们可以通过使用 bypass4netns, 使得 Rootless 的容器外通信速度将比 Rootful 更快, 操作如下所示:
1.安装与启用 bypass4netns
/usr/local/bin$ containerd-rootless-setuptool.sh install-bypass4netnsd
2.您可以选择执行如下命令,使用快速无根功能启动容器和以下--label nerdctl/bypass4netns=true
参数快速图像拉取。
$ nerdctl run --label nerdctl/bypass4netns=true ghcr.io/stargz-containers/python:3.10-esgz python -c "print('hello')"
Step 1.准备 redis 数据库配置文件极其相关目录
mkdir -vp /app/redis/data
tee /app/redis/redis.conf <<'EOF'
# 绑定任意接口、服务端口、后台运行。
bind 0.0.0.0
port 6379
# 容器里必须设置为no
daemonize no
supervised auto
# redis服务pid进程文件名
pidfile "/var/run/redis.pid"
# 关闭保护模式,并配置使用密码访问
protected-mode no
requirepass "123456"
# echo -e "weiyigeek"|sha256sum
# requirepass 097575a79efcd7ea7b1efa2bcda78a4fc7cbd0820736b2f2708e72c3d21f8b61
# 数据文件保存路径,rdb/AOF文件也保存在这里
dir "/data"
# 日志文件记录文件(notice / verbose)
# logfile "/logs/redis.log"
# loglevel verbose
# 最大客户端连接数
maxclients 10000
# 客户端连接空闲多久后断开连接,单位秒,0表示禁用
timeout 60
tcp-keepalive 60
# Redis 数据持久化(rdb/aof)配置
# RDB 文件名
dbfilename "dump.rdb"
# 数据自动保存脚本条件例如300s中有10key发生变化
save 300 10
save 60 10000
# 对RDB文件进行压缩,建议以(磁盘)空间换(CPU)时间。
rdbcompression yes
# 版本5的RDB有一个CRC64算法的校验和放在了文件的最后。这将使文件格式更加可靠。
rdbchecksum yes
# RDB自动触发策略是否启用,默认为yes
rdb-save-incremental-fsync yes
# AOF开启
appendonly yes
# AOF文件名
appendfilename "appendonly.aof"
# 可选值 always, everysec,no,建议设置为everysec
appendfsync everysec
# Redis风险命令重命名
# rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52
rename-command FLUSHDB b840fc02d524045429941cc15f59e41cb7be6c53
rename-command FLUSHALL b840fc02d524045429941cc15f59e41cb7be6c54
rename-command EVAL b840fc02d524045429941cc15f59e41cb7be6c55
rename-command DEBUG b840fc02d524045429941cc15f59e41cb7be6c56
# rename-command SHUTDOWN SHUTDOWN
EOF
Step 2.执行如下命令进行快速创建容器并运行redis服务。
$ nerdctl run -d -p 6379:6379 \
-v /app/redis/redis.conf:/etc/redis/redis.conf \
-v /app/redis/data:/data \
--name redis-server redis:6.2.6-alpine3.15 redis-server /etc/redis/redis.conf
5e854a58087ae1bba5a661b2941474560cbecc37f54c7f4e7a28afbaed6aebf0
Step 3.查看创建的容器并验证redis服务是否正常。
$ nerdctl ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5e854a58087a docker.io/library/redis:6.2.6-alpine3.15 "docker-entrypoint.s…" 42 seconds ago Up 0.0.0.0:6379->6379/tcp redis-server
$ nerdctl logs redis-server
1:C 25 Apr 2022 13:22:59.597 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 25 Apr 2022 13:22:59.597 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 25 Apr 2022 13:22:59.597 # Configuration loaded
# 如下返回,表示 redis 服务状态是正常的
$ nerdctl exec -it redis-server redis-cli -a 123456 ping
PONG
$ nerdctl exec -it redis-server redis-cli -a 123456 info
# Server
redis_version:6.2.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:63421500bb103677
redis_mode:standalone
os:Linux 5.4.0-96-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
Step 4.这里我们直接用telnet工具连接创建的redis容器中的redis数据库服务。
$ telnet 10.10.107.227 6379
Trying 10.10.107.227...
Connected to 10.10.107.227.
Escape character is '^]'.
auth 123456 # 认证
+OK
ping # 服务验证
+PONG
set name weiyigeek # 设置 字符串 类型的key
+OK
get name # 获取 key 的值
$9
weiyigeek
错误1.安装nerdctl(Minimal)时执行containerd-rootless-setuptool.sh脚本安装时报[ERROR] Refusing to install rootless containerd as the root user
错误
解决办法: 请切换到普通用户执行。
错误2.安装nerdctl(Minimal)时执行containerd-rootless-setuptool.sh脚本安装时报containerd-rootless-setuptool.sh: 110: rootlesskit: Permission denied
错误
/usr/local/bin$ containerd-rootless-setuptool.sh install
[INFO] Checking RootlessKit functionality
/usr/local/bin/containerd-rootless-setuptool.sh: 110: rootlesskit: Permission denied
问题原因: 由于未安装 rootlesskit 相关依赖工具
解决办法: 执行如下命令进行安装操作 apt install -y rootlesskit rootlessctl
错误3.安装nerdctl (full) 时执行containerd-rootless-setuptool.sh脚本安装时报exec: "newuidmap": executable file not found in $PATH
错误
./containerd-rootless-setuptool.sh install
[INFO] Checking RootlessKit functionality
[rootlesskit:parent] error: failed to setup UID/GID map: newuidmap 328299 [0 1000 1 1 100000 65536] failed: : exec: "newuidmap": executable file not found in $PATH
问题原因: newuidmap 在没有安装的环境下会出现上述错误,所以执行apt命令安装uidmap软件包。
解决版本: sudo apt install uidmap