一、概述二、MinIO 与 Ceph 对比1)架构设计对比2)数据一致性对比3)部署和管理对比4)生态系统和兼容性对比三、前期准备1)部署 docker2)部署 docker-compose四、创建网络五、MinIO 编排部署1)下载 MinIO 安装包2)配置3)启动脚本 bootstrap.sh4)构建镜像 Dockerfile5)编排 docker-compose.yaml6)开始部署7)简单测试验证8)web 访问
MinIO 是一个开源的对象存储服务器,它兼容Amazon S3(Simple Storage Service)API。它被设计用于构建分布式存储架构,提供高可用性、高性能和可扩展的对象存储解决方案。
下面是MinIO的一些主要特点和功能:
总的来说,MinIO是一个开源的高性能对象存储服务器,适用于构建分布式存储系统。它具有高可用性、可扩展性和数据保护机制,兼容Amazon S3 API,使其与现有的S3生态系统和工具集成无缝。MinIO在大数据、云计算和容器化环境中广泛应用,为应用程序提供了可靠、高效的对象存储服务。
官方文档:MinIO Object Storage for Kubernetes — MinIO Object Storage for Kubernetes ,中文文档:MinIO Object Storage for Kubernetes — MinIO Object Storage for Kubernetes
MinIO和Ceph都是流行的开源存储解决方案,它们在对象存储领域有不同的特点和适用场景。下面是MinIO和Ceph的对比:
综上所述,MinIO适用于简单、轻量级的对象存储需求,注重高性能和易用性。它适合中小规模部署,并且与Amazon S3兼容,易于与现有的S3生态系统集成。
无论选择MinIO还是Ceph,都需要仔细评估其与特定应用和环境的兼容性、性能需求、管理复杂性和可扩展性,以确保选择的解决方案能够满足实际需求并提供可靠的存储服务。
# 安装yum-config-manager配置工具
yum -y install yum-utils
# 建议使用阿里云yum源:(推荐)
#yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 安装docker-ce版本
yum install -y docker-ce
# 启动并开机启动
systemctl enable --now docker
docker --version
curl -SL https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose --version
# 创建,注意不能使用hadoop_network,要不然启动hs2服务的时候会有问题!!!
docker network create hadoop-network
# 查看
docker network ls
wget https://dl.min.io/server/minio/release/linux-amd64/minio
这里部署的是伪集群,但是需要的磁盘还是那么多,下面就是挂载磁盘的步骤:
### 1、格式化
mkfs.ext4 /dev/sdb
mkfs.ext4 /dev/sdc
mkfs.ext4 /dev/sdd
mkfs.ext4 /dev/sde
mkfs.ext4 /dev/sdf
mkfs.ext4 /dev/sdg
mkfs.ext4 /dev/sdh
mkfs.ext4 /dev/sdi
mkfs.ext4 /dev/sdj
mkfs.ext4 /dev/sdk
mkfs.ext4 /dev/sdl
mkfs.ext4 /dev/sdm
### 2、创建挂载目录
mkdir /opt/apache/docker-compose-minio/data/minio-node{1..3}/data{1..4}
### 3、挂载
# minio-node1
mount /dev/sdb /opt/apache/docker-compose-minio/data/minio-node1/data1
mount /dev/sdc /opt/apache/docker-compose-minio/data/minio-node1/data2
mount /dev/sdd /opt/apache/docker-compose-minio/data/minio-node1/data3
mount /dev/sde /opt/apache/docker-compose-minio/data/minio-node1/data4
# minio-node2
mount /dev/sdf /opt/apache/docker-compose-minio/data/minio-node2/data1
mount /dev/sdg /opt/apache/docker-compose-minio/data/minio-node2/data2
mount /dev/sdh /opt/apache/docker-compose-minio/data/minio-node2/data3
mount /dev/sdi /opt/apache/docker-compose-minio/data/minio-node2/data4
# minio-node3
mount /dev/sdj /opt/apache/docker-compose-minio/data/minio-node3/data1
mount /dev/sdk /opt/apache/docker-compose-minio/data/minio-node3/data2
mount /dev/sdl /opt/apache/docker-compose-minio/data/minio-node3/data3
mount /dev/sdm /opt/apache/docker-compose-minio/data/minio-node3/data4
### 4、持久化配置
# minio-node1
echo "/dev/sdb /opt/apache/docker-compose-minio/data/minio-node1/data1 ext4 defaults 0 0" >> /etc/fstab
echo "/dev/sdc /opt/apache/docker-compose-minio/data/minio-node1/data2 ext4 defaults 0 0" >> /etc/fstab
echo "/dev/sdd /opt/apache/docker-compose-minio/data/minio-node1/data3 ext4 defaults 0 0" >> /etc/fstab
echo "/dev/sde /opt/apache/docker-compose-minio/data/minio-node1/data4 ext4 defaults 0 0" >> /etc/fstab
# minio-node2
echo "/dev/sdf /opt/apache/docker-compose-minio/data/minio-node2/data1 ext4 defaults 0 0" >> /etc/fstab
echo "/dev/sdg /opt/apache/docker-compose-minio/data/minio-node2/data2 ext4 defaults 0 0" >> /etc/fstab
echo "/dev/sdh /opt/apache/docker-compose-minio/data/minio-node2/data3 ext4 defaults 0 0" >> /etc/fstab
echo "/dev/sdi /opt/apache/docker-compose-minio/data/minio-node1/data4 ext4 defaults 0 0" >> /etc/fstab
# minio-node3
echo "/dev/sdj /opt/apache/docker-compose-minio/data/minio-node3/data1 ext4 defaults 0 0" >> /etc/fstab
echo "/dev/sdk /opt/apache/docker-compose-minio/data/minio-node3/data2 ext4 defaults 0 0" >> /etc/fstab
echo "/dev/sdl /opt/apache/docker-compose-minio/data/minio-node3/data3 ext4 defaults 0 0" >> /etc/fstab
echo "/dev/sdm /opt/apache/docker-compose-minio/data/minio-node3/data4 ext4 defaults 0 0" >> /etc/fstab
#!/bin/bash
source /etc/profile
# 在三台机器上都执行该文件,即以分布式的方式启动了MINIO
# --address "0.0.0.0:9000" 挂载9001端口为api端口(如Java客户端)访问的端口
# --console-address ":9000" 挂载9000端口为web端口;
/opt/apache/minio/minio server --address 0.0.0.0:9000 --console-address 0.0.0.0:9001 --config-dir /etc/minio \
http://minio-node1/opt/apache/minio/data/export1 \
http://minio-node1/opt/apache/minio/data/export2 \
http://minio-node1/opt/apache/minio/data/export3 \
http://minio-node1/opt/apache/minio/data/export4 \
http://minio-node2/opt/apache/minio/data/export1 \
http://minio-node2/opt/apache/minio/data/export2 \
http://minio-node2/opt/apache/minio/data/export3 \
http://minio-node2/opt/apache/minio/data/export4 \
http://minio-node3/opt/apache/minio/data/export1 \
http://minio-node3/opt/apache/minio/data/export2 \
http://minio-node3/opt/apache/minio/data/export3 \
http://minio-node3/opt/apache/minio/data/export4 >/opt/apache/minio/logs/minio_server.log &
tail -f /opt/apache/minio/logs/minio_server.log
Minio默认9000端口,在配置文件中加入–address “127.0.0.1:9029” 可更改端口
【温馨提示】磁盘大小必须>1G,这里我添加的是4*2G的盘
FROM registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/centos-jdk:7.7.1908
# 创建日志存储目录
RUN mkdir -p /opt/apache/minio/logs
# 分别在三个节点上创建存储目录
RUN mkdir -p /opt/apache/minio/data/export{1..3}
# 创建配置目录
RUN mkdir -p /etc/minio
# 账号密码
ENV MINIO_ROOT_USER=admin
ENV MINIO_ROOT_PASSWORD=admin123456
# copy minio
COPY minio /opt/apache/minio/
# copy minio client mc
COPY mc /opt/apache/minio/
RUN ln -s /opt/apache/minio/mc /usr/local/sbin/mc
# copy bootstrap.sh
COPY bootstrap.sh /opt/apache/
RUN chmod +x /opt/apache/bootstrap.sh
WORKDIR /opt/apache
开始构建镜像
docker build -t registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/minio:20230619 . --no-cache --progress=plain
# 为了方便小伙伴下载即可使用,我这里将镜像文件推送到阿里云的镜像仓库
docker push registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/minio:20230619
### 参数解释
# -t:指定镜像名称
# . :当前目录Dockerfile
# -f:指定Dockerfile路径
# --no-cache:不缓存
version: '3'
services:
minio-node1:
image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/minio:20230619
container_name: minio-node1
hostname: minio-node1
restart: always
privileged: true
env_file:
- .env
expose:
- "${MinIO_PORT}"
ports:
- "${MinIO_HTTP_PORT}"
volumes:
- /opt/apache/docker-compose-minio/data/minio-node1/data1:/opt/apache/minio/data/export1
- /opt/apache/docker-compose-minio/data/minio-node1/data2:/opt/apache/minio/data/export2
- /opt/apache/docker-compose-minio/data/minio-node1/data3:/opt/apache/minio/data/export3
- /opt/apache/docker-compose-minio/data/minio-node1/data4:/opt/apache/minio/data/export4
command: ["sh","-c","/opt/apache/bootstrap.sh"]
networks:
- hadoop-network
healthcheck:
test: ["CMD-SHELL", "netstat -tnlp|grep :${MinIO_PORT} || exit 1"]
interval: 10s
timeout: 10s
retries: 5
minio-node2:
image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/minio:20230619
container_name: minio-node2
hostname: minio-node2
restart: always
privileged: true
env_file:
- .env
expose:
- "${MinIO_PORT}"
ports:
- "${MinIO_HTTP_PORT}"
volumes:
- /opt/apache/docker-compose-minio/data/minio-node2/data1:/opt/apache/minio/data/export1
- /opt/apache/docker-compose-minio/data/minio-node2/data2:/opt/apache/minio/data/export2
- /opt/apache/docker-compose-minio/data/minio-node2/data3:/opt/apache/minio/data/export3
- /opt/apache/docker-compose-minio/data/minio-node2/data4:/opt/apache/minio/data/export4
command: ["sh","-c","/opt/apache/bootstrap.sh"]
networks:
- hadoop-network
healthcheck:
test: ["CMD-SHELL", "netstat -tnlp|grep :${MinIO_PORT} || exit 1"]
interval: 10s
timeout: 10s
retries: 5
minio-node3:
image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/minio:20230619
container_name: minio-node3
hostname: minio-node3
restart: always
privileged: true
env_file:
- .env
expose:
- "${MinIO_PORT}"
ports:
- "${MinIO_HTTP_PORT}"
volumes:
- /opt/apache/docker-compose-minio/data/minio-node3/data1:/opt/apache/minio/data/export1
- /opt/apache/docker-compose-minio/data/minio-node3/data2:/opt/apache/minio/data/export2
- /opt/apache/docker-compose-minio/data/minio-node3/data3:/opt/apache/minio/data/export3
- /opt/apache/docker-compose-minio/data/minio-node3/data4:/opt/apache/minio/data/export4
command: ["sh","-c","/opt/apache/bootstrap.sh"]
networks:
- hadoop-network
healthcheck:
test: ["CMD-SHELL", "netstat -tnlp|grep :${MinIO_PORT} || exit 1"]
interval: 10s
timeout: 10s
retries: 5
# 连接外部网络
networks:
hadoop-network:
external: true
.env 文件内容
MinIO_PORT=9000
MinIO_HTTP_PORT=9001
# --project-name指定项目名称,默认是当前目录名称
docker-compose -f docker-compose.yaml up -d
# 查看
docker-compose -f docker-compose.yaml ps
# 卸载
docker-compose -f docker-compose.yaml down
# 登录容器内测试
docker exec -it minio-node1 bash
# 登录,密文输入(推荐)
[root@minio-node1 apache]# mc config host add minio http://localhost:9000
Enter Access Key: admin
Enter Secret Key: 输入密码
Added `minio` successfully.
# 查看集群信息
mc admin info minio
docker-compose -f docker-compose.yaml ps
账号/密码:admin/admin123456