openstack manila服务折腾笔记
国庆在家折腾了一把openstack manila,看了下现网还没有中文的manila入门介绍,于是决定写个笔记贴出来
一、manila简介
和我们传统存储服务器一样,openstack的存储也分为3种:块存储(cinder),对象存储(swift),文件存储(manila).manila 提供的是带有完整文件系统的存储服务.可以提供的类型有:nfs,cifs,glusterfs,hdfs等.云主机可以直接在系统里面挂载manila启动的实例
下面就是已经被虚拟机挂载的实例
[root@centos centos]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda1 xfs 10G 1.2G 8.9G 12% /
devtmpfs devtmpfs 900M 0 900M 0% /dev
10.254.0.7:/shares/share-f898d9b0-77b8-4231-9919-27b12e34cfa3 nfs4 976M 1.3M 908M 1% /mnt二 manila的组件
默认情况下manila有3个组件
manila-api 接受并验证REST请求，通过客户端及路由进行转发 manila-scheduler 决定共享创建的后端（后端及pool） manila-share 基于后端创建共享的服务 (类似于cinder volume 只是创建服务,并不一定提供服务)
以上3行 引用http://blog.csdn.net/Miss_yang_Cloud/article/details/51376480
二、manila-share的典型模式
(一)单独一个或若干个manila-share节点上通过lvm划分并格式化出一个ext4的卷通过external网络给云主机访问
典型架构如下
(二) 虚拟机模式
具体流程看第三段
架构图如下
(三)nas存储
manila share 节点通过管理网络连接nas存储的api管理口
用户云主机通过vrouter(可选)连接nas
三、虚拟机模式架构
笔者主要对openstack中vm as a service的模式有着比较大的兴趣,折腾完trove(虚拟机当数据库实例)和octavia(虚拟机当lb)之后花了几天时间折腾了一下manila的虚拟机+nfs模式
(一) 流程
系统会为manila创建一个manila networks,
用户创建manila 实例时会进行如下步骤,其中任何一个步骤出错都会导致创建实例出错
1 将manila网络接入和指定网络同一个vrouter下
2 调用nova 创建一个manila云主机,绑定安全组,创建cinder 卷
3 manila share会在5分钟内不停的 通过 manila networks 尝试ssh 连接manila虚拟机,
4 连接上之后进行nfs初始化
5 用户虚拟机 用过用户网络 到vrouter 到manila网络 实现对nfs的访问
(二)关键点的代码
1 检查manila网络有没有和用户指定网络有没有接入同一个vrouter
manila/share/drivers/service_instance.py
def _get_private_router(self, neutron_net_id, neutron_subnet_id):
"""Returns router attached to private subnet gateway."""
private_subnet = self.neutron_api.get_subnet(neutron_subnet_id)
if not private_subnet['gateway_ip']:
raise exception.ServiceInstanceException(
_('Subnet must have gateway.'))
private_network_ports = [p for p in self.neutron_api.list_ports(
network_id=neutron_net_id)]
for p in private_network_ports:
fixed_ip = p['fixed_ips'][0]
if (fixed_ip['subnet_id'] == private_subnet['id'] and
fixed_ip['ip_address'] == private_subnet['gateway_ip']):
private_subnet_gateway_port = p
break
else:
raise exception.ServiceInstanceException(
_('Subnet gateway is not attached to the router.'))
private_subnet_router = self.neutron_api.show_router(
private_subnet_gateway_port['device_id'])
return private_subnet_router先检查subnet 有没有gateway 有的话在检查有没有接入路由器
2 检查 manila share 节点 有没有打通网络
前面所述manila share 进程是可以直接ssh 进mannila vm的,因此需要打通manila share 节点和manila networks 这个租户网络
network/linux/interface.py
def plug(self, device_name, port_id, mac_address,
bridge=None, namespace=None, prefix=None):
"""Plugin the interface."""
ip = ip_lib.IPWrapper()
if prefix:
tap_name = device_name.replace(prefix, 'tap')
else:
tap_name = device_name.replace(self.DEV_NAME_PREFIX, 'tap')
if not ip_lib.device_exists(device_name,
namespace=namespace):
# Create ns_veth in a namespace if one is configured.
root_veth, ns_veth = ip.add_veth(tap_name, device_name,
namespace2=namespace)
ns_veth.link.set_address(mac_address)
else:
ns_veth = ip.device(device_name)
root_veth = ip.device(tap_name)
LOG.warning("Device %s already exists.", device_name)
root_veth.link.set_up()
ns_veth.link.set_up()先检查 是否存在这样的设备,如果不存在,就创建一个
最终会发现manila share 节点上生成一个 ns开头的tap设备
ns-7a203bcb-40: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.254.0.10 netmask 255.255.255.240 broadcast 10.254.0.15
inet6 fe80::f816:3eff:fe56:b6f7 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:56:b6:f7 txqueuelen 1000 (Ethernet)
RX packets 1190 bytes 193963 (189.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2236 bytes 135968 (132.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0通过他 我们可以直接在manila share 节点 访问 manila vm
3 主要创建过程
share/drivers/generic.py
def create_share(self, context, share, share_server=None):
"""Creates share."""
return self._create_share(
context, share,
snapshot=None,
share_server=share_server,
)
def _create_share(self, context, share, snapshot, share_server=None):
helper = self._get_helper(share)
server_details = share_server['backend_details']
volume = self._allocate_container(
self.admin_context, share, snapshot=snapshot)
volume = self._attach_volume(
self.admin_context, share, server_details['instance_id'], volume)
if not snapshot:
self._format_device(server_details, volume)
self._mount_device(share, server_details, volume)
export_locations = helper.create_exports(
server_details, share['name'])
return export_locations总体过程就是 :
(1)创建虚拟机
(2)创建volume
(3)挂载 volume
4 虚拟机创建后的操作
(1)manila share 在虚拟机被创建后不同的尝试ssh连接 虚拟机
share/drivers/service_instance.p
def _check_server_availability(self, instance_details):
t = time.time()
while time.time() - t < self.max_time_to_build_instance:
LOG.debug('Checking server availability.')
if not self._test_server_connection(instance_details):
time.sleep(5)
else:
return True
return False
def _test_server_connection(self, server):
try:
socket.socket().connect((server['ip'], 22))
LOG.debug('Server %s is available via SSH.',
server['ip'])
return True
except socket.error as e:
LOG.debug(e)
LOG.debug("Server %s is not available via SSH. Waiting...",
server['ip'])
return False如果规定时间连接不成功直接把 状态由creating变成error
def set_up_service_instance(self, context, network_info):
"""Finds or creates and sets up service vm.
:param context: defines context, that should be used
:param network_info: network info for getting allocations
:returns: dict with service instance details
:raises: exception.ServiceInstanceException
"""
instance_name = network_info['server_id']
server = self._create_service_instance(
context, instance_name, network_info)
instance_details = self._get_new_instance_details(server)
if not self._check_server_availability(instance_details):
e = exception.ServiceInstanceException(
_('%(conn_proto)s connection has not been '
'established to %(server)s in %(time)ss. Giving up.') % {
'conn_proto': self._INSTANCE_CONNECTION_PROTO,
'server': server['ip'],
'time': self.max_time_to_build_instance})
e.detail_data = {'server_details': instance_details}
raise e
return instance_detailsssh连上之后的操作
检查cinder volume 是否存在
def _is_device_file_available(self, server_details, volume):
"""Checks whether the device file is available"""
command = ['sudo', 'test', '-b', volume['mountpoint']]
self._ssh_exec(server_details, command)检测到存在之后进行格式化
def _format_device(self, server_details, volume):
"""Formats device attached to the service vm."""
self._is_device_file_available(server_details, volume)
command = ['sudo', 'mkfs.%s' % self.configuration.share_volume_fstype,
volume['mountpoint']]
self._ssh_exec(server_details, command)格式化后进行挂载
def _is_device_mounted(self, mount_path, server_details, volume=None):
"""Checks whether volume already mounted or not."""
log_data = {
'mount_path': mount_path,
'server_id': server_details['instance_id'],
}
if volume and volume.get('mountpoint', ''):
log_data['volume_id'] = volume['id']
log_data['dev_mount_path'] = volume['mountpoint']
msg = ("Checking whether volume '%(volume_id)s' with mountpoint "
"'%(dev_mount_path)s' is mounted on mount path '%(mount_p"
"ath)s' on server '%(server_id)s' or not." % log_data)
else:
msg = ("Checking whether mount path '%(mount_path)s' exists on "
"server '%(server_id)s' or not." % log_data)
LOG.debug(msg)
mounts_list_cmd = ['sudo', 'mount']
output, __ = self._ssh_exec(server_details, mounts_list_cmd)
mounts = output.split('\n')
for mount in mounts:
mount_elements = mount.split(' ')
if (len(mount_elements) > 2 and mount_path == mount_elements[2]):
if volume:
# Mount goes with device path and mount path
if (volume.get('mountpoint', '') == mount_elements[0]):
return True
else:
# Unmount goes only by mount path
return True
return False5 权限控制
虚拟机模式的权限控制主要依靠nfs 自己的权限控制来进行
manila access-allow demo-share1 ip 172.16.1.11share 节点 会ssh连接上 demo-share1所在虚拟机 的/etc/export加上一行
manila@ubuntu:~$ cat /etc/exports
/shares/share-f898d9b0-77b8-4231-9919-27b12e34cfa3 172.16.1.11(rw,sync,wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,anonuid=65534,anongid=65534,sec=sys,rw,no_root_squash,no_all_squash)上文说的安全组规则只是是通用规则,mannila 并不会会对每台虚拟机应用不同的安全组规则
share/drivers/helpers.py
def update_access(self, server, share_name, access_rules, add_rules,
delete_rules):
local_path = os.path.join(self.configuration.share_mount_path,
share_name)
out, err = self._ssh_exec(server, ['sudo', 'exportfs'])
# Recovery mode
if not (add_rules or delete_rules):
self.validate_access_rules(
access_rules, ('ip',),
(const.ACCESS_LEVEL_RO, const.ACCESS_LEVEL_RW))
hosts = self.get_host_list(out, local_path)
for host in hosts:
self._ssh_exec(server, ['sudo', 'exportfs', '-u',
':'.join((host, local_path))])
self._sync_nfs_temp_and_perm_files(server)
for access in access_rules:
rules_options = '%s,no_subtree_check'
if access['access_level'] == const.ACCESS_LEVEL_RW:
rules_options = ','.join((rules_options, 'no_root_squash'))
access_to = self._get_parsed_address_or_cidr(
access['access_to'])
self._ssh_exec(
server,
['sudo', 'exportfs', '-o',
rules_options % access['access_level'],
':'.join((access_to, local_path))])
self._sync_nfs_temp_and_perm_files(server)主要是通过 exportfs命令来进行 nfs权限管理
四 对manila目前状态的评价
堪用.在流程上没有大的bug,但是在安全,容错,高可用,文档,代码可读性等方面还有很多路要走.其实以上这些也是整个openstack项目的通病,但是反过来一想,如果没有这些问题,openstack还能吸引这么多人关注吗?