服务拆分之基础设施拆分

服务拆分之基础设施拆分

Infrastructure unbundling of services

背景: 因历史原因, 前期多个服务共用一个rds实例和一个redis实例, 在实际使用中经常会因某一个服务异常导致rds或redis负载异常，进而影响其他服务造成雪崩。 故进行基础资源拆分来隔离风险。

Background : Due to historical reasons, multiple services share one RDS instance and one REDIS instance in the early stage. In actual use, it is common to cause RDS or REDIS load abnormalities due to one service exception, thus affecting other services and causing avalanches. Therefore, base resource splitting is performed to isolate risks.

本次拆分基于AWS平台 The split is based on AWS

创建原实例的只读副本实例

Create a read-only copy instance of the original instance

原理 principle

aws console-> rds console-> databases -> select rds-> action-> creat read replica

将k8s上pod副本数调为0

Set the number of pod replicas on K8S to 0

kubectl edit deployments test -n test

set spec.replicas: 0

验证只读实例和原实例记录行数是否相同

Verify that the read-only instance and the original instance record the same number of rows

不建议用schema统计 有误差 It is not recommended to use schema statistics with errors information_schema.tables 对于InnoDB表，table_rows行计数仅是大概估计值，不准确。 information_schema.tables For InnoDB tables, table_rows rows count is only approximate and not accurate.

用下面sql生成查询库下所有表的记录行数：

Use the following SQL to generate the number of record rows for all tables under the query library:

select concat(
    'select  count(*) from ',
    TABLE_SCHEMA, 
    '.',
    TABLE_name,
    ' union all'
) from information_schema.tables 
where TABLE_SCHEMA='test_db';

生成类似如下的查询sql,在原实例和只读实例进行查询 generate query SQL similar to the following for the original and read-only instances

select count(*) from table1
union all
select count(*) from table2
union all
select count(*) from table3
...

将只读rds实例提升为正常rds实例

Promote a read-only RDS instance to a normal RDS instance

待只读实例和源实例一致后将只读实例提升为正常实例 Promote a read-only instance to a normal instance after it is consistent with the source instance

aws console-> rds console-> databases -> select rds-> action-> promote

创建redis备份,并恢复一个新的redis

Create a Redis backup and restore a new Redis

aws console->redis->action->backup

aws console->redis->resotre

修改配置中心连接信息

Example Modify the connection information of the configuration center

datasource.host

redis.host

...

将k8s pod副本数恢复之前数量

Restore the number of k8S pod copies to previous number

kubectl edit deployments test -n test

set spec.replicas: x

将aws拆出来的资源 rds redis导入到现有 terraform中

Import the resource RDS Redis from AWS into the existing Terraform

参考如下 Refer to the following

Terraform反向导出

总结

to summarize

本次拆分可以保证数据0损失，因进行了k8s pod 副本数调整，会对对拆分的服务根据实际情况会有部分时间不可用，建议在服务访问量低时进行此操作

This split can ensure zero data loss. Because the number of K8S POD copies is adjusted, the split service may be unavailable for some time according to the actual situation. You are advised to perform this operation when the service traffic is low