数据传输 | dtle 使用初探
作者：马莹乐
爱可生研发团队成员，负责 mysql 中间件和数据库管理平台的测试。擅长找茬（测试技术爱好者），欢迎大家试用 dtle~
本文来源：原创投稿
*爱可生开源社区出品，原创内容未经授权不得随意使用，转载请联系小编并注明来源。
dtle 是一款爱可生 MySQL 开源数据传输中间件，此文简要介绍初步使用方法。
项目地址：https://github.com/actiontech/dtle
一、下载安装
环境说明：
- 准备三台主机，主机名分别为：node4、node5、node6
- 三台均为 agent，另将 node4 和 node5 作为 manager
1.1 下载
下载 dtle 最新的 release rpm 包
地址：https://github.com/actiontech/dtle/releases
dtle 安装包下载：
wget https://github.com/actiontech/dtle/releases/download/v2.19.11.0/dtle-2.19.11.0.x86_64.rpm
1.2 安装
安装步骤：
https://actiontech.github.io/dtle-docs-cn/4/4.0_installation.html
分别将 node4，node5 和 node6 的 dtle 安装完成。
1.3 启动 dtle
dtle 安装完成后，并未启动，需要先配置 dtle 的 conf 文件，再启动。
1.4 配置 dtle.conf
配置相关说明：
https://actiontech.github.io/dtle-docs-cn/4/4.1nodeconfiguration.html
dtle.conf 准备：以下配置文件替分别替换原有 dtle.conf（/opt/dtle/etc/dtle/dtle.conf）
node4 的 dtle.conf
# Setup data dir
data_dir = "/opt/dtle/data"
log_level = "DEBUG"
log_file = "/opt/dtle.log"
#log_to_stdout = true
bind_addr = "172.100.9.4"
# Modify our port to avoid a collision with server
ports {
http = 8190
}
# Enable the manager
manager {
enabled = true
# Self-elect, should be 3 or 5 for production,
# Addresses to attempt to join when the server starts.
join = [ "172.100.9.4","172.100.9.5" ]
}
# Enable the agent
agent {
enabled = true
managers = ["172.100.9.4:8191","172.100.9.5:8191"]
}
metric{
collection_interval = "15s"
publish_allocation_metrics = "true"
publish_node_metrics = "true"
}
addresses {
http = "172.100.9.4"
rpc = "172.100.9.4"
serf = "172.100.9.4"
}
advertise {
http = "172.100.9.4"
rpc = "172.100.9.4"
serf = "172.100.9.4"
}
node5 的 dtle.conf
# Setup data dir
data_dir = "/opt/dtle/data"
log_level = "DEBUG"
log_file = "/opt/dtle.log"
#log_to_stdout = true
bind_addr = "172.100.9.5"
# Modify our port to avoid a collision with server
ports {
http = 8190
}
# Enable the manager
manager {
enabled = true
# Self-elect, should be 3 or 5 for production,
# Addresses to attempt to join when the server starts.
join = [ "172.100.9.4","172.100.9.5" ]
}
# Enable the agent
agent {
enabled = true
managers = ["172.100.9.4:8191","172.100.9.5:8191"]
}
metric{
collection_interval = "15s"
publish_allocation_metrics = "true"
publish_node_metrics = "true"
}
addresses {
http = "172.100.9.5"
rpc = "172.100.9.5"
serf = "172.100.9.5"
}
advertise {
http = "172.100.9.5"
rpc = "172.100.9.5"
serf = "172.100.9.5"
}
node6 的 dtle.conf
# Setup data dir
data_dir = "/opt/dtle/data"
log_level = "DEBUG"
log_file = "/opt/dtle.log"
#log_to_stdout = true
bind_addr = "172.100.9.6"
# Modify our port to avoid a collision with server
ports {
http = 8190
}
# Enable the manager
manager {
enabled = false
# Self-elect, should be 3 or 5 for production,
# Addresses to attempt to join when the server starts.
join = [ "172.100.9.4","172.100.9.5" ]
}
# Enable the agent
agent {
enabled = true
managers = ["172.100.9.4:8191","172.100.9.5:8191"]
}
metric{
collection_interval = "15s"
publish_allocation_metrics = "true"
publish_node_metrics = "true"
}
addresses {
http = "172.100.9.6"
rpc = "172.100.9.6"
serf = "172.100.9.6"
}
advertise {
http = "172.100.9.6"
rpc = "172.100.9.6"
serf = "172.100.9.6"
}
1.5 启动 dtle
启动命令说明：
https://actiontech.github.io/dtle-docs-cn/4/4.2_command.html
1.6 查看 3 个节点 dtle 进程
node 4
node 5
node 6
1.7 尝试查看 dtle 架构是否符合预期
命令说明：
https://actiontech.github.io/dtle-docs-cn/4/4.2_command.html
其中，members 展示的是集群中 manager 节点的信息。
HTTP API：
https://actiontech.github.io/dtle-docs-cn/4/4.4httpapi.html
输出内容为 json 格式，可以安装 jq，使之格式化展示。
二、源端/目标端准备
MySQL 迁移用户准备。
2.1 创建用户
源端用户创建，
目标端用户创建，
有关源端/目标端用户的最小权限说明：
https://actiontech.github.io/dtle-docs-cn/4/4.5mysqluser_privileges.html
2.2 数据准备
源端数据准备，
目标端测试前库表情况，
2.3 创建 job
创建全量 + 增量，表级别迁移 job，分别使用 node5 和 node6 节点的 dtle 作为源端和目标端迁移的 agent。
job 配置相关参数：
https://actiontech.github.io/dtle-docs-cn/4/4.3jobconfiguration.html
2.3.1 准备 job.json
job.json
{
"Name":"have_a_try",
"Failover":false,
"Orders":[],
"Type":"synchronous",
"Tasks":[
{
"Type":"Src",
"NodeId":"ee97dc49-85ed-febc-4d3c-cfbfa87f46bd",
"Config":{
"Gtid":"",
"DropTableIfExists":false,
"SkipCreateDbTable":false,
"ApproveHeterogeneous":true,
"ReplChanBufferSize":"600",
"ChunkSize":"2000",
"ExpandSyntaxSupport":false,
"MsgBytesLimit":"20480",
"MsgsLimit":"65536",
"BytesLimit":"67108864",
"GroupMaxSize":"1",
"GroupTimeout":"100",
"SqlFilter":[],
"ReplicateDoDb":[
{
"TableSchema":"test",
"Tables":[
{
"TableName":"test1"
}
]
}
],
"ConnectionConfig":{
"Host":"172.100.9.1",
"Port":"3306",
"User":"src_test",
"Password":"test"
}
}
},
{
"Type":"Dest",
"NodeId":"e623aedd-5c37-da67-4ddf-1a82ce1ac298",
"Config":{
"ParallelWorkers":"1",
"ConnectionConfig":{
"Host":"172.100.9.2",
"Port":"3306",
"User":"dest_test",
"Password":"test"
}
}
}
]
}
其中 NodeId 可通过命令或者 HTTP API 查询得知。
2.3.2 创建 job
创建 job 试用 HTTP API：
https://actiontech.github.io/dtle-docs-cn/4/4.4httpapi.html
创建完成后查询 job 状态为 running。
job 详细信息可使用 HTTP API 查询，示例：
[
{
"CreateIndex":109,
"ID":"ba13f3e1-0630-f141-4c60-a1f36c61cdb3",
"JobModifyIndex":2372,
"JobSummary":{
"Constraints":null,
"CreateIndex":109,
"Datacenters":[
"dc1"
],
"EnforceIndex":false,
"Failover":false,
"ID":"ba13f3e1-0630-f141-4c60-a1f36c61cdb3",
"JobModifyIndex":2372,
"ModifyIndex":2372,
"Name":"have_a_try",
"Orders":[
],
"Region":"global",
"Status":"running",
"StatusDescription":"",
"Tasks":[
{
"Config":{
"GroupMaxSize":"1",
"ChunkSize":"2000",
"BinlogFile":"1.000002",
"MsgsLimit":"65536",
"GroupTimeout":"100",
"DropTableIfExists":false,
"ExpandSyntaxSupport":false,
"BytesLimit":"67108864",
"MsgBytesLimit":"20480",
"ReplChanBufferSize":"600",
"NatsAddr":"172.100.9.6:8193",
"ApproveHeterogeneous":true,
"SqlFilter":[
],
"BinlogPos":4261,
"ConnectionConfig":{
"Host":"172.100.9.1",
"Port":"3306",
"User":"src_test",
"Password":"*"
},
"TrafficAgainstLimits":0,
"Gtid":"8868d98f-af5e-11e8-9aa9-0242ac110002:1-16",
"ReplicateDoDb":[
{
"TableSchema":"test",
"Tables":[
{
"TableName":"test1"
}
]
}
],
"SkipCreateDbTable":false
},
"ConfigLock":{
},
"Constraints":null,
"Driver":"MySQL",
"Leader":false,
"NodeID":"ee97dc49-85ed-febc-4d3c-cfbfa87f46bd",
"NodeName":"",
"Type":"Src"
},
{
"Config":{
"BinlogFile":"1.000002",
"BinlogPos":4261,
"ConnectionConfig":{
"Host":"172.100.9.2",
"Port":"3306",
"User":"dest_test",
"Password":"*"
},
"Gtid":"8868d98f-af5e-11e8-9aa9-0242ac110002:1-16",
"ParallelWorkers":"1",
"NatsAddr":"172.100.9.6:8193"
},
"ConfigLock":{
},
"Constraints":null,
"Driver":"MySQL",
"Leader":true,
"NodeID":"e623aedd-5c37-da67-4ddf-1a82ce1ac298",
"NodeName":"",
"Type":"Dest"
}
],
"Type":"synchronous"
},
"ModifyIndex":2372,
"Name":"have_a_try",
"Status":"running",
"StatusDescription":"",
"Type":"synchronous"
}
]
2.4 查询目标端迁移情况
目标端迁移前后对比：
新增两个库其中 test 库是我们 job 迁移的结果。
dtle 库相关说明：
https://actiontech.github.io/dtle-docs-cn/3/3.3impacton_dest.html
2.5 在源端执行普通 DML 操作
2.6 在源端执行普通 DDL 操作
小结
更多用法，详见项目文档，欢迎试用~
https://actiontech.github.io/dtle-docs-cn/