Postgres-XL是一个基于PostgreSQL数据库的横向扩展开源SQL数据库集群,具有足够的灵活性来处理不同的数据库工作负载,架构如下图所示:
Web 2.0
操作数据存储
GIS的地理空间
混合业务工作环境
OLTP 写频繁的业务
多租户服务提供商托管环境
完全ACID,保持事务一致性
包含JSON的Key-value 存储
需要MPP并行性商业智能/大数据分析
各个组件介绍如下:
Global Transaction Monitor (GTM)全局事务管理器,确保群集范围内的事务一致性。GTM负责发放事务ID和快照作为其多版本并发控制的一部分。集群也可以配置一个或多个备用GTM,以改进可用性。此外,可以在协调器间配置GTM代理, 可用于改善可扩展性,减少GTM的通信量。
GTM StandbyGTM的备用节点,在pgxc,pgxl中,GTM控制所有的全局事务分配,如果出现问题,就会导致整个集群不可用,为了增加可用性,增加该备用节点。当GTM出现问题时,GTM Standby可以升级为GTM,保证集群正常工作。
GTM ProxyGTM需要与所有的Coordinators通信,为了降低压力,可以在每个Coordinator机器上部署一个GTM Proxy。
Coordinator协调员管理用户会话,并与GTM和数据节点进行交互。协调员解析,并计划查询,并给语句中的每一个组件发送下一个序列化的全局性计划。为节省机器,通常此服务和数据节点部署在一起。
Data Node数据节点是数据实际存储的地方。数据的分布可以由DBA来配置。为了提高可用性,可以配置数据节点的热备以便进行故障转移准备。总之,GTM是负责ACID的,保证分布式数据库全局事务一致性。得益于此,就算数据节点是分布的,但是在主节点操作增删改查事务时,就如同只操作一个数据库一样简单。Coordinator是调度的,将操作指令发送到各个数据节点。datanodes是数据节点,分布式存储数据。
[postgres@pg01 ~]$ pgxc_ctl -c /u01/pgxl/pgxc_ctl/pgxc_ctl.conf init all
/bin/bash
Installing pgxc_ctl_bash script as /u01/pgxl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /u01/pgxl/pgxc_ctl_bash.
Reading configuration using /u01/pgxl/pgxc_ctl_bash --home /u01/pgxl --configuration /u01/pgxl/pgxc_ctl/pgxc_ctl.conf
Finished reading configuration.
******** PGXC_CTL START ***************
Current directory: /u01/pgxl
Initialize GTM master
The files belonging to this GTM system will be owned by user "postgres".
This user must also own the server process.
fixing permissions on existing directory /u01/pgxl/nodes/gtm ... ok
creating configuration files ... ok
creating control file ... ok
Success.
Done.
Start GTM master
server starting
Initialize GTM slave
The files belonging to this GTM system will be owned by user "postgres".
This user must also own the server process.
fixing permissions on existing directory /u01/pgxl/nodes/gtm_slave ... ok
creating configuration files ... ok
creating control file ... ok
Success.
Done.
Start GTM slavepgxc_ctl(13032):1811220924_48 server starting
Done.
Initialize all the gtm proxies.
Initializing gtm proxy gtm_pxy1.
Initializing gtm proxy gtm_pxy2.
The files belonging to this GTM system will be owned by user "postgres".
This user must also own the server process.
fixing permissions on existing directory /u01/pgxl/nodes/gtm_pxy ... ok
creating configuration files ... ok
Success.
The files belonging to this GTM system will be owned by user "postgres".
This user must also own the server process.
fixing permissions on existing directory /u01/pgxl/nodes/gtm_pxy ... ok
creating configuration files ... ok
Success.
Done.
Starting all the gtm proxies.
Starting gtm proxy gtm_pxy1.
Starting gtm proxy gtm_pxy2.
server starting
server starting
Done.
Initialize all the coordinator masters.
Initialize coordinator master coord1.
Initialize coordinator master coord2.
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /u01/pgxl/nodes/coord ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... creating cluster information ... ok
syncing data to disk ... ok
freezing database template0 ... ok
freezing database template1 ... ok
freezing database postgres ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success.
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /u01/pgxl/nodes/coord ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... creating cluster information ... ok
syncing data to disk ... ok
freezing database template0 ... ok
freezing database template1 ... ok
freezing database postgres ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success.
Done.
Starting coordinator master.
Starting coordinator master coord1
Starting coordinator master coord2
2018-11-22 09:24:59.252 CST [13950] LOG: listening on IPv4 address "0.0.0.0", port 5433
2018-11-22 09:24:59.252 CST [13950] LOG: listening on IPv6 address "::", port 5433
2018-11-22 09:24:59.254 CST [13950] LOG: listening on Unix socket "/tmp/.s.PGSQL.5433"
2018-11-22 09:24:59.302 CST [13950] LOG: redirecting log output to logging collector process
2018-11-22 09:24:59.302 CST [13950] HINT: Future log output will appear in directory "pg_log".
2018-11-22 09:24:59.060 CST [13633] LOG: listening on IPv4 address "0.0.0.0", port 5433
2018-11-22 09:24:59.061 CST [13633] LOG: listening on IPv6 address "::", port 5433
2018-11-22 09:24:59.062 CST [13633] LOG: listening on Unix socket "/tmp/.s.PGSQL.5433"
2018-11-22 09:24:59.109 CST [13633] LOG: redirecting log output to logging collector process
2018-11-22 09:24:59.109 CST [13633] HINT: Future log output will appear in directory "pg_log".
Done.
Initialize all the coordinator slaves.
Initialize the coordinator slave coord1.
Initialize the coordinator slave coord2.
Done.
Starting all the coordinator slaves.
Starting coordinator slave coord1.
Starting coordinator slave coord2.
2018-11-22 09:25:05.987 CST [13330] LOG: listening on IPv4 address "0.0.0.0", port 5433
2018-11-22 09:25:05.987 CST [13330] LOG: listening on IPv6 address "::", port 5433
2018-11-22 09:25:05.989 CST [13330] LOG: listening on Unix socket "/tmp/.s.PGSQL.5433"
2018-11-22 09:25:06.039 CST [13330] LOG: redirecting log output to logging collector process
2018-11-22 09:25:06.039 CST [13330] HINT: Future log output will appear in directory "pg_log".
2018-11-22 09:25:05.517 CST [13266] LOG: listening on IPv4 address "0.0.0.0", port 5433
2018-11-22 09:25:05.517 CST [13266] LOG: listening on IPv6 address "::", port 5433
2018-11-22 09:25:05.518 CST [13266] LOG: listening on Unix socket "/tmp/.s.PGSQL.5433"
2018-11-22 09:25:05.567 CST [13266] LOG: redirecting log output to logging collector process
2018-11-22 09:25:05.567 CST [13266] HINT: Future log output will appear in directory "pg_log".
Done
Initialize all the datanode masters.
Initialize the datanode master datanode1.
Initialize the datanode master datanode2.
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /u01/pgxl/nodes/dnmaster ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... creating cluster information ... ok
syncing data to disk ... ok
freezing database template0 ... ok
freezing database template1 ... ok
freezing database postgres ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success.
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /u01/pgxl/nodes/dnmaster ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... creating cluster information ... ok
syncing data to disk ... ok
freezing database template0 ... ok
freezing database template1 ... ok
freezing database postgres ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success.
Done.
Starting all the datanode masters.
Starting datanode master datanode1.
Starting datanode master datanode2.
2018-11-22 09:25:15.129 CST [13604] LOG: listening on IPv4 address "0.0.0.0", port 5436
2018-11-22 09:25:15.130 CST [13604] LOG: listening on IPv6 address "::", port 5436
2018-11-22 09:25:15.131 CST [13604] LOG: listening on Unix socket "/tmp/.s.PGSQL.5436"
2018-11-22 09:25:15.148 CST [13604] LOG: redirecting log output to logging collector process
2018-11-22 09:25:15.148 CST [13604] HINT: Future log output will appear in directory "pg_log".
2018-11-22 09:25:14.657 CST [13539] LOG: listening on IPv4 address "0.0.0.0", port 5436
2018-11-22 09:25:14.657 CST [13539] LOG: listening on IPv6 address "::", port 5436
2018-11-22 09:25:14.659 CST [13539] LOG: listening on Unix socket "/tmp/.s.PGSQL.5436"
2018-11-22 09:25:14.676 CST [13539] LOG: redirecting log output to logging collector process
2018-11-22 09:25:14.676 CST [13539] HINT: Future log output will appear in directory "pg_log".
Done.
ALTER NODE coord1 WITH (HOST='pg01', PORT=5433);
ALTER NODE
CREATE NODE coord2 WITH (TYPE='coordinator', HOST='pg02', PORT=5433);
CREATE NODE
CREATE NODE datanode1 WITH (TYPE='datanode', HOST='pg03', PORT=5436);
CREATE NODE
CREATE NODE datanode2 WITH (TYPE='datanode', HOST='pg04', PORT=5436);
CREATE NODE
SELECT pgxc_pool_reload();
pgxc_pool_reload
------------------
t
(1 row)
CREATE NODE coord1 WITH (TYPE='coordinator', HOST='pg01', PORT=5433);
CREATE NODE
ALTER NODE coord2 WITH (HOST='pg02', PORT=5433);
ALTER NODE
CREATE NODE datanode1 WITH (TYPE='datanode', HOST='pg03', PORT=5436);
CREATE NODE
CREATE NODE datanode2 WITH (TYPE='datanode', HOST='pg04', PORT=5436);
CREATE NODE
SELECT pgxc_pool_reload();
pgxc_pool_reload
------------------
t
(1 row)
Done.
EXECUTE DIRECT ON (datanode1) 'CREATE NODE coord1 WITH (TYPE=''coordinator'', HOST=''pg01'', PORT=5433)';
EXECUTE DIRECT
EXECUTE DIRECT ON (datanode1) 'CREATE NODE coord2 WITH (TYPE=''coordinator'', HOST=''pg02'', PORT=5433)';
EXECUTE DIRECT
EXECUTE DIRECT ON (datanode1) 'ALTER NODE datanode1 WITH (TYPE=''datanode'', HOST=''pg03'', PORT=5436)';
EXECUTE DIRECT
EXECUTE DIRECT ON (datanode1) 'CREATE NODE datanode2 WITH (TYPE=''datanode'', HOST=''pg04'', PORT=5436)';
EXECUTE DIRECT
EXECUTE DIRECT ON (datanode1) 'SELECT pgxc_pool_reload()';
pgxc_pool_reload
------------------
t
(1 row)
EXECUTE DIRECT ON (datanode2) 'CREATE NODE coord1 WITH (TYPE=''coordinator'', HOST=''pg01'', PORT=5433)';
EXECUTE DIRECT
EXECUTE DIRECT ON (datanode2) 'CREATE NODE coord2 WITH (TYPE=''coordinator'', HOST=''pg02'', PORT=5433)';
EXECUTE DIRECT
EXECUTE DIRECT ON (datanode2) 'CREATE NODE datanode1 WITH (TYPE=''datanode'', HOST=''pg03'', PORT=5436)';
EXECUTE DIRECT
EXECUTE DIRECT ON (datanode2) 'ALTER NODE datanode2 WITH (TYPE=''datanode'', HOST=''pg04'', PORT=5436)';
EXECUTE DIRECT
EXECUTE DIRECT ON (datanode2) 'SELECT pgxc_pool_reload()';
pgxc_pool_reload
------------------
t
(1 row)
Done.
通过查询pgxc_node表可以获取集群节点信息:
[postgres@pg01 ~]$ pgxc_ctl -c /u01/pgxl/pgxc_ctl/pgxc_ctl.conf stop all
[postgres@pg01 ~]$ pgxc_ctl -c /u01/pgxl/pgxc_ctl/pgxc_ctl.conf start all
2.4 Posgres-XL集群服务监控
[postgres@pg01 ~]$ pgxc_ctl -c /u01/pgxl/pgxc_ctl/pgxc_ctl.conf monitor all
/bin/bash
Installing pgxc_ctl_bash script as /u01/pgxl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /u01/pgxl/pgxc_ctl_bash.
Reading configuration using /u01/pgxl/pgxc_ctl_bash --home /u01/pgxl --configuration /u01/pgxl/pgxc_ctl/pgxc_ctl.conf
Finished reading configuration.
******** PGXC_CTL START ***************
Current directory: /u01/pgxl
Running: gtm master
Running: gtm slave
Running: gtm proxy gtm_pxy1
Running: gtm proxy gtm_pxy2
Running: coordinator master coord1
Running: coordinator slave coord1
Running: coordinator master coord2
Running: coordinator slave coord2
Running: datanode master datanode1
Running: datanode slave datanode1
Running: datanode master datanode2
Running: datanode slave datanode2
创建一张log表,然后插入500条数据,如下:
[postgres@pg01 ~]$ psql -p5433
psql (PGXL 10r1, based on PG 10.5 (Postgres-XL 10r1))
Type "help" for help.
postgres=# CREATE TABLE log (id numeric NOT NULL,stamp timestamp with time zone,user_id numeric);
postgres=# copy log from '/u02/tmp/log.csv' with csv;
COPY 499
可以在协调器节点或者各个数据节点查看,这里使用下面的语句直接统计每个数据节点的表数据分布情况:
postgres=# SELECT xc_node_id, count(*) FROM log GROUP BY xc_node_id;
distribute表默认情况下,系统会将插入的数据,按照拆分规则,分配到不同的datanode节点中存储,也就是sharding技术。每个datanode节点只保存了部分数据,通过coordinate节点可以查询完整的数据视图。上面创建的log表就是distribute表。建表语法如下:
replication表各个datanode节点中,表的数据完全相同。也就是说,在插入数据时,系统会分别在每个datanode节点插入相同数据。读数据时,只需要读任意一个datanode节点上的数据即可。建表语法如下:
[postgres@pg01 ~]$ pgxc_ctl -c /u01/pgxl/pgxc_ctl/pgxc_ctl.conf
/bin/bash
Installing pgxc_ctl_bash script as /u01/pgxl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /u01/pgxl/pgxc_ctl_bash.
Reading configuration using /u01/pgxl/pgxc_ctl_bash --home /u01/pgxl --configuration /u01/pgxl/pgxc_ctl/pgxc_ctl.conf
Finished reading configuration.
******** PGXC_CTL START ***************
Current directory: /u01/pgxl
PGXC add datanode slave datanode1 pg05 5436 5437 /u01/pgxl/nodes/dnslave /u01/pgxl/nodes/dn_slave_war /u01/pgxl/nodes/dn_archlog
PGXC add datanode slave datanode2 pg06 5436 5437 /u01/pgxl/nodes/dnslave /u01/pgxl/nodes/dn_slave_war /u01/pgxl/nodes/dn_archlog
检查所有服务运行正常:
[postgres@pg01 ~]$ pgxc_ctl -c /u01/pgxl/pgxc_ctl/pgxc_ctl.conf monitor all
/bin/bash
Installing pgxc_ctl_bash script as /u01/pgxl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /u01/pgxl/pgxc_ctl_bash.
Reading configuration using /u01/pgxl/pgxc_ctl_bash --home /u01/pgxl --configuration /u01/pgxl/pgxc_ctl/pgxc_ctl.conf
Finished reading configuration.
******** PGXC_CTL START ***************
Current directory: /u01/pgxl
Running: gtm master
Running: gtm slave
Running: gtm proxy gtm_pxy1
Running: gtm proxy gtm_pxy2
Running: coordinator master coord1
Running: coordinator slave coord1
Running: coordinator master coord2
Running: coordinator slave coord2
Running: datanode master datanode1
Running: datanode slave datanode1
Running: datanode master datanode2
Running: datanode slave datanode2
当前的主节点是pg03和pg04,如下图:
切换主备节点:
[postgres@pg01 ~]$ pgxc_ctl -c /u01/pgxl/pgxc_ctl/pgxc_ctl.conf monitor all
/bin/bash
Installing pgxc_ctl_bash script as /u01/pgxl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /u01/pgxl/pgxc_ctl_bash.
Reading configuration using /u01/pgxl/pgxc_ctl_bash --home /u01/pgxl --configuration /u01/pgxl/pgxc_ctl/pgxc_ctl.conf
Finished reading configuration.
******** PGXC_CTL START ***************
Current directory: /u01/pgxl
Running: gtm master
Running: gtm slave
Running: gtm proxy gtm_pxy1
Running: gtm proxy gtm_pxy2
Running: coordinator master coord1
Running: coordinator slave coord1
Running: coordinator master coord2
Running: coordinator slave coord2
Running: datanode master datanode1
Running: datanode slave datanode1
Running: datanode master datanode2
Running: datanode slave datanode2
[postgres@pg01 ~]$ pgxc_ctl -c /u01/pgxl/pgxc_ctl/pgxc_ctl.conf failover datanode datanode1 datanode2
/bin/bash
Installing pgxc_ctl_bash script as /u01/pgxl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /u01/pgxl/pgxc_ctl_bash.
Reading configuration using /u01/pgxl/pgxc_ctl_bash --home /u01/pgxl --configuration /u01/pgxl/pgxc_ctl/pgxc_ctl.conf
Finished reading configuration.
******** PGXC_CTL START ***************
Current directory: /u01/pgxl
Failover specified datanodes.
Failover the datanode datanode1.
Failover datanode datanode1 using GTM itself
Actual Command: ssh postgres@pg05 "( pg_ctl promote -Z datanode -D /u01/pgxl/nodes/dnslave ) > /tmp/pg01_STDOUT_4264_0 2>&1" < /dev/null > /dev/null 2>&1
Bring remote stdout: scp postgres@pg05:/tmp/pg01_STDOUT_4264_0 /tmp/STDOUT_4264_1 > /dev/null 2>&1
Actual Command: ssh postgres@pg05 "( pg_ctl restart -w -Z datanode -D /u01/pgxl/nodes/dnslave -o -i; sleep 1 ) > /tmp/pg01_STDOUT_4264_2 2>&1" < /dev/null > /dev/null 2>&1
Bring remote stdout: scp postgres@pg05:/tmp/pg01_STDOUT_4264_2 /tmp/STDOUT_4264_3 > /dev/null 2>&1
2018-11-23 14:52:34.655 CST [26323] LOG: listening on IPv4 address "0.0.0.0", port 5436
2018-11-23 14:52:34.655 CST [26323] LOG: listening on IPv6 address "::", port 5436
2018-11-23 14:52:34.657 CST [26323] LOG: listening on Unix socket "/tmp/.s.PGSQL.5436"
2018-11-23 14:52:34.675 CST [26323] LOG: redirecting log output to logging collector process
2018-11-23 14:52:34.675 CST [26323] HINT: Future log output will appear in directory "pg_log".
ALTER NODE
pgxc_pool_reload
------------------
t
(1 row)
EXECUTE DIRECT
pgxc_pool_reload
------------------
t
(1 row)
EXECUTE DIRECT
pgxc_pool_reload
------------------
t
(1 row)
ALTER NODE
pgxc_pool_reload
------------------
t
(1 row)
Failover the datanode datanode2.
Failover datanode datanode2 using GTM itself
Actual Command: ssh postgres@pg06 "( pg_ctl promote -Z datanode -D /u01/pgxl/nodes/dnslave ) > /tmp/pg01_STDOUT_4264_4 2>&1" < /dev/null > /dev/null 2>&1
Bring remote stdout: scp postgres@pg06:/tmp/pg01_STDOUT_4264_4 /tmp/STDOUT_4264_5 > /dev/null 2>&1
Actual Command: ssh postgres@pg06 "( pg_ctl restart -w -Z datanode -D /u01/pgxl/nodes/dnslave -o -i; sleep 1 ) > /tmp/pg01_STDOUT_4264_6 2>&1" < /dev/null > /dev/null 2>&1
Bring remote stdout: scp postgres@pg06:/tmp/pg01_STDOUT_4264_6 /tmp/STDOUT_4264_7 > /dev/null 2>&1
2018-11-23 14:52:38.607 CST [26317] LOG: listening on IPv4 address "0.0.0.0", port 5436
2018-11-23 14:52:38.607 CST [26317] LOG: listening on IPv6 address "::", port 5436
2018-11-23 14:52:38.609 CST [26317] LOG: listening on Unix socket "/tmp/.s.PGSQL.5436"
2018-11-23 14:52:38.628 CST [26317] LOG: redirecting log output to logging collector process
2018-11-23 14:52:38.628 CST [26317] HINT: Future log output will appear in directory "pg_log".
ALTER NODE
pgxc_pool_reload
------------------
t
(1 row)
EXECUTE DIRECT
pgxc_pool_reload
------------------
t
(1 row)
EXECUTE DIRECT
pgxc_pool_reload
------------------
t
(1 row)
ALTER NODE
pgxc_pool_reload
------------------
t
(1 row)
Done.
[postgres@pg01 ~]$ pgxc_ctl -c /u01/pgxl/pgxc_ctl/pgxc_ctl.conf monitor all
/bin/bash
Installing pgxc_ctl_bash script as /u01/pgxl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /u01/pgxl/pgxc_ctl_bash.
Reading configuration using /u01/pgxl/pgxc_ctl_bash --home /u01/pgxl --configuration /u01/pgxl/pgxc_ctl/pgxc_ctl.conf
Finished reading configuration.
******** PGXC_CTL START ***************
Current directory: /u01/pgxl
Running: gtm master
Running: gtm slave
Running: gtm proxy gtm_pxy1
Running: gtm proxy gtm_pxy2
Running: coordinator master coord1
Running: coordinator slave coord1
Running: coordinator master coord2
Running: coordinator slave coord2
Running: datanode master datanode1
Running: datanode master datanode2
切换完成后,旧的master节点作废,而新建的两个slave节点就转变为master节点角色。
[postgres@pg01 ~]$ pgxc_ctl -c /u01/pgxl/pgxc_ctl/pgxc_ctl.conf remove datanode slave datanode1 clean
[postgres@pg01 ~]$ pgxc_ctl -c /u01/pgxl/pgxc_ctl/pgxc_ctl.conf remove datanode slave datanode2 clean