saltstack returners 结果转存

returners 是saltstack对minion执行操作后,对返回的数据进行存储,可以存储到一个文件或者数据库当中。

支持的returners http://docs.saltstack.cn/zh_CN/latest/ref/returners/all/index.html 官方例子:

https://github.com/saltstack/salt/tree/develop/salt/returners 参考:http://lixcto.blog.51cto.com/4834175/1430638 http://blog.cunss.com/?p=282

我们先来试下返回到本地(返回给master屏幕)
流程:
  • 1.把自定义的returner传到minion端
  • 2.master上执行操作,指定retuner的具体对象
  • 3.minion执行命令,命令执行结果为ret表示,
  • 4.这个结果一方面会按照通常的方式返回给master并在master的屏幕上面输出打印。
  • 5.另一方面,ret这个结果会传给我们自定义的returner,然后在minion端执行returner,
  • 6.执行returner产生的结果就是把最终的结果导入文件或者数据库 returner到minion控制台
[root@localhost ~]# cd /etc/salt/
[root@localhost salt]# mkdir _returners
[root@localhost salt]# cd _returners/
[root@localhost _returners]# cat local.py            
在控制台输入执行的结果
from __future__ import absolute_import, print_function
 
 
def returner(ret):
    '''
    Print the return data to the terminal to verify functionality
    '''
    print(ret)
[root@localhost _returners]# 
[root@localhost _returners]# salt '*' saltutil.sync_returners     状returner同步到minion端
[root@localhost salt]# salt-minion  -l debug   在minion上启动debug模式
[root@localhost _returners]# salt '*' test.ping --return    local   执行一条命令return到local的控制台
client1.roddypy.com:
    True
client1.roddypy.com:       
    True
[root@localhost _returners]# 
 
在minion的debug控制台中就有结果了
{'fun_args': [], 'jid': '20180306170253692413', 'return': True, 'retcode': 0, 'success': True, 'fun': 'test.ping', 'id': 'node76'}
returner到minion本地文件中
[root@localhost _returners]# vim local_file.py          自定义和脚本,其实就是把ret结果保留
import json
import time
  
def returner(ret):
    now = time.localtime() 
    now = time.strftime("%Y-%m-%d %H:%M:%S",now)
    result_file = '/tmp/returners_roddpy.log'
    result = file(result_file,'a+')
    result.write('At'+str(json.dumps(now))+'\n')
    result.write(str(json.dumps(ret))+'\n')
    result.close()
[root@localhost _returners]# 
[root@localhost _returners]# salt '*' saltutil.sync_returners   分发到minion中
client1.roddypy.com:
    - returners.local_file
[root@localhost _returners]#
 
 
[root@localhost _returners]# salt '*' cmd.run 'df -h ' --return local_file
client1.roddypy.com:
    Filesystem                    Size  Used Avail Use% Mounted on
    /dev/mapper/VolGroup-lv_root   18G  1.4G   16G   9% /
    tmpfs                         495M   12K  495M   1% /dev/shm
    /dev/sda1                     485M   34M  426M   8% /boot
[root@localhost _returners]#
  • 查看log
[root@localhost tmp]# cat returners_roddpy.log    嘿嘿结果已经保存到minion中的/tmp/目录下了
[DEBUG   ] Re-using SAuth for ('/etc/salt/pki/minion', 'node76', 'tcp://192.168.104.76:4506')
[INFO    ] User root Executing command cmd.run with jid 20180306184341492564
[DEBUG   ] Command details {'tgt_type': 'glob', 'jid': '20180306184341492564', 'tgt': '*', 'ret': '', 'user': 'root', 'arg': ['uptime'], 'fun': 'cmd.run'}
[INFO    ] Starting a new job with PID 23218
[DEBUG   ] LazyLoaded cmd.run
[INFO    ] Executing command 'uptime' in directory '/root'
[DEBUG   ] output:  18:43:41 up 162 days,  7:48,  4 users,  load average: 0.15, 0.10, 0.12
[DEBUG   ] Minion return retry timer set to 8 seconds (randomized)
[INFO    ] Returning information for job: 20180306184341492564
[DEBUG   ] Re-using SAuth for ('/etc/salt/pki/minion', 'node76', 'tcp://192.168.104.76:4506')
returner到redis
  • 1.安装redis server
[root@localhost _returners]# yum install redis -y
 
[root@localhost _returners]# vim /etc/redis.conf 
# If you want you can bind a single interface, if the bind option is not
bind 0.0.0.0
[root@localhost _returners]# 
 
[root@localhost _returners]# /etc/init.d/redis restart
停止 redis-server:                                        [确定]
启动 :                                                    [确定]
[root@localhost _returners]#
检查启动端口:
[root@localhost _returners]# netstat -nlpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      12166/mysqld       
tcp        0      0 0.0.0.0:6379              0.0.0.0:*                   LISTEN      14119/redis-server     #已经启动
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1425/sshd          
tcp        0      0 192.168.140.129:4505        0.0.0.0:*                   LISTEN      10378/python2.6     
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      1504/master        
tcp        0      0 192.168.140.129:4506        0.0.0.0:*                   LISTEN      10400/python2.6     
tcp        0      0 :::22                       :::*                        LISTEN      1425/sshd          
tcp        0      0 ::1:25  
  • 2.minion安装redis client
[root@localhost _returners]# salt '*' cmd.run_all 'yum install python-pip -y'
 
[root@localhost _returners]# salt '*' cmd.run_all 'pip install redis'
 
在minion上修改配置文件指定redis信息
[root@localhost tmp]# vim /etc/salt/minion
 
 
#return: mysql
  redis.db: '0'
  redis.host: 192.168.104.76
  redis.port: 6379

重新启动minion
[root@localhost tmp]# /etc/init.d/salt-minion restart
Stopping salt-minion daemon:                               [确定]
Starting salt-minion daemon:                               [确定]
[root@localhost tmp]# 

在master上执行一条命令
[root@localhost _returners]# salt '*' cmd.run 'hostname' --return redis
client1.roddypy.com:
    localhost.localdomain
[root@localhost _returners]# 

在master上另起一个终端看看 redis的实时信息

[root@localhost ~]# redis-cli monitor
OK
1449284517.054169 "monitor"
1449284519.768636 "SELECT" "0"
1449284519.768868 "SET" "client1.roddypy.com:20181205110159739907" "{\"fun_args\": [\"hostname\"], \"jid\": \"20181205110159739907\", \"return\": \"localhost.localdomain\", \"retcode\": 0, \"success\": true, \"fun\": \"cmd.run\", \"id\": \"client1.roddypy.com\"}"
1449284519.769302 "LPUSH" "client1.roddypy.com:cmd.run" "20181205110159739907"
1449284519.769703 "SADD" "minions" "client1.roddypy.com"
1449284519.769963 "SADD" "jids" "20181205110159739907"

在master中查看redis keys

[root@localhost _returners]# redis-cli 
redis 127.0.0.1:6379> ?
redis-cli 2.4.10
Type: "help @<group>" to get a list of commands in <group>
      "help <command>" for help on <command>
      "help <tab>" to get a list of possible help topics
      "quit" to exit
redis 127.0.0.1:6379> get ?
(nil)
redis 127.0.0.1:6379> get "client1.roddypy.com:20181205110159739907"
"{\"fun_args\": [\"hostname\"], \"jid\": \"20181205110159739907\", \"return\": \"localhost.localdomain\", \"retcode\": 0, \"success\": true, \"fun\": \"cmd.run\", \"id\": \"client1.roddypy.com\"}"
 
redis 127.0.0.1:6379> keys *      查询所有的key
1) "client1.roddypy.com:20181205110159739907"
2) "client1.roddypy.com:20181205110142750466"
3) "minions"
4) "client1.roddypy.com:20181205110419271228"
5) "client1.roddypy.com:cmd.run"
6) "name"
7) "jids"

不错,,成功进入redis db里面

returner到mysql
[root@localhost salt]# yum install mysql mysql-server -y     安装mysql
已加载插件:fastestmirror, security
设置安装进程
Loading mirror speeds from cached hostfile
 * base: centos.ustc.edu.cn
 * epel: ftp.cuhk.edu.hk
 * extras: mirrors.sina.cn
 * updates: mirrors.sina.cn
 * webtatic: sp.repo.webtatic.com
包 mysql-5.6.73-5.el7_6.x86_64 已安装并且是最新版本
包 mysql-server-5.6.73-5.el7_6.x86_64 已安装并且是最新版本
无须任何处理
[root@localhost salt]# /etc/init.d/mysqld start
正在启动 mysqld:                                          [确定]
[root@localhost salt]# /etc/init.d/mysqld status
mysqld (pid  12166) 正在运行...
[root@localhost salt]# 
[root@localhost _returners]# mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 11
Server version: 5.6.73 Source distribution
 
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
 
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
mysql> use salt;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
 
Database changed
mysql>
CREATE DATABASE `salt`
      DEFAULT CHARACTER SET utf8
      DEFAULT COLLATE utf8_general_ci;
    USE `salt`;
    --
    -- Table structure for table `jids`
    --
    DROP TABLE IF EXISTS `jids`;
    CREATE TABLE `jids` (
      `jid` varchar(255) NOT NULL,
      `load` mediumtext NOT NULL,
      UNIQUE KEY `jid` (`jid`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
    --
    -- Table structure for table `salt_returns`
    --
    DROP TABLE IF EXISTS `salt_returns`;
    CREATE TABLE `salt_returns` (
      `fun` varchar(50) NOT NULL,
      `jid` varchar(255) NOT NULL,
      `return` mediumtext NOT NULL,
      `id` varchar(255) NOT NULL,
      `success` varchar(10) NOT NULL,
      `full_ret` mediumtext NOT NULL,
      KEY `id` (`id`),
      KEY `jid` (`jid`),
      KEY `fun` (`fun`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
mysql> grant all on salt.* to  'salt'@'%' identified by 'salt';
Query OK, 0 rows affected (0.03 sec)
 
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
 
mysql>

参考官方github:https://github.com/saltstack/salt/blob/develop/salt/returners/mysql.py

[root@localhost _returners]# vim mysql.py            编辑returners mysql程序
from contextlib import contextmanager
import sys
import json
import logging
try:
    import MySQLdb
    HAS_MYSQL = True
except ImportError:
    HAS_MYSQL = False
log = logging.getLogger(__name__)
def __virtual__():
    if not HAS_MYSQL:
        return False
    return 'mysql'
def _get_options():
    '''
    Returns options used for the MySQL connection.
    '''
    defaults = {'host': '192.168.140.129',
                'user': 'salt',
                'pass': 'salt',
                'db': 'salt',
                'port': 3306}
    _options = {}
    for attr in defaults:
        _attr = __salt__['config.option']('mysql.{0}'.format(attr))
        if not _attr:
            log.debug('Using default for MySQL {0}'.format(attr))
            _options[attr] = defaults[attr]
            continue
        _options[attr] = _attr
    return _options
@contextmanager
def _get_serv(commit=False):
    '''
    Return a mysql cursor
    '''
    _options = _get_options()
    conn = MySQLdb.connect(host=_options['host'], user=_options['user'], passwd=_options['pass'], db=_options['db'], port=_options['port'])
    cursor = conn.cursor()
    try:
        yield cursor
    except MySQLdb.DatabaseError as err:
        error, = err.args
        sys.stderr.write(error.message)
        cursor.execute("ROLLBACK")
        raise err
    else:
        if commit:
            cursor.execute("COMMIT")
        else:
            cursor.execute("ROLLBACK")
    finally:
        conn.close()
def returner(ret):
    '''
    Return data to a mysql server
    '''
    with _get_serv(commit=True) as cur:
        sql = '''INSERT INTO `salt_returns`
                (`fun`, `jid`, `return`, `id`, `success`, `full_ret` )
                VALUES (%s, %s, %s, %s, %s, %s)'''
        cur.execute(sql, (ret['fun'], ret['jid'],
                            str(ret['return']), ret['id'],
                            ret['success'], json.dumps(ret)))
[root@localhost _returners]#
  • 操作
[root@localhost _returners]# salt '*' saltutil.sync_returners  同步到minion
 
[root@localhost _returners]# salt '*' cmd.run_all 'yum install MySQL-python* -y'   客户端需要安装Mysqldb模块
 
[root@localhost _returners]# salt '*' cmd.run 'uptime' --return mysql              执行一条命令
client1.roddypy.com:
     19:20:26 up 1 day,  1:35,  2 users,  load average: 0.07, 0.02, 0.00
  • 检查return过来的数据
[root@localhost _returners]# mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 11
Server version: 5.6.73 Source distribution
 
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
 
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
mysql> use salt;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
 
Database changed
mysql> show tables;
+----------------+
| Tables_in_salt |
+----------------+
| jids           |
| salt_returns   |
+----------------+
2 rows in set (0.00 sec)
 
 
 
mysql> select * from salt_returns;
+---------+----------------------+----------------------------------------------------------------------+---------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| fun     | jid                  | return                                                               | id                  | success | full_ret                                                                                                                                                                                                                |
+---------+----------------------+----------------------------------------------------------------------+---------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| cmd.run | 20151206045940088527 |  19:20:26 up 1 day,  1:35,  2 users,  load average: 0.07, 0.02, 0.00 | client1.roddypy.com | 1       | {"fun_args": ["uptime"], "jid": "20151206045940088527", "return": " 19:20:26 up 1 day,  1:35,  2 users,  load average: 0.07, 0.02, 0.00", "retcode": 0, "success": true, "fun": "cmd.run", "id": "client1.roddypy.com"} |
+---------+----------------------+----------------------------------------------------------------------+---------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
 
mysql>

在高并发业务情况下,如果Minion非常多,minion同时写到数据的话,会造成mysql的压力

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏高爽的专栏

Oracle学习记录

       oracle学习过程中记录的一些知识点,包括sqlplus一些命令、角色、DML、DCL、DDL、数据字典、表空间、函数。 1. sys 超级管...

1850
来自专栏杨建荣的学习笔记

MySQL query rewrite插件简单测试

在机场继续努力一把,学习了下MySQL query rewrite这个插件,感觉还不错,j简答测试了下,已经找到Oracle FGA的影子了。

942
来自专栏跟着阿笨一起玩NET

SQL将本地图片文件插入到数据库

962
来自专栏数据和云

【安全为王】听说你最喜欢给所有用户授DBA的权限

编辑手记:最近很多朋友会问关于数据库用户授权的问题,经常问这样的问题,我能不能直接给用户DBA的角色,多方便。恩,方便是方便了,恶意操作和误操作也方便了。今天再...

2906
来自专栏鬼谷君

inception安装使用

注意bison版本,centos7如果yum安装bison,会因为版本过高导致编译报错

842
来自专栏杨建荣的学习笔记

浅谈exp/imp(下) (r5笔记第84天)

相关链接:浅谈exp/imp(上) (r5笔记第81天) 你可能 不了解的dump文件 在工作中,dump文件对于dba而言是再平常不过的文件了。不过因为dum...

3099
来自专栏应用案例

索引,视图,存储过程和触发器文档

实验案例一:验证索引的作用 1、首先创建一个数据量大的表,名称为“学生表”,分别有三列,学号,姓名和班级,如下图所示,学号为自动编号,班级为默认值“一班”。 ?...

1958
来自专栏Hadoop实操

CM启动报InnoDB engine not found分析

将/tmp目录权限修改为777,重启mysql和cloudera-scm-server服务

3775
来自专栏菩提树下的杨过

mysql: Illegal mix of collations (utf8_unicode_ci,IMPLICIT) and (utf8_general_ci,IMPLICIT) for opera

昨天把mysql里所有table的varchar字段的字符集,批量换成了utf8mb4/utf8mb4_unicode_ci ,以便能保存一些emoji火星文 ...

20010
来自专栏乐沙弥的世界

使用FREDATED引擎实现跨实例访问

    跨数据库服务器,跨实例访问是比较常见的一种访问方式,在Oracle中可以通过DB LINK的方式来实现。对于MySQL而言,有一个FEDERATED存储...

701

扫码关注云+社区