ODBParser是一款公开资源情报工具,可以帮助广大研究人员从Elasticsearch和MongoDB目录中搜索、解析并导出我们感兴趣的数据。除此之外,这款工具还可以帮助广大研究人员从开放数据库中搜索出曝光的个人可标识信息(PII)。
ODBParser的主要目标是创建一个一站式公开资源情报工具,用于搜索、解析和分析开放数据库,以便识别第三方服务器上的PII泄漏。
导出选项
首先,我们需要使用下列命令将该项目源码克隆至本地:
git clone https://github.com/citcheese/ODBParser.git
接下来,我们需要自行获取Shodan和BinaryEdge的API密钥,并在ODBconfig.py文件中配置相应的参数。
最后,根据项目目录中的requirements.txt安装并配置该工具所需的依赖组件。
我建议广大研究人员创建一个ODBParser的虚拟环境,该工具目前仅在Windows 10(Python 3.7.3)环境中进行过测试。
Examples: python ODBParser.py -cn US -p 8080 -t users --elastic --shodan --csv --limit 100
python ODBParser.py -ip 192.168.2:8080 --mongo --ignorelogs --nosizelimits
Damage to-date: 0 servers parsed | 0 databases dumped | 0 records pulled
_____________________________________________________________________________
optional arguments:
-h, --help show this help message and exit
Query Options:
--shodan, -sh Add this flag if using Shodan. Specify ES or MDB w/flags.
--binary, -be Add this flag if using BinaryEdge. Specify ES or MDB w/ flags.
--ip , -ip Query one server. Add port like so '192.165.2.1:8080' or will use default ports for each db type. Add ES or MDB flags to specify parser.
--file , -f Load line-separated IPs from file. Add port or will assume default ports for each db type. Add ES or MDB flags to specify parser.
--paste, -v Query line-separated IPs from clipboard. Add port or will assume default ports for each db type, e.g. 9200 for ES. Add ES or MDB flags to specify parser.
Shodan/BinaryEdge Options:
--limit , -l Max number of results per query. Default is 500.
--port , -p Filter by port.
--country , -cn Filter by country (two-letter country code).
--terms , -t Enter any additional query terms you want here, e.g.'users'
Dump Options:
--mongo, -mdb Use for IP, Shodan, BinaryEdge & Paste methods to specify parser.
--elastic, -es Use for IP, Shodan, BinaryEdge & Paste methods to specify parser.
--properjson, -pj Add this flag if would like out put to be proper JSON file. Default is one JSON string object per line.
--database , -db Specify database you want to grab. For MDB must be in format format 'db:collection'. Use with IP arg & 'es' or 'mdb' flag
--getall, -g Get all indices regardless of fields and collection/index names (overrides selections in config file).
--ignorelogs Connect to a server you've already checked out.
--nosizelimits, -n Dump index no matter how big it is. Default max doc count is 800,000.
--csv Convert JSON dumps into CSV format on the fly. (Puts JSON files in backup folder in case there is issue with coversion)
CSV/Post-processing Options:
--convertToCSV , -c Convert JSON file or folder of JSON dumps to CSVs after the fact. Enter full path or folder name in current working directory
--dontflatten Use if run into memory issues converting JSON files to CSV during post-processing.
--basic Use with --convertToCSV flag if your JSON dumps are not true JSON files, but rather line separated JSON objects that you got from other sources.
--dontclean, -dc Choose if want to keep useless data when convert to CSV. See docs for more info.
项目地址
https://github.com/citcheese/ODBParser