子域名枚举&收集

子域名枚举&收集

引言：信息收集 --> 特此整理 (译文扩展整理)

1. what & why

什么是子域名枚举？

子域名枚举是为一个或多个域名查找子域名的过程，它是安全评估和渗透测试前期侦查和信息收集的重要手段。

为什么要子域名枚举？

• Sub-domain enumeration can reveal a lot of domains/sub-domains that are in scope of a security assessment which in turn increases the chances of finding vulnerabilities
  • 子域枚举可以揭示安全评估范围内的许多域/子域，从而增加发现漏洞的机会
• Finding applications running on hidden, forgotten sub-domains may lead to uncovering critical vulnerabilities
  • 查找在隐藏的、被遗忘的子域上运行的应用程序可能会导致发现关键漏洞
• Often times the same vulnerabilities tend to be present across different domains/applications of the same organization
  • 通常，相同的漏洞往往存在于同一组织的不同域/应用程序中

2. 被动子域名枚举

什么是被动子域名枚举？

• Passive sub-domain enumeration is where an attacker/tester gathers sub-domain information without generating any traffic directly between him and the infrastructure managed by the target organization
  • 被动子域枚举是攻击者/测试者收集子域信息而不在他和目标组织管理的基础设施之间直接产生任何流量的地方
• The objective is to be stealthy and leave low or no footprint
  • 目标是隐身并留下历史足迹或不留下足迹

2.1 证书透明度（Certificate Transparency）

• Under Certificate Transparency(CT), a Certificate Authority(CA) will have to publish all SSL/TLS certificates they issue in a public log
  • 在证书透明度 (CT) 下，证书颁发机构 (CA) 必须在公共日志中发布他们颁发的所有 SSL/TLS 证书
• Anyone can look through the CT logs and find certificates issued for a domain
  • 任何人都可以查看 CT 日志并找到为域颁发的证书
• CT allows website users and domain owners to identify mistakenly or worse maliciously issued certificates. This aids domain owners and browser vendors in identifying erring CAs
  • CT 允许网站用户和域所有者识别错误或更糟的恶意颁发的证书。这有助于域所有者和浏览器供应商识别错误的 CA

更多可阅读：https://blog.csdn.net/www_helloworld_com/article/details/90403233

2.1.1 证书透明度（CT）- 开源网络情报角度

• Certificate Transparency(CT) logs by design contain all the certificates issued by a participating CA for any given domain. SSL/TLS certificates generally contain domain names, sub-domain names and email addresses. These logs are available publicly and anyone can look through these logs. This makes them a treasure trove of information for attackers.
  • 证书透明度 (CT) 日志按设计包含由参与 CA 为任何给定域颁发的所有证书。SSL/TLS 证书一般包含域名、子域名和电子邮件地址。这些日志是公开可用的，任何人都可以查看这些日志。这使它们成为攻击者的信息宝库。
• By looking through the CT logs an attacker can gather a lot of information about an organization’s infrastructure i.e. internal domains, email addresses in a completely passive manner.
  • 通过查看 CT 日志，攻击者可以以完全被动的方式收集有关组织基础设施（即内部域、电子邮件地址）的大量信息。

2.1.2 搜索 CT logs

1. crtsh：https://crt.sh/
2. censys：https://censys.io/
3. facebook：https://developers.facebook.com/tools/ct/
4. google：https://google.com/transparencyreport/https/ct/
5. spyse：https://spyse.com/search/certificate
6. certspotter：https://sslmate.com/certspotter/api/
7. entrust：https://www.entrust.com/ct-search/

2.1.2.1 crtsh

访问https://crt.sh/，输入子域，例如：example.com

crtsh 也提供了使用https://crt.sh/atom?q={sub-domain}查询的 RSS 提要

还可以使用PostgreSQL接口来查询数据，shell脚本如下：

#!/bin/sh
# Script by Hanno Bock - https://github.com/hannob/tlshelpers/blob/master/getsubdomain

query="SELECT ci.NAME_VALUE NAME_VALUE FROM certificate_identity ci WHERE ci.NAME_TYPE = 'dNSName' AND reverse(lower(ci.NAME_VALUE)) LIKE reverse(lower('%.$1'));"

echo $query | \
    psql -t -h crt.sh -p 5432 -U guest certwatch | \
    sed -e 's:^ *::g' -e 's:^*\.::g' -e '/^$/d' | \
    sort -u | sed -e 's:*.::g'

无psql命令，使用sudo apt-get install postgresql-client安装psql客户端。

2.1.2.2 censys

https://search.censys.io/certificates?q={sub-domain}

枚举脚本：https://github.com/0xbharath/censys-enumeration

2.1.2.3 massdns - 提取唯一的可解析子域

https://github.com/blechschmidt/massdns

• Massdns 是一个极快的 DNS 解析器，可以在更短的时间内解析大量域名
• Massdns 可以与从 CT 日志中提取子域的脚本结合使用，以快速识别唯一的可解析域名

2.2.3 证书透明度（CT）的缺点

• Certificate Transparency logs are append only which means once a SSL/TLS certificate is appended to a CT log, there is no way to delete them
  • 证书透明度日志仅附加，这意味着一旦 SSL/TLS 证书附加到 CT 日志，就无法删除它们
• The obvious downside of this during recon is that the domain/sub-domain names found in CT Logs maynot exist anymore and thus will not resolve to any valid IP address
  • 侦察期间明显的缺点是在 CT 日志中找到的域/子域名可能不再存在，因此将无法解析为任何有效的 IP 地址

2.2 搜索引擎

Search engines like Google and Bing supports various advanced search operators to refine search queries. These operators are often referred to as “Google dorks”.

site:{sub-domain}

2.2.1 Google

2.2.2 百度

2.2.4 Bing

https://help.bing.microsoft.com/#apex/18/en-US/10001/-1

2.2.5 Sogou

https://www.sogou.com/

2.2.6 360搜索

https://www.so.com/

2.2.7 Yahoo

http://www.yahoo.com/

2.2.8 Yandex

https://www.yandex.ru/

2.2.9 Exalead

https://www.exalead.com/search

2.2.10 Dogpile

http://www.dogpile.com/

2.3 DNS aggregators

There are a lot of the third party services that will do DNS enumeration on your behalf or they aggregate massive DNS datasets and look through them for sub-domains.

2.3.1 VirusTotal

https://www.virustotal.com/gui/domain/{sub-domain}/relations

https://github.com/appsecco/the-art-of-subdomain-enumeration/blob/master/virustotal_subdomain_enum.py

请求头需添加 X-VT-Anti-Abuse-Header

2.3.2 DNSdumpster

https://github.com/PaulSec/API-dnsdumpster.com

2.3.3 Netcraft

https://searchdns.netcraft.com/?host={sub-domain}

2.4 ASN发现

• 查找 ASN 将帮助我们识别域的网络段
• 使用 dig 或 host 解析给定域的 IP 地址
• 找到给定 IP 地址的 ASN工具
  • https://asn.cymru.com/cgi-bin/whois.cgi
• 查找给定域名的 ASN
  • http://bgp.he.net/

curl -s http://ip-api.com/json/220.181.32.148 | jq -r .as
# install jq: sudo apt-get install jq

• 找到的 ASN 号可用于查找域的网络块
• 我们可以使用高级 WHOIS 查询来查找属于 ASN 的所有 IP 范围

whois -h whois.radb.net  -- '-i origin AS36459' | grep -Eo "([0-9.]+){4}/[0-9]+" | uniq

nmap脚本查找ASN

https://nmap.org/nsedoc/scripts/targets-asn.html

nmap --script targets-asn --script-args targets-asn.asn=xxxx

2.5 Subject Alternate Name(SAN)

The Subject Alternative Name (SAN) is an extension to the X.509 specification that allows to specify additional host names for a single SSL certificate.

2.5.1 从SAN中提取域名

 sed -ne 's/^\( *\)Subject:/\1/p;/X509v3 Subject Alternative Name/{
    N;s/^.*\n//;:a;s/^\( *\)\(.*\), /\1\2\n\1/;ta;p;q; }' < <(
    openssl x509 -noout -text -in <(
        openssl s_client -ign_eof 2>/dev/null <<<$'HEAD / HTTP/1.0\r\n\r' \
            -connect baidu.com:443 ) )

也可以使用python脚本

• https://github.com/appsecco/the-art-of-subdomain-enumeration/blob/master/san_subdomain_enum.py

2.6 公共数据

There are projects that gather Internet wide scan data and make it available to researchers and the security community. The datasets published by this projects are a treasure trove of sub-domain information. Although finding sub-domains in this massive datasets is like finding a needle in the haystack, it is worth the effort.

Following are few public datasets that aggregate information that could be of interest during sub-domain enumeration:

来源：https://github.com/hdm/inetdata

2.6.1 Rapid7 Forward DNS 数据集发现子域

参考https://github.com/rapid7/sonar/wiki/Forward-DNS

curl -silent https://scans.io/data/rapid7/sonar.fdns_v2/20170417-fdns.json.gz | pigz -dc | grep ".baidu.com" | jq

数据文件很大！

3. 主动子域名

什么是主动子域名枚举？

• Active sub-domain enumeration is where an attacker/tester gathers sub-domain information by probing the infrastructure managed by the target organization
  • 主动子域枚举是攻击者/测试者通过探测目标组织管理的基础设施来收集子域信息的地方
• Sometimes the target organization might have delegated the maintainence of infrastructure to 3rd party and attacker probes the infra maintained by a third party example: nameservers
  • 有时目标组织可能已将基础设施的维护委托给 3rd 方，而攻击者会探测由第三方维护的基础设施，例如：名称服务器
• The key characteristic of active enumeration is that it generates traffic that may possibly lead to detection and can point to the attacker/tester
  • 主动枚举的关键特征是它生成的流量可能会导致检测并可以指向攻击者/测试者

3.1 基于字典的枚举

3.1.1 Subbrute

• GitHub - TheRook/subbrute: A DNS meta-query spider that enumerates DNS records, and subdomains.
  • 根据DNS记录查询子域名

安装

aptitude install python-dnspython
git clone https://github.com/TheRook/subbrute.git
cd subbrute
make

3.1.2 DNSRecon

• GitHub - darkoperator/dnsrecon: DNS Enumeration Script
• DNSRecon 是一个强大的 DNS 枚举工具，它的一个特点是使用预定义的词表进行基于字典的子域枚举。

安装

aptitude install dnsrecon   # kali
# or python3.6+
git clone https://github.com/darkoperator/dnsrecon.git
cd dnsrecon
pip install -r requirements.txt

# 暴力破解
python3.7 dnsrecon.py -d baidu.com -D wordlist.txt -t brt

# DNS
python3.7 dnsrecon.py -t snoop -D wordlist.txt -n 1.1.1.1
# 1.1.1.1 是目标的NS server地址

其他参数选项：

• --threads 8 # 线程
• -n nsserver.com # 使用自定义的解析服务器
  • 输出选项：
  • --db: SQLite 3 文件
  • --xml: XML 文件
  • --json: JSON 文件
  • --csv: CSV 文件

3.2 Permutation scanning

Permutation scanning is another interesting technique to identify sub-domains. In this technique, we identify new sub-domains using permutations, alterations and mutations of already known domains/sub-domains.

3.2.1 aultdns

GitHub - infosec-au/altdns: Generates permutations, alterations and mutations of subdomains and then resolves them

安装

#Version: Python2
pip install py-altdns

altdns -i subdomains.txt -o data_output -w words.txt -r -s results_output.txt

3.3 区域传输

区域传输是一种 DNS 事务，其中 DNS 服务器将其区域文件的全部或部分副本传递给另一台 DNS 服务器。

dig +multi AXFR @ns_server target.com

更多参考：

• https://www.cnblogs.com/cyjaysun/p/4265245.html
• https://blog.csdn.net/n5xxxx__zy/article/details/90704865
• DNS Zone Transfer | Briskinfosec

3.4 Zone walking DNSSEC

• DNSSEC provides a layer of security by adding cryptographic signatures to existing DNS records
  • DNSSEC 通过向现有 DNS 记录添加加密签名来提供一层安全性
• These signatures are stored alongside common record types like A, AAAA, MX
  • 这些签名与 A、AAAA、MX 等常见记录类型一起存储

3.4.1 DNSSEC - New records

更多详细内容：https://appsecco.com/books/subdomain-enumeration/active_techniques/zone_walking.html

3.5 DNS 记录

3.5.1 CNAME 记录

CNAME 代表规范名称。CNAME 记录可用于将一个名称别名为另一个名称。CNAME 记录将具有主机名的值。有时，CNAME 会显示组织的子域或显示有关在域上运行的服务类型的信息。

3.5.2 SPF记录

参考：OSINT Through Sender Policy Framework (SPF) Records | Rapid7 Blog

3.6 HTTP 标头下的子域枚举

很少有安全头暴露子域名信息。

CSP定义的Content-Security-Policy请求头字段，它允许您创建可信内容来源的白名单，并指示浏览器仅执行或呈现来自这些来源的资源。所以基本上，Content-Security-Policy标头将列出我们作为攻击者可能感兴趣的一堆来源（域）。有不推荐使用的 CSP 标头形式，它们是 X-Content-Security-Policy 和X-Webkit-CSP

脚本工具：https://github.com/0xbharath/domains-from-csp

4. 扩展：常见工具

4.1 oneforall

https://github.com/shmilylty/OneForAll

安装

git clone https://github.com/shmilylty/OneForAll.git
cd OneForAll/
python3 -m pip install -U pip setuptools wheel -i https://mirrors.aliyun.com/pypi/simple/
pip3 install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
python3 oneforall.py --help

使用

python3 oneforall.py --target example.com run
python3 oneforall.py --targets ./example.txt run

4.2 Sublist3r

https://github.com/aboul3la/Sublist3r

• Baidu, Yahoo, Google, Bing, Ask, Netcraft, DNSdumpster, VirusTotal, Threat Crowd, SSL Certificates, PassiveDNS

安装

git clone https://github.com/aboul3la/Sublist3r.git
cd Subllist3r
pip3 install -r requirements.txt

使用

python3 sublist3r.py -d target.com -o $outfile
# 暴力破解
python3 sublist3r.py -b -d target.com -o $outfile

其他选项：

• -p 80,443 # 仅显示开放80,443端口的域名
• -e google,yahoo,virustotal # 仅使用google,yahoo,virustotal枚举子域名

4.3 Gobuster

https://github.com/OJ/gobuster

安装

git clone https://github.com/OJ/gobuster.git
cd gobuster/
go get && go build
go install

简单使用

gobuster -m dns -u target.com -w $wordlist

4.4 amass

https://github.com/OWASP/Amass

安装

sudo snap install amass

简单使用

amass enum -d example.com

4.5 Subfinder

https://github.com/subfinder/subfinder

VirusTotal, PassiveTotal, SecurityTrails, Censys, Riddler, Shodan, Bruteforce

安装：https://github.com/projectdiscovery/subfinder/releases/tag/v2.4.8

./subfinder -d target.com -o $outfile

4.6 theHarvester

https://github.com/laramies/theHarvester

安装

#kali

#or
$ ~ > git clone https://github.com/laramies/theHarvester 
$ ~ > cd theHarvester

If developing do:
$ ~ > python3 -m pip install -r requirements/dev.txt
Else:
$ ~ > python3 -m pip install -r requirements/base.txt

$ ~ > python3 theHarvester.py -h  

使用：

theharvester -d target.com -b all

4.7 Sudomy

https://github.com/Screetsec/Sudomy

./sudomy -d target.com -dP -eP -rS -cF -pS -tO -gW --httpx --dnsprobe  -aI webanalyze -sS

4.8 Knock

https://github.com/guelfoweb/knock

• AXFR, virustotal, brute-force

apt-get install python-dnspython
git clone https://github.com/guelfoweb/knock.git
cd knock
nano knockpy/config.json # <- 设置你的virustotal API_KEY
python setup.py install

knockpy target.com
knockpy target.com -w $wordlist

4.9 dnssearch

https://github.com/evilsocket/dnssearch

4.10 Dnscan

https://github.com/rbsec/dnscan

4.11 Domained

集成Sublist3r, Knock, Subbrute, Massdns, Recon-ng, Amass & SubFinder

https://github.com/cakinney/domained

4.12 subDomainsBrute

https://github.com/lijiejie/subDomainsBrute

4.13 Layer子域名挖掘机

4.14 teemo

https://github.com/bit4woo/teemo

5. 在线工具

• https://dnsdumpster.com/
• https://www.nmmapper.com/sys/tools/subdomainfinder/
• https://spyse.com/tools/subdomain-finder
• https://searchdns.netcraft.com/
• https://detectify.com/
• https://pentest-tools.com/information-gathering/find-subdomains-of-domain
• https://fofa.so/
• https://www.zoomeye.org/
• https://www.shodan.io/
• https://phpinfo.me/domain

6. 字典相关

• https://gist.github.com/jhaddix/86a06c5dc309d08580a018c66354a056
• https://github.com/assetnote/commonspeak2-wordlists
• https://github.com/danielmiessler/SecLists/tree/master/Discovery/DNS

7. 威胁情报数据

1. 微步：https://x.threatbook.cn/
2. alienvault：https://otx.alienvault.com/
3. riskiq：https://www.riskiq.com/
4. threatminer：https://www.threatminer.org/
5. virustotal：https://www.virustotal.com/gui/home/search

8. 细节

• robots文件
• sitemap文件
• 域传送漏洞
• js敏感信息泄露

9. 参考

• https://appsecco.com/books/subdomain-enumeration/
• https://pentester.land/cheatsheets/2018/11/14/subdomains-enumeration-cheatsheet.html#search-engines
• https://ricardoiramar.medium.com/subdomain-enumeration-tools-evaluation-57d4ec02d69e
• https://geekflare.com/find-subdomains/
• https://blog.csdn.net/w1590191166/article/details/104160404