爬虫指纹 JA3

小小咸鱼YwY

发布于 2022-05-09 15:22:16

8840

发布于 2022-05-09 15:22:16

JA3 指纹里面，很大的一块就是 Cipher Suits，也就是加密算法。而 requests 里面默认的加密算法如下参考文件:https://www.cnblogs.com/Eeyhan/p/15662849.html ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:!eNULL:!MD5

设计到代码

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.ssl_ import create_urllib3_context
 
ORIGIN_CIPHERS = ('ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES')
 
 
class DESAdapter(HTTPAdapter):
    def __init__(self, *args, **kwargs):
        """
        A TransportAdapter that re-enables 3DES support in Requests.
        """
        CIPHERS = ORIGIN_CIPHERS.split(':')
        random.shuffle(CIPHERS)
        CIPHERS = ':'.join(CIPHERS)
        self.CIPHERS = CIPHERS + ':!aNULL:!eNULL:!MD5'
        super().__init__(*args, **kwargs)
        
        
    def init_poolmanager(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=self.CIPHERS)
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).init_poolmanager(*args, **kwargs)
 
    def proxy_manager_for(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=self.CIPHERS)
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).proxy_manager_for(*args, **kwargs)

常出现情况有时候scrapy没法爬,而request可以爬

则需要在scrapy的配置文件中配置

DOWNLOADER_CLIENT_TLS_CIPHERS = 'DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:!eNULL:!MD5'

另外情况安全通过这个特效对你使用python,request进行检测

这时候有几个方法

1.访问ip指定host绕过waf

2.代理中转请求

在本地启动代理服务器，如Burp Suite，发起http请求时指定代理服务器为burp的地址，让burp来进行TLS握手，算是一种曲线救国的方法

3.更换request工具库

Requests其实是对urllib3的一个封装，那python有没有不用urllib的http request库呢？

翻了翻aiohttp的源码发现貌似并没有用urllib3，抓包发现tls指纹和requests也有着明显的差异

实际测试aiohttp确实没有被拦截

4.魔改requests

从根本上解决问题，debug跟踪到了几处可能可以修改TLS握手特征的代码

/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py