爬取巴比特快讯遇到状态码“521”

原创

bosh123

修改于 2021-01-29 14:23:58

9740

修改于 2021-01-29 14:23:58

文章被收录于专栏：my notebook my notebookmy notebook

https://www.jianshu.com/p/77dc4d75540d

最近在爬区块链相关的快讯，上周巴比特改版后重写了爬虫，跑了一天就挂了。原来是网站使用了加速乐的服务，爬虫每次都返回521的状态码。

浏览器访问网站时：第一次请求：返回521状态码和一段js代码。js会生成一段cookie并重新请求访问。第二次请求：带着第一次得到的cookie去请求然后正确返回状态码200

而爬虫不能像浏览器一样执行js所以一直报错521 解决办法：

让爬虫模拟浏览器的行为：将返回的js代码放在一个字符串中，然后利用execjs对这段代码进行解密，得到cookie放入下一次访问请求的头部中。

具体过程：

直接请求

将返回的这段js代码整理下：

<html>
<body>
<script language="javascript"> window.onload=setTimeout("ar(75)", 200); 
function ar(YH) {
    var qo, mo="", no="", oo = [0xc2,0x0c,0x22,0xa2,0x68,0x21,0xe8,0x3d,0x1e,0xbb,0x94,0x15,0x16,0x17,0x95,0x17,0x58,0x18,0xce,0xc6,0xc1,0xd6,0x16,0xb5,0x36,0xd6,0x96,0xd6,0xd0,0x2f,0x6f,0x50,0xd0,0x90,0x47,0x18,0xcd,0xa3,0x39,0x57,0x37,0x77,0x89,0x49,0x47,0x9d,0xdd,0x14,0x6a,0xab,0x8b,0x81,0x3f,0x15,0x4c,0xc2,0x49,0x68,0x08,0x1f,0x36,0xb6,0xec,0xaa,0x63,0x39,0x57,0xd7,0x6d,0x26,0x08,0x9d,0x1e,0x74,0x8b,0x44,0x84,0xb1,0x8f,0xe5,0x1d,0xd5,0xec,0x8c,0xa0,0xe0,0x18,0xd7,0x0f,0x46,0xe5,0x23,0x00,0xb6,0x37,0xb7,0x70,0xa6,0x4e,0x04,0x7a,0x18,0x0e,0xc3,0x79,0x4a,0x68,0xbe,0x74,0xeb,0x04,0xc3,0x67,0x86,0xa4,0xe5,0x44,0x04,0x82,0xcb,0x82,0x47,0x48,0x21,0xb9,0xd1,0xfa,0x51,0x6f,0x28,0x64,0x22,0x22,0xc0,0x71,0xaf,0xc6,0xde,0xf4,0x0c,0xd4,0x2c,0xe1,0xff,0x57,0xad,0x63,0x8c,0xa4,0xa8,0x65,0x07,0x7e,0x96,0xa7,0x47,0x48,0x01,0x41,0x82,0x63,0x33,0xe9,0xc2,0xd9,0x3a,0xdf,0x60,0x73,0x4c,0xcc,0xcd,0x8e,0x06,0x1e,0x1b,0x39,0x79,0x1f,0x40,0xf6,0xef,0xa3,0x9b,0x13,0x2b,0x29,0x6a,0x4b,0x6b,0x0b,0x0c,0x0a,0xe2,0x82,0x83,0x27,0xa7,0x65,0x26,0xe5,0xc6,0x64,0xef,0xc8,0x61,0x62,0xe2,0x23,0xc8,0xd0,0x0a,0x0b,0xeb,0xa2,0x42,0x43,0xee,0x6f,0x2d,0xed,0xad,0x8e,0x2c,0xfc,0xd5,0x97,0xf1,0xf0,0x3b];
    qo = "qo=228; do{oo[qo]=(-oo[qo])&0xff; oo[qo]=(((oo[qo]>>3)|((oo[qo]<<5)&0xff))-251)&0xff;} while(--qo>=2);"; 
    eval(qo);
    qo = 227; 
    do { 
        oo[qo] = (oo[qo] - oo[qo - 1]) & 0xff; 
    } while (-- qo >= 3 );
    qo = 1; 
    for (;;) { 
        if (qo > 227) break; 
        oo[qo] = ((((((oo[qo] + 28) & 0xff) + 148) & 0xff) << 6) & 0xff) | (((((oo[qo] + 28) & 0xff) + 148) & 0xff) >> 2); 
        qo++;
    }
    po = ""; 
    for (qo = 1; qo < oo.length - 1; qo++) if (qo % 7) po += String.fromCharCode(oo[qo] ^ YH);
    eval("qo=eval;qo(po);");
} 
</script> 
</body>
</html>

然后存为html文件用Chrome打开调试，在eval处打上断点。可以看到变量po的值："document.cookie='_ydclearance=5640fae72a12f756938d88c1-60c4-4c28-a629-8da9e99d65cc-1534755025; expires=Mon, 20-Aug-18 08:50:25 GMT; domain=.8btc.com; path=/'; window.document.location=document.URL" 而字符串po的前半段的意思是给浏览器添加Cooklie,后半段window.document.location=document.URL是刷新当前页面。

所以我们的关键点是要获得cookie。python中可以用execjs执行js:

import requests
import re
import execjs

headers = {
        'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/65.0.3325.181 Chrome/65.0.3325.181 Safari/537.36',
    }

def get_html(url):
    first_html = requests.get(url=url,headers=headers).content.decode('utf-8')
    return first_html


def executejs(first_html):
    # 提取其中的JS加密函数
    js_string = ''.join(re.findall(r'(function .*?)</script>', first_html))

    # 提取其中执行JS函数的参数
    js_arg = ''.join(re.findall(r'setTimeout\(\"\D+\((\d+)\)\"', first_html))
    js_name = re.findall(r'function (\w+)',js_string)[0]

    # 修改JS函数，使其返回Cookie内容
    js_string = js_string.replace('eval("qo=eval;qo(po);")', 'return po')

    func = execjs.compile(js_string)
    return func.call(js_name,js_arg)

def parse_cookie(string):
    string = string.replace("document.cookie='", "")
    clearance = string.split(';')[0]
    return {clearance.split('=')[0]: clearance.split('=')[1]}



def return_cookie(url):
    first_html = get_html(url)
    # 执行JS获取Cookie
    cookie_str = executejs(first_html)

    # 将Cookie转换为字典格式
    cookie = parse_cookie(cookie_str)
    print('cookies = ',cookie)
    return cookie


return_cookie(url='https://www.8btc.com/flash')

#结果：
cookies =  {'_ydclearance': '8c83e7fe9d6bd359e1eedc40-b55a-4ab5-98e2-22eb9b2ea9a7-1534917111'}
[Finished in 2.0s]

悲伤的夏洛特@席玉铎666