前言一、requests模块使用1.1 requests模块发送get请求1.2 response响应对象1.3 response.text与response.content的区别1.4 通过对response.content进行decode,来解决中文乱码1.5 response响应对象的其他常用的属性和方法二、requests模块发送请求2.1 发送带headers的请求2.1.1思考2.1.2 携带请求头发送请求的方法2.2 发送带参数的请求2.2.1 在url携带参数2.2.2 通过params携带参数字典2.3 在headers参数中携带cookie2.3.1 github登录抓包分析3.3.2 完成代码2.4 cookie参数的使用2.5 cookiejar对象转换为cookies字典的方法2.6 超时timeout的使用2.7 代理proxies的使用2.7.1 理解使用代理的过程2.7.2正向代理和反向代理2.7.3 代理IP(代理服务器)的分类2.7.4 proxies代理参数的使用2.8 使用verify参数忽略CA证书三、 requests模块发送post请求3.1 requests发送post请求的方法四、利用requests.session进行状态保持4.1 requests.session的作用及应用场景4.2 requests.session的使用方法4.3 实例:模拟登录github精彩链接最后
爬虫的门槛不高,高就在于往后余生的每一次实操都会让你崩溃。在这个大数据的时代,数据就是金钱!所以越来越多的企业重视数据,然后再通过爬虫的手段获取公开的数数据,为企业项目进行赋能。
上一篇文章中,我带大家入门了爬虫,知道什么是爬虫,对爬虫有了大体的了解。
本篇博文将带领大家进入新的内容,爬虫最常用的库:requests库,最后并以综合案例模拟登录github
,带你实战。
本次文章主要分享的是requests这个http模块的使用,该模块主要用于发起请求获取响应,该模块有很多替代模块,比如说urllib模块,但是在工作中使用最多的是requests模块,requests的代码语法简单易懂,相对于臃肿的urllib模块,使用requests模块写爬虫会大大减少代码量,而且实现某一功能会更简单,因此推荐大家使用requests模块。
知识点
1、需求:通过requests向百度发送请求,获取页面的源码 2、运行下面代码观察打印结果
demo1.py
<pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="python" cid="n35" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: normal; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import requests
url = 'http://www.baidu.com'
response = requests.get(url)
print(response.text)</pre>
观察上面代码运行的结果观察发现,有好多乱码,这是因为编码与解码所使用的字符集不同造成的;我们尝试使用下边的办法来解决中文乱码问题。
demo2.py
<pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="python" cid="n40" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: normal; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import requests
url = 'http://www.baidu.com'
response = requests.get(url)
print(response.content.decode()) # 注意这里</pre>
1、response.text是requests模块按照charset模块推测出的编码字符串进行解码的结果。
2、网络传输的字符串都是bytes类型的数据,所以requests.text = response.content.decode('推测出来的编码字符集')
3、我们可以在网页源码中搜索charset,尝试参考该编码的字符集,注意:存在不准确的情况。
类型:str
解码类型:requests模块自动根据http头部对响应的编码做出有根据的推测,推测文本编码。
我们可以手动设定编码格式
demo3.py
<pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="python" cid="n52" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: normal; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import requests
url = 'http://www.baidu.com'
response = requests.get(url) response.encoding='utf-8'
print(response.text)</pre>
类型:bytes
解码类型:没有设定。可以自行进行设定。
知识点:掌握利用decode函数对requests.content解决中文乱码
utf-8
gbk
gb2312
ascill(读音:阿斯克码)
iso-8859-1
知识点:掌握利用decode函数对requests.content解决中文乱码
response = requests.get(url)中response是发送请求获取的响应对象;response响应对象中除了text,content获取响应内容以外还有其他常用的属性或方法。
demo4.py
<pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="python" cid="n97" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: normal; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import requests
url = 'http://www.baidu.com'
response = requests.get(url) response.encoding='utf-8' print(response.url) print(response.status_code) print(response.request.headers) print(response.headers) print(response.request._cookies) print(response.cookies)</pre>
知识点:掌握response响应对象的其他常用属性
我们先写一个获取百度首页的代码
demo5
.py
<pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="python" cid="n106" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: normal; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import requests
url = 'http://www.baidu.com'
response = requests.get(url)
print(response.content.decode())
print(response.request.headers)</pre>
1、对比浏览器上百度网页的源码和代码中百度首页的源码,看看有什么不同?
查看网页源代码的方法:
2、对比url响应内容和代码中的百度首页的源码,有什么不同?
3、代码中的百度首页的源码非常少,为什么?
需要带上请求头信息
回顾爬虫的概念,模拟浏览器,欺骗服务器,获取和浏览器一致的内容
请求头中有很多字段,其中User-Agent字段必不可少,表示客户端的操作系统以及浏览器的信息
<mark style="box-sizing: border-box; background: rgb(211, 212, 14); color: rgb(0, 0, 0);">requests.get(url, headers)</mark>
demo6.py
<pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="python" cid="n142" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: normal; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import requests
url = 'http://www.baidu.com'
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"}
response = requests.get(url, headers=headers)
print(response.content.decode())
print(response.request.headers) </pre>
我们在使用百度的时候经常发现URL地址中会有一个<mark style="box-sizing: border-box; background: rgb(211, 212, 14); color: rgb(0, 0, 0);">?</mark>,那么该问号后面的就是请求参数,又叫做查询字符串。
直接对含有参数的url发送请求
demo7.py
<pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="python" cid="n149" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: normal; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import requests
url = 'https://www.baidu.com/s?wd=Python'
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"}
response = requests.get(url, headers=headers) with open('baidu.html', 'wb') as f: f.write(response.content)</pre>
关键参数
demo8.py
<pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="python" cid="n158" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: normal; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import requests
url = 'https://www.baidu.com/s?'
kw = {'wd': 'python'}
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"}
response = requests.get(url, headers=headers, params=kw) with open('baidu1.html', 'wb') as f: f.write(response.content)</pre>
知识点:掌握发送带参数的请求方法
网站经常利用请求头中的Cookie字段来做用户状态的保持,那么我们可以在headers参数中添加Cookie,模拟普通用户的请求,我们以github为例。
image
demo9.py
<pre spellcheck="false" class="md-fences mock-cm md-end-block" lang="python" cid="n185" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: pre-wrap; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import requests
headers = { 'Cookie': '你的cookie' }
url = 'https://github.com/Zhimin7'
response = requests.get(url, headers=headers) with open('github_withcookie.html', 'wb') as f: f.write(response.content)</pre>
网页对比
接下来写一个不包含cookie的爬虫,看看对比后的结果
demo10.py
<pre spellcheck="false" class="md-fences mock-cm md-end-block" lang="python" cid="n189" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: pre-wrap; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import requests
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36'
}
url = 'https://github.com/Zhimin7'
response = requests.get(url, headers=headers) with open('github_without_cookie.html', 'wb') as f: f.write(response.content)</pre>
image
不同之处就相当明显了。
上一个小节中我们在headers参数中携带cookie,也可以使用专门的cookie参数
1.cookie参数的形式:字典
cookies = {'cookie的name' : 'cookie的value'}
2.cookies参数的使用方法
response = requests.get(url, cookies)
3.将cookie字符串转换为cookies参数所需要的字典
cookie_dict = {cookie.split('=')[0] : cookie.split('=')[-1] for cookie in temp.split(';')}
当然,如果你的字典生成式学的不够熟悉的话,那你可以使用较为稳妥方法
demo11.py
<pre spellcheck="false" class="md-fences mock-cm md-end-block" lang="python" cid="n210" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: pre-wrap; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">temp = 'octo=GH1.1.1102395001.1582362358; _ga=GA1.2.454155278.1582362359; _device_id=0442b4dd494cafc0301c2ad3e9eeca31; experiment:homepage_signup_flow=eyJ2ZXJzaW9uIjoiMSIsInJvbGxPdXRQbGFjZW1lbnQiOjI1LjY3MjIzNTIyOTQ0MTk1Miwic3ViZ3JvdXAiOiJjb250cm9sIiwiY3JlYXRlZEF0IjoiMjAyMC0wMy0yNlQxNDozNToxNC45ODdaIiwidXBkYXRlZEF0IjoiMjAyMC0wMy0yNlQxNDozNToxNC45ODdaIn0=; user_session=vsC4WPrJRjDLSTC3Up0h0D5i0Knfyah9hGXzhfrchfW_5eyc; __Host-user_session_same_site=vsC4WPrJRjDLSTC3Up0h0D5i0Knfyah9hGXzhfrchfW_5eyc; logged_in=yes; dotcom_user=Zhimin7; has_recent_activity=1; tz=Asia%2FShanghai; _gh_sess=e9HSDZpXyMNlwvsRH7kjV39DisarWcGKdXqnr65Z3VfFlChN0onUNHwROBPqX2yfS9WudAE71IQF2h7TRiVQ3rvVp1KbvbmfOOkULatFZsHoVRi5UUCI%2FY8wz0QVBLXF3VY0WgLwoUoZhaJ5MhPG%2F22am%2Bowt2XigTISZm289i%2BCYxkDvWz8N7J61WTPz9i3--3YPo3PUW%2B3asHJSS--AmjAHcbcaKfU%2BneNyzA13w%3D%3D' cookie_list = temp.split(';') cookies = {}
for cookie in cookie_list: cookies[cookie.split('=')[0]] = cookie.split('=')[-1] print(cookies)</pre>
使用request获取的Response对象,具有cookie属性。该属性值是一个cookieJar类型,包含了对方服务器设置在本地的cookie。我们如何将其转换为cookie字典呢?
1.转换方法
cookie_dict = requests.utils.dict_from_cookieJar(response.cookies)
2.其中response.cookies返回的就是cookieJar类型的对象。
3.requests.utils.dict_from_cookieJar函数返回cookie字典。
demo12.py
<pre spellcheck="false" class="md-fences mock-cm md-end-block" lang="python" cid="n219" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: pre-wrap; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">from requests import utils import requests
url = 'http://www.baidu.com' response = requests.get(url) print(type(response.cookies)) print(response.cookies)
dict_cookies = requests.utils.dict_from_cookiejar(response.cookies) print(dict_cookies)
jar_cookies = requests.utils.cookiejar_from_dict(dict_cookies) print(jar_cookies)</pre>
不过这种方法会造成域名缺失,不是很常用。在接下来的章节中会具体说明如何使用cookie保存会话。
在平时上网的过程中,我们经常会遇到网络波动,这个时候,一个请求等待了很久的时间仍然没有结果。 在爬虫中,一个请求很久没有结果,就会让整个项目的效率变得非常低,这个时候我们就需要对请求进行强制要求,让他必须在特定的时间内返回结果,否则就会报错。
1.超时参数timeout的使用方法
reponse = requests.get(url, timeout=3)
*timeout=3,表示3秒内程序访问服务器仍然没有响应,程序就会终止运行并报错
1.代理IP是一个IP,指向的是一个代理服务器
2.代理服务器能够帮我们向目标服务器发起请求
代理服务器的意思是在浏览器与服务器之间搭建一个桥梁,相当于用Python向代理服务器发起请求,在通过代理服务器向服务器发起请求。服务器返回响应也是如此,服务器将响应返回给代理服务器,代理服务器再将响应返回给浏览器。
前面提到proxy参数指定的代理IP指向的是正向代理服务器,那么相应的就有反向代理服务器;现在来了解一下正向代理服务器和反向代理服务器的区别
根据网站所使用的协议不同,需要使用相应协议的代理服务。从代理服务器请求的协议可以分为:
为了让服务器以为是不同客户端发送的请求,防止频繁向同一个域名发送请求被封IP,所以我们要使用代理IP。
response = requests.get(url, proxies=proxies)
proxies的形式:字典
<pre spellcheck="false" class="md-fences mock-cm md-end-block" lang="python" cid="n262" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: pre-wrap; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">proxies = { 'http':'http://12.32.56.78:8000', 'https':'https://12.32.56.78:8000' }</pre>
注意:如果proxies字典中含有多个键值对,发送请求的时候将按照url地址的协议来选择使用相应的代理IP。
在使用浏览器上网的时候,有时会看到,【您的链接不是私密连接】
所以作为爬虫,我们需要避免这种情况的发生,必须无视这个信息。
<pre spellcheck="false" class="md-fences mock-cm md-end-block" lang="python" cid="n271" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: pre-wrap; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import requests
url = '' # 填写没有认证的URL
response = requests.get(url, verify=False)</pre>
思考哪些地方会用到POST请求
所以同样我们爬虫也需要在这两个方面模拟浏览器发送post请求
requests模块中的session类能够自动的处理发送请求获取响应的过程中产生cookie,进而达到状态保持的目的
自动处理cookie,即下一次请求会自动带上前一次的cookie
自动处理连续请求多次请求过程产生的cookie
session示例在请求一个网站后,对方服务器设置在本地的cookie会保存在session中,下一次再用session请求网站的时候,会带上前一次的cookie
<pre spellcheck="false" class="md-fences mock-cm md-end-block" lang="python" cid="n304" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: pre-wrap; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">session = requests.session() #实例化session对象 response = session.get(url, headers, ...) response = session.post(url, data, ...)</pre>
github_sesseion.py
<pre spellcheck="false" class="md-fences mock-cm md-end-block" lang="python" cid="n310" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: Monaco, Consolas, "Andale Mono", "DejaVu Sans Mono", monospace; margin-top: 0px; margin-bottom: 20px; font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; white-space: pre-wrap; background: rgb(51, 51, 51); position: relative !important; padding: 10px 10px 10px 30px; width: inherit; color: rgb(184, 191, 198); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">import requests from lxml import etree
class GitHub(object): def init(self): self.session = requests.session() self.session.headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36' } self.login_url = 'https://github.com/login'
def login(self):
response = self.session.get(self.login_url)
html = etree.HTML(response.content.decode())
return html
def get_token(self):
authenticity_token = self.login().xpath('//form/input[1]/@value')[0]
return authenticity_token
def get_timestamp_secret(self):
timestamp_secret = self.login().xpath('//div[@class="auth-form-body mt-3"]/input[11]/@value')[0]
return timestamp_secret
def get_timestamp(self):
timestamp = self.login().xpath('//div[@class="auth-form-body mt-3"]/input[10]/@value')[0]
return timestamp
def get_profile(self):
url_session = 'https://github.com/session'
url_profile = 'https://github.com/Zhimin7'
data = {
'commit': 'Sign in',
'authenticity_token': self.get_token(),
'ga_id':'',
'login': '你的邮箱',
'password': '你的密码',
'webauthn - support': 'supported',
'webauthn - iuvpaa - support': 'supported',
'return_to':'',
'allow_signup':'',
'client_id':''
'integration:',
'required_field_86b0':'',
'timestamp': self.get_timestamp(),
'timestamp_secret': self.get_timestamp_secret()
}
self.session.post(url_session, data=data)
html = self.session.get(url_profile).content
with open('github.html', 'wb') as f:
f.write(html)
print('获取完毕')
if name == "main": github = GitHub() github.get_token() github.get_timestamp() github.get_timestamp_secret() github.get_profile()</pre>
原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。
原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。