我正在尝试抓取this页面,但使用以下代码时遇到了cookies问题:
SelectProxy.select_proxy()
local_proxy = SelectProxy.global_proxy
session = requests.Session()
session.proxies = {local_proxy}
cookies = session.cookies
url = movie_url
headers ={
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8',
'Cache-Control': 'max-age=0',
'Connection': 'keep-alive',
'Cookie': cookies,
'Host': 'www.sky.com',
'If-Modified-Since': 'Sat, 18 Aug 2018 14:45:31 GMT',
'Upgrade-Insecure-Requests': '1',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
}
r = session.get(url, headers=headers)
我得到的错误是这样的:
Traceback (most recent call last):
File "G:\Python27\Kodi\Sky Q Movies Scraper.py", line 33, in <module>
class sky_movies:
File "G:\Python27\Kodi\Sky Q Movies Scraper.py", line 90, in sky_movies
r = session.get(url, headers=headers)
File "G:\Python27\lib\site-packages\requests\sessions.py", line 488, in get
return self.request('GET', url, **kwargs)
File "G:\Python27\lib\site-packages\requests\sessions.py", line 461, in request
prep = self.prepare_request(req)
File "G:\Python27\lib\site-packages\requests\sessions.py", line 394, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "G:\Python27\lib\site-packages\requests\models.py", line 295, in prepare
self.prepare_headers(headers)
File "G:\Python27\lib\site-packages\requests\models.py", line 409, in prepare_headers
check_header_validity(header)
File "G:\Python27\lib\site-packages\requests\utils.py", line 800, in check_header_validity
"not %s" % (value, type(value)))
InvalidHeader: Header value <RequestsCookieJar[]> must be of type str or bytes, not <class 'requests.cookies.RequestsCookieJar'>
有人能告诉我哪里做错了吗?
谢谢
发布于 2018-08-19 03:19:54
基本上,如果请求从服务器接收到任何cookies,它们都被包装在一个CookieJar对象中。现在您尝试在头中使用该对象,它只接受字符串或字节。
正如heemayl正确地指出的那样,通常在请求中使用cookie的最佳方法是通过任何请求函数(例如get、post、head等)中的cookies参数传递它们。
如果您想要传递自己的cookies,则需要自己创建一个CookieJar对象,在jar上设置cookies,并通过cookies参数传递,如here所述。
https://stackoverflow.com/questions/51911742
复制相似问题