如何渲染JS为cookie生成指纹？
如何渲染JS为cookie生成指纹？
提问于 2019-06-07 20:03:34
本网站使用JS设置cookie。
如何运行JS来模拟为浏览器以避免429错误？
from requests_html import HTMLSession
with HTMLSession() as s:
url = 'https://www.realestate.com.au/auction-results/nsw'
r = s.get(url)
print(r.status_code)
print(r.text)
r.html.render()
print(r.text)Stack Overflow用户
发布于 2019-08-23 22:24:22
如果没有某种形式的浏览器模拟，似乎几乎不可能绕过指纹(即使使用seleniumm，我也必须设置一些选项)。这是我想出的方法，使用Selenium来获取发出请求所需的唯一关键信息(一个名为‘FGJK’的cookie )，该信息在随后的请求头中发送，并使用async获取所有郊区结果页面。
from requests_html import AsyncHTMLSession
import asyncio
from selenium import webdriver
import nest_asyncio
#I'm using IPython which doesn't like async unless the following is applied:
nest_asyncio.apply()
async def get_token():
options = webdriver.ChromeOptions()
options.add_experimental_option('excludeSwitches', ['enable-automation'])
driver = webdriver.Chrome(options=options)
driver.get('https://www.realestate.com.au/auction-results/nsw')
cookies = driver.get_cookies()
while True:
for cookie in cookies:
if cookie['name'] == 'FGJK':
token = cookie['value']
return token
else:
cookies = driver.get_cookies()
async def get_results(s, endpoint, headers):
r = await s.get(f'https://www.realestate.com.au/auction-results/{endpoint}', headers=headers)
#do something with r.html
print(r, endpoint)
async def main():
token = await get_token()
s = AsyncHTMLSession()
headers = {'Cookie': f'FGJK={token}',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'}
r = await s.get(f'https://sales-events-api.realestate.com.au/sales-events/nsw')
suburbs = r.json()['data']['suburbResults']
endpoints = [burb['suburb']['urlValue'] for burb in suburbs]
asyncio.gather(*(get_results(s, endpoint, headers) for endpoint in endpoints))
asyncio.run(main()) 页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/56493960
复制相关文章
相似问题
腾讯云开发者
