盘点一个Python网络爬虫问题

Python进阶者

发布于 2023-08-31 08:27:25

1900

发布于 2023-08-31 08:27:25

文章被收录于专栏：Python爬虫与数据挖掘

一、前言

前几天在Python最强王者群【刘桓鸣】问了一个Python网络爬虫的问题，这里拿出来给大家分享下。

他自己的代码如下：

import requests

key = input("请输入关键字")

res = requests.post(
    url="https://jf.10086.cn/cmcc-web-shop/search/query",
    data={
    "sortColumn" : "default",
    "sortType": "DESC",
    "pageSize": "60",
    "pageNum": "1",
    "firstKeyword": key,
    "integral": "",
    "province": ""},
    headers={"User_Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"}
)


print(res.json())

二、实现过程

这里【隔壁😼山楂】指出拿到的数据需要用json解析，后来【瑜亮老师】指出是参数加少了。

甯同学指出，需要在请求头里边加上origin，后来【eric】给出了一个对应代码，如下所示：

import requests


headers = {
    "authority": "jf.10086.cn",
    "accept": "*/*",
    "accept-language": "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6",
    "content-type": "application/x-www-form-urlencoded;charset=UTF-8",
    "origin": "https://jf.10086.cn",
    "referer": "https://jf.10086.cn/",
    "sec-ch-ua": "\"Microsoft Edge\";v=\"113\", \"Chromium\";v=\"113\", \"Not-A.Brand\";v=\"24\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "\"Windows\"",
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "same-origin",
    "sessionid": "",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.42"
}
cookies = {
    "sajssdk_2015_cross_new_user": "1",
    "sensorsdata2015jssdkcross": "%7B%22distinct_id%22%3A%221882e060ca319-0c9999999999998-7b515477-921600-1882e060ca46ed%22%2C%22first_id%22%3A%22%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%7D%2C%22identities%22%3A%22eyIkaWRlbnRpdHlfY29va2llX2lkIjoiMTg4MmUwNjBjYTMxOS0wYzk5OTk5OTk5OTk5OTgtN2I1MTU0NzctOTIxNjAwLTE4ODJlMDYwY2E0NmVkIn0%3D%22%2C%22history_login_id%22%3A%7B%22name%22%3A%22%22%2C%22value%22%3A%22%22%7D%2C%22%24device_id%22%3A%221882e060ca319-0c9999999999998-7b515477-921600-1882e060ca46ed%22%7D",
    "BSFIT_EXPIRATION": "1684453169465",
    "BSFIT_DEVICEID": "eNgfgPaqBaS8qXzJHKXgXxJUCen3U5WF8tO1cjBaMqaDL7EKt2xK0J5XwThnB_kC-VbJC2t-N4axkF2UXAKhRvM7w7kNMRWX8pyxlMitEPPbnWVSnXSU4e2MZvpGBme1L3PX7et2B40xYhXg0MpYpfmUtnuJJTEQ"
}
url = "https://jf.10086.cn/cmcc-web-shop/search/query"
data = {
    "sortColumn": "default",
    "sortType": "DESC",
    "pageSize": "60",
    "pageNum": "1",
    "firstKeyword": "食用油",
    "integral": "",
    "province": ""
}
response = requests.post(url, headers=headers, cookies=cookies, data=data)

代码运行之后，可以得到预期的数据：