现代网站很多数据不是直接写在 HTML 里,而是页面加载后由 JavaScript 发起 XHR / Fetch 请求,从后端接口取回 JSON 数据再渲染。所以爬虫不要只盯着网页源码,应该先找到真正的数据接口。
对爬虫来说,XHR/Fetch 请求通常有这些特征:
/api/、/data/、/list/ 等路径Cookie、Authorization、Referer、X-Requested-With以 Chrome / Edge 为例:
重点看这几项:
字段 | 作用 |
|---|---|
Request URL | 真实数据接口地址 |
Method | GET / POST |
Headers | 判断是否需要 Cookie、Token、Referer |
Query/Payload | 分页、关键词、筛选条件 |
Response | 确认是否返回目标 JSON 数据 |
如果右键请求选择 Copy as cURL,还可以快速验证接口是否能复现。
import requests
def fetch_json(url: str, headers: dict | None = None, params: dict | None = None):
"""模拟 XHR/Fetch GET 请求"""
if headers is None:
headers = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/120.0.0.0 Safari/537.36"
)
}
response = requests.get(url, headers=headers, params=params, timeout=10)
response.raise_for_status()
return response.json()
if __name__ == "__main__":
api_url = "https://wlt.njkhui.com/products"
data = fetch_json(
url=api_url,
params={"page": 1, "size": 20},
)
print(data)如果接口需要登录态或反爬校验,可以把浏览器里的 Cookie、Referer、Authorization 等补进 headers。
有些分页、搜索、筛选接口是 POST 请求,参数通常放在请求体中。
import requests
def post_json(url: str, payload: dict, headers: dict | None = None):
"""模拟 XHR/Fetch POST 请求"""
if headers is None:
headers = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/120.0.0.0 Safari/537.36"
),
"Content-Type": "application/json",
}
response = requests.post(url, json=payload, headers=headers, timeout=10)
response.raise_for_status()
return response.json()
if __name__ == "__main__":
api_url = "https://wlt.njkhui.com/search"
data = post_json(
url=api_url,
payload={
"keyword": "手机",
"page": 1,
"sort": "price_desc",
},
)
print(data)如果接口接收的是表单数据,可以把 json=payload 换成 data=payload,并调整 Content-Type。
很多动态列表接口通过 page、offset、cursor 控制分页。
import time
import random
import requests
def crawl_pages(base_url: str, total_pages: int = 5):
headers = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/120.0.0.0 Safari/537.36"
),
}
all_items = []
for page in range(1, total_pages + 1):
params = {"page": page, "size": 20}
response = requests.get(base_url, headers=headers, params=params, timeout=10)
response.raise_for_status()
data = response.json()
items = data.get("list", [])
all_items.extend(items)
print(f"第 {page} 页,获取 {len(items)} 条数据")
# 随机休眠,降低被封风险
time.sleep(random.uniform(1, 2))
return all_items
if __name__ == "__main__":
items = crawl_pages("https://wlt.njkhui.com/items", total_pages=3)
print(f"共获取 {len(items)} 条数据")问题 | 原因 | 处理方式 |
|---|---|---|
返回 403 | 缺少 UA、Cookie、Referer | 补全 headers |
返回空数据 | 接口参数不对 | 对比浏览器中的 Query/Payload |
Token 失效 | 登录态过期 | 重新抓取 Cookie/Token |
参数被加密 | 前端生成签名 | 分析 JS,或改用浏览器自动化 |
如果接口参数需要前端动态签名、加密,或者页面逻辑非常复杂,可以直接用浏览器自动化拦截 XHR/Fetch 响应。
from playwright.sync_api import sync_playwright
import json
def intercept_xhr(url: str, keyword: str = "api/data"):
results = []
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
def handle_response(response):
if keyword in response.url:
try:
data = response.json()
results.append(data)
except Exception:
pass
page.on("response", handle_response)
page.goto(url, wait_until="networkidle", timeout=30000)
browser.close()
return results
if __name__ == "__main__":
data_list = intercept_xhr("https://example.com/list", keyword="/api/list")
for item in data_list:
print(json.dumps(item, ensure_ascii=False, indent=2))最简单的工作流就是:打开 Network → 筛选 XHR/Fetch → 找到返回 JSON 的接口 → 复制 URL、Headers、参数 → 用 requests 复现。接口能直接复现时,通常比解析 HTML 更稳定、更高效。
原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。