常用自动化操作模块特征隐藏

不止于python

发布于 2023-12-14 09:21:08

3260

发布于 2023-12-14 09:21:08

前言

爬虫的路上总有我们这些小白解不了的密，反不了的爬。这时候就需要自动化工具了，但是一般情况下，直接使用自动化工具都会被目标网站监测到，因为有几十个特征会被暴露的特征。所以这篇文章写一下，常见的浏览器如何执行js，和隐藏浏览器特征。文章不会涉及到配安装和配置环境步骤。自行查教程

selemium

最早接触的自动化模块

# -*- coding: utf-8 -*-
# @Author: Mehaei
# @Date: 2023-12-07 19:58:47
# @Last Modified by: Mehaei
# @Last Modified time: 2023-12-07 21:03:31
import time
from selenium import webdriver


def start():
    driver = webdriver.Chrome()
    with open('stealth.min.js', 'r') as f:
        js = f.read()
    driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {'source': js})
    driver.get("https://bot.sannysoft.com/")
    time.sleep(60)


if __name__ == '__main__':
    start()

pyppeteer

实测还是会有少部分特征会无法隐藏, 不过还有其它办法 pyppeteer_stealth隐藏pyppeteer特征天花板神

# -*- coding: utf-8 -*-
# @Author: Mehaei
# @Date: 2023-12-07 19:58:47
# @Last Modified by: Mehaei
# @Last Modified time: 2023-12-07 21:22:31
import asyncio
from pyppeteer import launch


async def start():
    browser = await launch(headless=False)
    page = await browser.newPage()
    with open('stealth.min.js', 'r') as f:
        js = f.read()
    await page.evaluateOnNewDocument(js)
    await page.goto("https://bot.sannysoft.com/")
    await asyncio.sleep(60)


if __name__ == '__main__':
    asyncio.get_event_loop().run_until_complete(start())

playwright

新一代爬虫工具可以录制手动的操作，自动生成代码。自动化神器
官网 https://playwright.dev/

# -*- coding: utf-8 -*-
# @Author: Mehaei
# @Date: 2023-12-07 19:58:47
# @Last Modified by: Mehaei
# @Last Modified time: 2023-12-07 20:52:55
import time
from playwright.sync_api import sync_playwright


def start():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        context = browser.new_context()
        context.add_init_script(path='stealth.min.js')
        page = context.new_page()
        page.goto("https://bot.sannysoft.com/", timeout=100000)
        time.sleep(60)


if __name__ == '__main__':
    start()

DrissionPage

新的自动化工具, 同时兼容requests便利性和自动化工具的强大行且会自动隐藏掉一些自动化特征和无需安装驱动，感兴趣的可以看官网
https://g1879.gitee.io/drissionpagedocs/

# -*- coding: utf-8 -*-
# @Author: Mehaei
# @Date: 2023-12-07 19:58:47
# @Last Modified by: Mehaei
# @Last Modified time: 2023-12-07 22:02:58
import time
from DrissionPage import ChromiumPage


def start():
    page = ChromiumPage()
    with open('stealth.min.js', 'r') as f:
        js = f.read()
    """
    运行js, 但是运行这个stealth脚本会报错
    """
    # page.run_js(js)
    page.get("https://bot.sannysoft.com/")
    time.sleep(60)


if __name__ == '__main__':
    start()

本文参与腾讯云自媒体分享计划，分享自微信公众号。

原始发表：2023-12-07，如有侵权请联系 cloudcommunity@tencent.com 删除

自动化