专栏首页Python七号微信公众号文章爬虫,这个就够了

微信公众号文章爬虫,这个就够了

我订阅了近 100 个公众号,有时候想再找之前读过的文章,发现搜索起来特别困难,如果忘了收藏,估计得找半小时,更让人无语的是,文章已经发布者删除,或者文章因违规被删除。那么有没有这样的爬虫,可以将公众号的文章全部爬到本地,并提供便捷的搜索功能,这样当我想查找某类文章的时候会非常方便,同时文章都在本地,也不用担心被人删除。

最近正好看到一个牛逼的 Python 爬虫项目,就是爬取微信公众号的文章的,看了一下功能介绍,真是想见恨晚啊,作者水平真的是牛逼,我已经献出了自己的崇拜,特分享出来,你可以使用它的功能,也可以研究它的技术,请拿走不谢。访问项目地址,相信你完全有独立部署的能力。

项目地址:https://github.com/wonderfulsuccess/weixin_crawler (阅读原文可直接访问链接)

功能展示

UI主界面

爬虫主界面.gif

添加公众号爬取任务和已经爬取的公众号列表

公众号.png

爬虫界面

设置界面

设置.png

公众号历史文章列表

历史文章列表.gif

报告

报告.gif

搜索

搜索.gif

简介

weixin_crawler是一款使用Scrapy、Flask、Echarts、Elasticsearch等实现的微信公众号文章爬虫,自带分析报告和全文检索功能,几百万的文档都能瞬间搜索。weixin_crawler设计的初衷是尽可能多、尽可能快地爬取微信公众的历史发文。

weixin_crawler 尚处于维护之中, 方案有效, 请放心尝试。

免部署马上体验公众号数据采集

通过免安装可执行程序WCplus.exe https://shimo.im/docs/E1IjqOy2cYkPRlZd 可马上体验weixin_crawler的数据采集功、导出Excel和PDF功能。

主要特点

  1. 使用Python3编写 Python3 is used
  2. 爬虫框架为Scrapy并且实际用到了Scrapy的诸多特性,是深入学习Scrapy的不错开源项目 Made full use of scrapy, if you are struggling with scrapy this repo helps to spark
  3. 利用Flask、Flask-socketio、Vue实现了高可用性的UI界面。功能强大实用,是新媒体运营等岗位不错的数据助手 Flask、Flask-socketio、Vue are used to build a full stack project crawler
  4. 得益于Scrapy、MongoDB、Elasticsearch的使用,数据爬取、存储、索引均简单高效 Thanks to scrapy mongodb elasticsearch weixin_crawler is not only a crawler but also a search engine
  5. 支持微信公众号的全部历史发文爬取 Able to crawl all the history articles of any weixin official account
  6. 支持微信公众号文章的阅读量、点赞量、赞赏量、评论量等数据的爬取 Able to crawl the reading data
  7. 自带面向单个公众号的数据分析报告 Released with report module based on sigle official account
  8. 利用Elasticsearch实现了全文检索,支持多种搜索和模式和排序模式,针对搜索结果提供了趋势分析图表 It is also a search engine
  9. 支持对公众号进行分组,可利用分组数据限定搜索范围 Able to group official account which can be used to define searching range
  10. 原创手机自动化操作方法,可实现爬虫无人监管 Whith the help of adb, weixin_crawler is able to opereate Android phone automatically, which means it can work without any human monitoring
  11. 支持多微信APP同时采集, 理论上采集速度可线性增加 Mutiple weixin app is supported to imporove crawling speed linearly

使用到的主要工具

语言

Python3.6

前端

web框架

Flask / Flask-socketio / gevent

js/css库

Vue / Jquery / W3css / Echarts / Front-awsome

后端

爬虫

Scrapy

存储

Mongodb / Redis

索引

Elasticsearch

运行方法

weixin_crawler已经在Win/Mac/Linux系统下运行成功, 建议优先使用win系统尝试 weixin_crawler could work on win/mac/linux, although it is suggested to try on win os firstly

Insatall mongodb / redis / elasticsearch and run them in the background

  1. downlaod mongodb / redis / elasticsearch from their official sites and install them
  2. run them at the same time under the default configuration. In this case mongodb is localhost:27017 redis is localhost:6379(or you have to config in weixin_crawler/project/configs/auth.py)
  3. Inorder to tokenize Chinese, elasticsearch-analysis-ik have to be installed for Elasticsearch

Install proxy server and run proxy.js

  1. install nodejs and then npm install anyproxy and redis in weixin_crawler/proxy
  2. cd to weixin_crawler/proxy and run node proxy.js
  3. install anyproxy https CA in both computer and phone side
  4. if you are not sure how to use anyproxy, here is the doc

Install the needed python packages

  1. NOTE: you may can not simply type pip install -r requirements.txt to install every package, twisted is one of them which is needed by scrapy.
  2. I am not sure if your python enviroment will throw other package not found error, just install any package that is needed

Some source code have to be modified(maybe it is not reasonable)

  1. scrapy Python36\Lib\site-packages\scrapy\http\request\ __init__.py --> weixin_crawler\source_code\request\__init__.py
  2. scrapy Python36\Lib\site-packages\scrapy\http\response\ __init__.py --> weixin_crawler\source_code\response\__init__.py
  3. pyecharts Python36\Lib\site-packages\pyecharts\base.py --> weixin_crawler\source_code\base.py. In this case function get_echarts_options is added in line 106

If you want weixin_crawler work automatically those steps are necessary or you shoud operate the phone to get the request data that will be detected by Anyproxy manual

  1. Install adb and add it to your path(windows for example)
  2. install android emulator(NOX suggested) or plugin your phone and make sure you can operate them with abd from command line tools
  3. If mutiple phone are connected to your computer you have to find out their adb ports which will be used to add crawler
  4. adb does not support Chinese input, this is a bad news for weixin official account searching. In order to input Chinese, adb keyboard has to be installed in your android phone and set it as the default input method, more is here

Why could weixin_crawler work automatically? Here is the reason:

  • If you want to crawl a wechat official account, you have to search the account in you phone and click its "全部消息" then you will get a message list , if you roll down more lists will be loaded. Anyone of the messages in the list could be taped if you want to crawl this account's reading data
  • If a nickname of a wechat official account is given, then wexin_crawler operate the wechat app installed in a phone, at the same time anyproxy is 'listening background'…Anyway weixin_crawler get all the request data requested by wechat app, then it is the show time for scrapy
  • As you supposed, in order to let weixin_crawler operate wechat app we have to tell adb where to click swap and input, most of them are defined in weixin_crawler/project/phone_operate/config.py. BTW phone_operate is responsible for wechat operate just like human beings, its eyes are baidu OCR API and predefined location tap area, its fingers are adb

Run the main.py cd weixin_crawler/project/ python(3) ./main.py Now open the browser and everything you want would be in localhost:5000. In this long step list you may get stucked, join our community for help, tell us what you have done and what kind of error you have found. Let's go to explore the world in localhost:5000 together

(完)

本文分享自微信公众号 - Python七号(PythonSeven),作者:somenzz

原文出处及转载信息见文内详细说明,如有侵权,请联系 yunjia_community@tencent.com 删除。

原始发表时间:2019-03-10

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

我来说两句

0 条评论
登录 后参与评论

相关文章

  • Django的设计哲学

    Django 读作姜戈,第一个 D 不发音,与电影《被解救的姜戈》的姜戈除了读音一样,没有其他半毛钱关系。Django 是一个优秀的 Web 框架,用 Pyth...

    somenzz
  • 在Python中如何基于接口编程

    软件行业,唯一不变的就是变化。产品经理会变,产品需求会变,代码同样要跟着变。不同的代码设计,变化所带来的工作量更是不同,有的每改一次需求,近乎一次重构,而有的只...

    somenzz
  • Python 排序-插入排序-优化

    插入排序,我想你也并不陌生。可以简单地这样理解,插入排序就是就是往一个有序的数列中添中新的数据,插入之后保证数据列仍然有序,因此叫插入排序。

    somenzz
  • SAP CDS view自学教程之四:SAP CDS view注解@OData.publish的奥妙

    In part1 of this tutorial, the old way to create OData service on top of CDS vie...

    Jerry Wang
  • TypeScript 3.7 RC 发布,备受瞩目的 Optional Chaining 来了

    We’re pleased to announce TypeScript 3.7 RC, the release candidate of TypeScript...

    苏南
  • rxjs switchMap的实现原理

    SwitchMap can cancel in-flight network requests.

    Jerry Wang
  • 噪音:让生成模型稳定训练的技巧

    Generative Adversarial Networks (GANs) are notoriously hard to train. In a recen...

    用户1908973
  • Filebeat快速入门

    可能需要额外授予执行权限。命令为:PowerShell.exe -ExecutionPolicy RemoteSigned -File .\install-s...

    趣学程序-shaofeer
  • 如何使用SAP HANA Studio的PlanViz分析CDS view性能问题

    I am a newbie in CDS performance area and frequently I meet with some “weird” pe...

    Jerry Wang
  • SAP Cloud for Customer页面根据后台数据响应的刷新原理介绍

    Recently I am working with partner and they are asking for the reason of one UI5...

    Jerry Wang

扫码关注云+社区

领取腾讯云代金券