985校训中的频繁词

用户6021899

发布于 2019-10-30 11:29:10

4260

发布于 2019-10-30 11:29:10

文章被收录于专栏：Python编程 pyqt matplotlib

本篇通过爬虫和Fp-growth的简单应用，从网页上记载的985校训中发现频繁词。

首先利用requests模块爬取上述指定网页的全部html内容。

import requests
import re
from bs4 import BeautifulSoup


def download(url,user_agent='wswp',num_retries=2,proxies=None):
    print("Downloading: ", url)
    headers = {'User-Agent' : user_agent}
    resp = requests.get(url, headers=headers, proxies=proxies)
    html = None
    try:
        resp = requests.get(url, headers=headers, proxies=proxies)
        html = resp.text
        if resp.status_code >= 400:
            print("Download error: ", html)
            html = None
            if num_retries>0 and 500 < resp.status_code <600:
                #递归调用，遇到5xx错误，最多重试 2 次
                return download(url, user_agent, num_retries-1, proxies)
    except requests.exceptions.RequestException as e:
        print('Download error: ' ,e.reason)
        html = None
    finally:
        return html
        
url = 'https://baijiahao.baidu.com/s?id=1597717152971067895&wfr=spider&for=pc'
html = download(url)

接着利用BeautifulSoup提取我们感兴趣的内容，即校训部分：

soup = BeautifulSoup(html, 'html.parser')
html = soup.prettify() #修正可能存在的Html错误
print()
mottos = []
for matched in soup.find_all("span", attrs = {"class": "bjh-p"}): #提取
    text = matched.text
    print(matched.text) #会自动去掉多余的空格符
    if "：" in text:#去掉非校训部分
        mottos.append(text.split("：")

注意，这个985名单好像不全。

然后利用jieba分词库将各个校训分词：

import jieba
words = []
for motto in mottos:
    words.append([x for x in jieba.lcut(motto[1]) if x!=' ' ]) #分词，并去掉空格符
print("共有%d条校训"%len(words))
print(words)

最后利用FP-growth算法发现校训中的频繁项集：

import fpGrowth_py36 as fpG
def findFreq(dataset, minSup):
    initSet = fpG.createInitSet(dataset)
    myFPtree, myHeaderTab = fpG.createTree(initSet, minSup)
    freqList = []
    if myFPtree is not None:
        #myFPtree.disp()
        fpG.mineTree(myFPtree, myHeaderTab, minSup, set([]), freqList)
    return freqList
    
dataset = words
minSup = 4
freqList = findFreq(dataset, minSup)
print("支持度为%d时，频繁项数为%d:"%(minSup, len(freqList)))
print("频繁项集为:\n", freqList)