爬取搞笑视频

HuangWeiAI

发布于 2019-11-01 09:32:31

8030

发布于 2019-11-01 09:32:31

文章被收录于专栏：浊酒清味

最近小编经常刷知乎上的一个问题“你见过哪些是「以为是个王者，结果是个青铜」的视频或图片？”。从这个问题我们就已经可以看出来里面的幽默成分了，点进去看果然是笑到停不下来。于是，我想一个个点进去看，还不如把这些视频都下载下来，享受一顿爆笑。

获取url

我们使用Google浏览器的“开发者工具”获取网页的url，然后用requests.get函数获得json文件，再使用json.loads函数转换成Python对象：

url = "https://www.zhihu.com/api/v4/questions/312311412/answers?include=data%5B%2A%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cis_labeled%3Bdata%5B%2A%5D.mark_infos%5B%2A%5D.url%3Bdata%5B%2A%5D.author.follower_count%2Cbadge%5B%2A%5D.topics&limit=20&offset="+str(i*20)+"&platform=desktop&sort_by=default"
r = requests.get(url,headers = kv)
dicurl = json.loads(r.text)

获取content

我们使用谷歌浏览器的一个开发者工具JSONview，可以看到打开的url中有一个content，这里面就是我们要找的回答内容，视频url也在里面。将返回的json转化成python对象后，获取其中content里面的内容。也就是说，我们获得了每一个回答的内容，包括了视频的地址。

for k in range(20):#每条dicurl里可以解析出20条content数据
    name = dicurl["data"][k]["author"]["name"]
    ID = dicurl["data"][k]["id"]
    question = dicurl["data"][k]["question"]["title"]
    content = dicurl["data"][k]["content"]
    data_lens = re.findall(r'data-lens-id="(.*?)"',content)

获得视频地址

打开获取的content，找到href后面的url，打开看一下打开后视频正是我们要的内容，但是发现url不是我们获取的真实地址。仔细观察后发现，这个url发生了跳转。想要知道如何跳转来的，我们再次F12，打开开发者工具，发现请求了一个新的URL。观察发现，其实后面一串数字就是之前的data-lens-id。

对这个地址进行构造：

videoUrl = "https://lens.zhihu.com/api/v4/videos/"+str(data_lens[j])
R = requests.get(videoUrl,headers = kv)
Dicurl = json.loads(R.text)
playurl = Dicurl["playlist"]["LD"]["play_url"]
#print(playurl)#跳转后的视频url
videoread = request.urlopen(playurl).read()

完成之后，我们就可以下载视频了。

完整版代码：

from urllib import request
from bs4 import BeautifulSoup
import requests
import re
import json
import math
def getVideo():
    m = 0#计数字串个数
    num = 0#回答者个数
    path = u'/home/zhihuvideo1'
    #path = u'/home/zhihuimage'
    kv = {'user-agent':'Mozillar/5.0'}
    for i in range(math.ceil(900/20)):
        try:
            url = "https://www.zhihu.com/api/v4/questions/312311412/answers?include=data%5B%2A%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cis_labeled%3Bdata%5B%2A%5D.mark_infos%5B%2A%5D.url%3Bdata%5B%2A%5D.author.follower_count%2Cbadge%5B%2A%5D.topics&limit=20&offset="+str(i*20)+"&platform=desktop&sort_by=default"
            r = requests.get(url,headers = kv)
            dicurl = json.loads(r.text)
            for k in range(20):#每条dicurl里可以解析出20条content数据
                name = dicurl["data"][k]["author"]["name"]
                ID = dicurl["data"][k]["id"]
                question = dicurl["data"][k]["question"]["title"]
                content = dicurl["data"][k]["content"]
                data_lens = re.findall(r'data-lens-id="(.*?)"',content)
                print("正在处理第" + str(num+1) + "个回答--回答者昵称:" + name + "--回答者ID:" + str(ID) + "--" + "问题:" + question)
                num = num + 1  # 每次碰到一个content就增加1，代表回答者人数
                for j in range(len(data_lens)):
                    try:
                        videoUrl = "https://lens.zhihu.com/api/v4/videos/"+str(data_lens[j])
                        R = requests.get(videoUrl,headers = kv)
                        Dicurl = json.loads(R.text)
                        playurl = Dicurl["playlist"]["LD"]["play_url"]
                        #print(playurl)#跳转后的视频url
                        videoread = request.urlopen(playurl).read()
                        
                        fileName = path +"/" + str(m+1) + '.mp4'
                        print ('===============================================')
                        print(">>>>>>>>>>>>>>>>>第---" + str(m+1) + "---个视频下载完成<<<<<<<<<<<<<<<<<")
                        videoname = open(fileName,'wb')
                        
                        videoname.write(videoread)
                        m = m+1
                    except:
                        print("此URL为外站视频,不符合爬取规则")
        except:
            print("构造第"+str(i+1)+"条json数据失败")

if __name__ == "__main__":
    getVideo()

跑这个程序需要注意的是需要按照代码存储视频的路径建立一个文件夹：