前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >python抓取不得姐动图(报错 urllib.error.HTTPError: HTTP Error 403: Forbidden)

python抓取不得姐动图(报错 urllib.error.HTTPError: HTTP Error 403: Forbidden)

作者头像
未来sky
发布2018-08-30 14:31:25
1.3K0
发布2018-08-30 14:31:25
举报
文章被收录于专栏:好好学习吧好好学习吧

抓取不得姐动图(报错)

代码语言:javascript
复制
# -*- coding:utf-8 -*-
#__author__ :kusy
#__content__:文件说明
#__date__:2018/7/23 17:01
import urllib.request
import re

def getHtml(url):
    page = urllib.request.urlopen(url)
    html = page.read()
    # print(html)
    return html

def getImg(reg,savePath):
    iCnt = 0
    def giveImg(html):
        imgre = re.compile(reg)
        imglist = re.findall(imgre, html.decode('utf-8'))
        nonlocal iCnt
        for imgurl in imglist:
            urllib.request.urlretrieve(imgurl, savePath + '%s.gif' % iCnt)
            iCnt += 1
    return giveImg


# html = getHtml("http://pic.sogou.com/")
# reg = r'"image":"(.+?)"'  #sougou

reg = r'data-original="(.+?\.gif)"'
savePath = 'image/gif/'
g = getImg(reg,savePath)
for i in range(10):
    if i >1:
        print("http://www.budejie.com/" + str(i))
        html = getHtml("http://www.budejie.com/" + str(i))
    else:
        html = getHtml("http://www.budejie.com/")
    g(html)

报错如下

代码语言:javascript
复制
E:\kusy\python\venv\Scripts\python.exe E:/kusy/python/getJpg.py
http://www.budejie.com/2
Traceback (most recent call last):
  File "E:/kusy/python/getJpg.py", line 35, in <module>
    html = getHtml("http://www.budejie.com/" + str(i))
  File "E:/kusy/python/getJpg.py", line 9, in getHtml
    page = urllib.request.urlopen(url)
  File "C:\Users\jingjing\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\jingjing\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 532, in open
    response = meth(req, response)
  File "C:\Users\jingjing\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Users\jingjing\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 570, in error
    return self._call_chain(*args)
  File "C:\Users\jingjing\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "C:\Users\jingjing\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

Process finished with exit code 1

百度了下已解决:

代码语言:javascript
复制
# -*- coding:utf-8 -*-
#__author__ :kusy
#__content__:文件说明
#__date__:2018/7/23 17:01
import urllib.request
import re

def getHtml(url):
    # 如果不加上下面的这行出现会出现urllib.error.HTTPError: HTTP Error 403: Forbidden错误
    # 主要是由于该网站禁止爬虫导致的,可以在请求加上头信息,伪装成浏览器访问User-Agent,具体的信息可以通过火狐的FireBug插件查询
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0'}
    req = urllib.request.Request(url=url,headers=headers)
    page = urllib.request.urlopen(req)
    html = page.read()
    # print(html)
    return html

def getImg(reg,savePath):
    iCnt = 0
    def giveImg(html):
        imgre = re.compile(reg)
        imglist = re.findall(imgre, html.decode('utf-8'))
        nonlocal iCnt
        for imgurl in imglist:
            urllib.request.urlretrieve(imgurl, savePath + '%s.gif' % iCnt)
            iCnt += 1
    return giveImg


# html = getHtml("http://pic.sogou.com/")
# reg = r'"image":"(.+?)"'  #sougou

reg = r'data-original="(.+?\.gif)"'
savePath = 'image/gif/'
g = getImg(reg,savePath)
for i in range(10):
    if i >1:
        print("http://www.budejie.com/" + str(i))
        html = getHtml("http://www.budejie.com/" + str(i))
    else:
        html = getHtml("http://www.budejie.com/")
    g(html)

下载成功

本文参与 腾讯云自媒体分享计划,分享自作者个人站点/博客。
原始发表:2018-07-23 ,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档