我正在尝试将一个javascripted网页呈现到填充的HTML中以便抓取。研究不同的解决方案(selenium,对页面进行逆向工程等)让我学到了this技术,但我不能让它工作。顺便说一句,我是python的新手,基本上处于剪切/粘贴/实验阶段。过去的安装和缩进问题,但我现在卡住了。
在下面的测试代码中,print(sample_html)工作并返回目标页面的原始html,但是print(render(sample_html))总是返回单词'None‘。
有趣的是,如果你在amazon.com上运行它,他们会检测到它不是真正的浏览器,并返回html并警告自动访问。然而,其他测试页面提供了应该呈现的真正的html,除了它没有。
如何诊断总是返回“None”的结果?
def render(source_html):
"""Fully render HTML, JavaScript and all."""
import sys
from PyQt5.QtWidgets import QApplication
from PyQt5.QtWebEngineWidgets import QWebEngineView
class Render(QWebEngineView):
def __init__(self, html):
self.html = None
self.app = QApplication(sys.argv)
QWebEngineView.__init__(self)
self.loadFinished.connect(self._loadFinished)
self.setHtml(html)
self.app.exec_()
def _loadFinished(self, result):
# This is an async call, you need to wait for this
# to be called before closing the app
self.page().toHtml(self.callable)
def callable(self, data):
self.html = data
# Data has been stored, it's safe to quit the app
self.app.quit()
return Render(source_html).html
import requests
#url = 'http://webscraping.com'
#url='http://www.amazon.com'
url='https://www.ncbi.nlm.nih.gov/nuccore/CP002059.1'
sample_html = requests.get(url).text
print(sample_html)
print(render(sample_html))
编辑:感谢您的回复,这些回复被合并到代码中。但是现在它返回一个错误,并且脚本挂起,直到我终止python启动器,这将导致一个段错误:
以下是修改后的代码:
def render(source_url):
"""Fully render HTML, JavaScript and all."""
import sys
from PyQt5.QtWidgets import QApplication
from PyQt5.QtCore import QUrl
from PyQt5.QtWebEngineWidgets import QWebEngineView
class Render(QWebEngineView):
def __init__(self, url):
self.html = None
self.app = QApplication(sys.argv)
QWebEngineView.__init__(self)
self.loadFinished.connect(self._loadFinished)
# self.setHtml(html)
self.load(QUrl(url))
self.app.exec_()
def _loadFinished(self, result):
# This is an async call, you need to wait for this
# to be called before closing the app
self.page().toHtml(self._callable)
def _callable(self, data):
self.html = data
# Data has been stored, it's safe to quit the app
self.app.quit()
return Render(source_url).html
# url = 'http://webscraping.com'
# url='http://www.amazon.com'
url = "https://www.ncbi.nlm.nih.gov/nuccore/CP002059.1"
print(render(url))
它抛出了这些错误:
$ python3 -tt fees-pkg-v2.py
Traceback (most recent call last):
File "fees-pkg-v2.py", line 30, in _callable
self.html = data
AttributeError: 'method' object has no attribute 'html'
None (hangs here until force-quit python launcher)
Segmentation fault: 11
$
我已经开始阅读python类,以便完全理解我在做什么(总是一件好事)。我认为我的环境中的某些东西可能是问题(OSX Yosemite,Python 3.4.3,Qt5.4.1,sip-4.16.6)。还有其他建议吗?
发布于 2017-07-24 22:14:23
问题出在环境上。我手动安装了Python 3.4.3、Qt5.4.1和sip-4.16.6,一定是搞砸了什么。安装Anaconda后,脚本开始工作。再次感谢。
https://stackoverflow.com/questions/45265143
复制相似问题