blocks|key|1688056|text|处理这种情况的最好方法可能是拆分代码，这样处理页面内容的逻辑就可以从获取页面的代码中分离出来。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1688057|然后将抓取器代码的一个实例传递到处理逻辑中，然后您可以轻松地将其替换为单元测试的模拟抓取器。|1688058|例如：|1688059|class+Processor(oject):
++++def+__init__(self,+fetcher):
++++++++self.m_fetcher+=+fetcher

++++def+doProcessing(self):
++++++++##+use+self.m_fetcher+to+get+page+contents

class+RealFetcher(object):
++++def+fetchPage(self,+url):
++++++++##+get+real+contents

class+FakeFetcher(object):
++++def+fetchPage(self,+url):
++++++++##+Return+whatever+fake+contents+are+required+for+this+test|code-block|syntax|javascript|1688060|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|N|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|O|8|@]|9|@]|A|$]]|$1|F|3|G|5|H|7|P|8|@]|9|@]|A|$I|J]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

Probably the best way to handle this is to split up the code, so that logic that processes the page contents is split from the code that fetches the page.

Then pass an instance of the fetcher code into the processing logic, then you can easily replace it with a mock fetcher for the unit test.

e.g.

<pre><code>class Processor(oject):
 def __init__(self, fetcher):
 self.m_fetcher = fetcher

 def doProcessing(self):
 ## use self.m_fetcher to get page contents

class RealFetcher(object):
 def fetchPage(self, url):
 ## get real contents

class FakeFetcher(object):
 def fetchPage(self, url):
 ## Return whatever fake contents are required for this test
</code></pre>

blocks|key|1473006|text|最简单的方法是更改您的函数，使其不一定使用urllib.urlopen。假设这是您的原始函数：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1473007|def+my_grabber(arg1,+arg2,+arg3):
++++#+..+do+some+stuff+..
++++url+=+make_url_somehow()
++++data+=+urllib.urlopen(url)
++++#+..+do+something+with+data+..
++++return+answer|code-block|syntax|javascript|1473008|添加一个参数，该参数是用于打开URL的函数。然后，您可以提供一个模拟函数来执行所需的任何操作：|1473009|def+my_grabber(arg1,+arg2,+arg3,+urlopen=urllib.urlopen):
++++#+..+do+some+stuff+..
++++url+=+make_url_somehow()
++++data+=+urlopen(url)
++++#+..+do+something+with+data+..
++++return+answer

def+test_my_grabber():
++++my_grabber(arg1,+arg2,+arg3,+urlopen=my_mock_open)|1473010|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

The simplest way is to change your function so that it doesn't necessarily use urllib.urlopen. Let's say this is your original function:

<pre><code>def my_grabber(arg1, arg2, arg3):
 # .. do some stuff ..
 url = make_url_somehow()
 data = urllib.urlopen(url)
 # .. do something with data ..
 return answer
</code></pre>

Add an argument which is the function to use to open the URL. Then you can provide a mock function to do whatever you need:

<pre><code>def my_grabber(arg1, arg2, arg3, urlopen=urllib.urlopen):
 # .. do some stuff ..
 url = make_url_somehow()
 data = urlopen(url)
 # .. do something with data ..
 return answer

def test_my_grabber():
 my_grabber(arg1, arg2, arg3, urlopen=my_mock_open)
</code></pre>

blocks|key|1473051|text|另一个简单的方法是让您的测试覆盖urllib的urlopen()函数。例如，如果您的模块具有|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1473052|import+urllib

def+some_function_that_uses_urllib():
++++...
++++urllib.urlopen()
++++...|code-block|syntax|javascript|1473053|你可以这样定义你的测试：|1473054|import+mymodule

def+dummy_urlopen(url):
++++...

mymodule.urllib.urlopen+=+dummy_urlopen|1473055|然后，当您的测试调用mymodule中的函数时，将调用dummy_urlopen()而不是真正的urlopen()。像Python这样的动态语言使得将方法和类存根出来进行测试变得非常容易。|1473056|请参阅我在http://softwarecorner.wordpress.com/上的博客文章，了解有关清除测试依赖项的更多信息。|1473057|entityMap|0|LINK|mutability|MUTABLE|url|http://softwarecorner.wordpress.com/^0|N|9|0|0|0|0|A|8|R|F|1C|9|0|5|10|0|0^^$0|@$1|2|3|4|5|6|7|10|8|@$9|11|A|12|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|13|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|14|8|@]|D|@]|E|$]]|$1|M|3|N|5|H|7|15|8|@]|D|@]|E|$I|J]]|$1|O|3|P|5|6|7|16|8|@$9|17|A|18|B|C]|$9|19|A|1A|B|C]|$9|1B|A|1C|B|C]]|D|@]|E|$]]|$1|Q|3|R|5|6|7|1D|8|@]|D|@$9|1E|A|1F|1|1G]]|E|$]]|$1|S|3|-4|5|6|7|1H|8|@]|D|@]|E|$]]]|T|$U|$5|V|W|X|E|$Y|Z]]]]

Another simple approach is to have your test override urllib's <code>urlopen()</code> function. For example, if your module has

<pre><code>import urllib

def some_function_that_uses_urllib():
 ...
 urllib.urlopen()
 ...
</code></pre>

You could define your test like this:

<pre><code>import mymodule

def dummy_urlopen(url):
 ...

mymodule.urllib.urlopen = dummy_urlopen
</code></pre>

Then, when your tests invoke functions in <code>mymodule</code>, <code>dummy_urlopen()</code> will be called instead of the real <code>urlopen()</code>. Dynamic languages like Python make it super easy to stub out methods and classes for testing.

See my blog posts at <a href="http://softwarecorner.wordpress.com/" rel="noreferrer">http://softwarecorner.wordpress.com/</a> for more information about stubbing out dependencies for tests.

blocks|key|1473147|text|你看了Mox一眼吗？它应该能做你需要的一切。下面是一个简单的交互式会话，说明了您需要的解决方案：|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1473148|>>>+import+urllib
>>>+#+check+that+it+works
>>>+urllib.urlopen('http://www.google.com/')
<addinfourl+at+3082723820L+...>
>>>+#+check+what+happens+when+it+doesn't
>>>+urllib.urlopen('http://hopefully.doesnotexist.com/')
#--+snip+--
IOError:+[Errno+socket+error]+(-2,+'Name+or+service+not+known')

>>>+#+OK,+let's+mock+it+up
>>>+import+mox
>>>+m+=+mox.Mox()
>>>+m.StubOutWithMock(urllib,+'urlopen')
>>>+#+We+can+be+verbose+if+we+want+to+:)
>>>+urllib.urlopen(mox.IgnoreArg()).AndRaise(
...+++IOError('socket+error',+(-2,+'Name+or+service+not+known')))

>>>+#+Let's+check+if+it+works
>>>+m.ReplayAll()
>>>+urllib.urlopen('http://www.google.com/')
Traceback+(most+recent+call+last):
++File+"<stdin>",+line+1,+in+<module>
++File+"/usr/lib/python2.5/site-packages/mox.py",+line+568,+in+__call__
++++raise+expected_method._exception
IOError:+[Errno+socket+error]+(-2,+'Name+or+service+not+known')

>>>+#+yay!+now+unset+everything
>>>+m.UnsetStubs()
>>>+m.VerifyAll()
>>>+#+and+check+that+it+still+works
>>>+urllib.urlopen('http://www.google.com/')
<addinfourl+at+3076773548L+...>|code-block|syntax|javascript|1473149|entityMap|0|LINK|mutability|MUTABLE|url|https://pypi.python.org/pypi/mox^0|3|3|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$G|H]]|$1|I|3|-4|5|6|7|V|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

Did you give <a href="https://pypi.python.org/pypi/mox" rel="nofollow noreferrer">Mox</a> a look? It should do everything you need. Here is a simple interactive session illustrating the solution you need:

<pre><code>&gt;&gt;&gt; import urllib
&gt;&gt;&gt; # check that it works
&gt;&gt;&gt; urllib.urlopen('http://www.google.com/')
&lt;addinfourl at 3082723820L ...&gt;
&gt;&gt;&gt; # check what happens when it doesn't
&gt;&gt;&gt; urllib.urlopen('http://hopefully.doesnotexist.com/')
#-- snip --
IOError: [Errno socket error] (-2, 'Name or service not known')

&gt;&gt;&gt; # OK, let's mock it up
&gt;&gt;&gt; import mox
&gt;&gt;&gt; m = mox.Mox()
&gt;&gt;&gt; m.StubOutWithMock(urllib, 'urlopen')
&gt;&gt;&gt; # We can be verbose if we want to :)
&gt;&gt;&gt; urllib.urlopen(mox.IgnoreArg()).AndRaise(
... IOError('socket error', (-2, 'Name or service not known')))

&gt;&gt;&gt; # Let's check if it works
&gt;&gt;&gt; m.ReplayAll()
&gt;&gt;&gt; urllib.urlopen('http://www.google.com/')
Traceback (most recent call last):
 File "&lt;stdin&gt;", line 1, in &lt;module&gt;
 File "/usr/lib/python2.5/site-packages/mox.py", line 568, in __call__
 raise expected_method._exception
IOError: [Errno socket error] (-2, 'Name or service not known')

&gt;&gt;&gt; # yay! now unset everything
&gt;&gt;&gt; m.UnsetStubs()
&gt;&gt;&gt; m.VerifyAll()
&gt;&gt;&gt; # and check that it still works
&gt;&gt;&gt; urllib.urlopen('http://www.google.com/')
&lt;addinfourl at 3076773548L ...&gt;
</code></pre>

blocks|key|1688244|text|我正在使用Mock's补丁装饰器：|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1688245|from+mock+import+patch

[...]

@patch('urllib.urlopen')
def+test_foo(self,+urlopen_mock):
++++urlopen_mock.return_value+=+MyUrlOpenMock()|code-block|syntax|javascript|1688246|entityMap|0|LINK|mutability|MUTABLE|url|http://www.voidspace.org.uk/python/mock/^0|5|6|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$G|H]]|$1|I|3|-4|5|6|7|V|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

I am using <a href="http://www.voidspace.org.uk/python/mock/" rel="noreferrer">Mock's</a> patch decorator:

<pre><code>from mock import patch

[...]

@patch('urllib.urlopen')
def test_foo(self, urlopen_mock):
 urlopen_mock.return_value = MyUrlOpenMock()
</code></pre>

blocks|key|1473236|text|以防你甚至不想加载模块：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1473237|import+sys,types
class+MockCallable():
++"""+Mocks+a+function,+can+be+enquired+on+how+many+calls+it+received+"""
++def+__init__(self,+result):
++++self.result++=+result
++++self._calls++=+[]

++def+__call__(self,+*arguments):
++++"""Mock+callable"""
++++self._calls.append(arguments)
++++return+self.result

++def+called(self):
++++"""docstring+for+called"""
++++return+self._calls

class+StubModule(types.ModuleType,+object):
++"""+Uses+a+stub+instead+of+loading+libraries+"""

++def+__init__(self,+moduleName):
++++self.__name__+=+moduleName
++++sys.modules[moduleName]+=+self

++def+__repr__(self):
++++name++=+self.__name__
++++mocks+=+',+'.join(set(dir(self))+-+set(['__name__']))
++++return+"<StubModule:+%25(name)s;+mocks:+%25(mocks)s>"+%25+locals()

class+StubObject(object):
++pass|code-block|syntax|javascript|1473238|然后：|1473239|>>>+urllib+=+StubModule("urllib")
>>>+import+urllib+#+won't+actually+load+urllib

>>>+urls.urlopen+=+MockCallable(StubObject())

>>>+example+=+urllib.urlopen('http://example.com')
>>>+example.read+=+MockCallable('foo')

>>>+print(example.read())
'foo'|1473240|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

In case you don't want to even load the module:

<pre><code>import sys,types
class MockCallable():
 """ Mocks a function, can be enquired on how many calls it received """
 def __init__(self, result):
 self.result = result
 self._calls = []

 def __call__(self, *arguments):
 """Mock callable"""
 self._calls.append(arguments)
 return self.result

 def called(self):
 """docstring for called"""
 return self._calls

class StubModule(types.ModuleType, object):
 """ Uses a stub instead of loading libraries """

 def __init__(self, moduleName):
 self.__name__ = moduleName
 sys.modules[moduleName] = self

 def __repr__(self):
 name = self.__name__
 mocks = ', '.join(set(dir(self)) - set(['__name__']))
 return "&lt;StubModule: %(name)s; mocks: %(mocks)s&gt;" % locals()

class StubObject(object):
 pass
</code></pre>

And then:

<pre><code>&gt;&gt;&gt; urllib = StubModule("urllib")
&gt;&gt;&gt; import urllib # won't actually load urllib

&gt;&gt;&gt; urls.urlopen = MockCallable(StubObject())

&gt;&gt;&gt; example = urllib.urlopen('http://example.com')
&gt;&gt;&gt; example.read = MockCallable('foo')

&gt;&gt;&gt; print(example.read())
'foo'
</code></pre>

blocks|key|1473308|text|HTTPretty的工作方式与FakeWeb完全相同。HTTPretty在套接字层工作，因此它应该可以拦截任何python+http客户端库。它与urllib2、httplib2和requests进行了战斗测试|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1473309|import+urllib2
from+httpretty+import+HTTPretty,+httprettified


@httprettified
def+test_one():
++++HTTPretty.register_uri(HTTPretty.GET,+"http://yipit.com/",
+++++++++++++++++++++++++++body="Find+the+best+daily+deals")

++++fd+=+urllib2.urlopen('http://yipit.com')
++++got+=+fd.read()
++++fd.close()

++++assert+got+==+"Find+the+best+daily+deals"|code-block|syntax|javascript|1473310|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/gabrielfalcao/HTTPretty^0|0|9|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$G|H]]|$1|I|3|-4|5|6|7|V|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

<a href="https://github.com/gabrielfalcao/HTTPretty" rel="nofollow noreferrer">HTTPretty</a> works in the exact same way that FakeWeb does. HTTPretty works in the socket layer, so it should work intercepting any python http client libraries. It's battle tested against urllib2, httplib2 and requests

<pre><code>import urllib2
from httpretty import HTTPretty, httprettified


@httprettified
def test_one():
 HTTPretty.register_uri(HTTPretty.GET, "http://yipit.com/",
 body="Find the best daily deals")

 fd = urllib2.urlopen('http://yipit.com')
 got = fd.read()
 fd.close()

 assert got == "Find the best daily deals"
</code></pre>

blocks|key|1473346|text|除了Clint+Miller的回答之外，为了做到这一点，我不得不创建一个伪类来实现如下的read方法：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1473347|class+FakeURL:
++++def+read(foo):
++++++++return+'{"some":"json_text"}'|code-block|syntax|javascript|1473348|然后清除urllib2.open：|1473349|#+Stub+out+urllib2.open.
def+dummy_urlopen(foo,+bar,+baz):
++return+FakeURL()
urllib2.urlopen+=+dummy_urlopen|1473350|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

Adding onto Clint Miller's answer, to do this I had to create a fake class that implements a read method like this:

<pre class="lang-py prettyprint-override"><code>class FakeURL:
 def read(foo):
 return '{"some":"json_text"}'
</code></pre>

Then to stub out urllib2.open:

<pre class="lang-py prettyprint-override"><code># Stub out urllib2.open.
def dummy_urlopen(foo, bar, baz):
 return FakeURL()
urllib2.urlopen = dummy_urlopen
</code></pre>

I need to test a function that needs to query a page on an external server using urllib.urlopen (it also uses urllib.urlencode). The server could be down, the page could change; I can't rely on it for a test.

What is the best way to control what urllib.urlopen returns?

How can one mock/stub python module like urllib

单元测试

服务器

Python

我需要测试一个函数，该函数需要使用urllib.urlopen (它也使用urllib.urlencode)查询外部服务器上的页面。服务器可能会宕机，页面可能会改变；我不能依赖它进行测试。控制urllib.urlopen返回内容的最佳方法是什么？

问像urllib这样的模拟/存根python模块
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问像urllib这样的模拟/存根python模块EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问像urllib这样的模拟/存根python模块
EN