尽管表单在响应中填写正确，但FormRequest不返回任何结果
尽管表单在响应中填写正确，但FormRequest不返回任何结果
提问于 2019-05-11 22:47:22
我试图从哈利法克斯银行分行的官方分行查找程序网站(https://www.halifax.co.uk/branchfinder/search.asp)上抓取营业时间，但我一定是在FormRequest.from_response()调用中做错了什么，因为对FormRequest的响应似乎与最初的响应没有变化。
传入'postcode‘值'EH'，应该只有一个结果(爱丁堡，王子街131号，EH2 4AH)，就像你在网站上做同样的事情时看到的那样，但当我使用FormRequest做同样的事情时，我什么也得不到
这是该网站来源的相关部分：
<form action="" id="branch-finder-search-form" method="post">
<div style="display:none;" class="notice error" aria-live="assertive"></div>
<div class="field split-2 clearfix">
<div class="split radio">
<div class="field-radio">
<input value="branch" name="searchType" checked id="r1" type="radio"><label for="r1"><span></span>Branch</label>
</div>
<div class="field-radio">
<input value="cash" name="searchType" id="r2" type="radio"><label for="r2"><span></span>Cash Machine</label>
</div>
</div>
</div>
<div class="field split-3 clearfix">
<div class="split">
<label for="street">Street</label><input name="street" id="street" type="text" value="">
</div>
<div class="split">
<label for="town">Town</label><input name="town" id="town" type="text" value="">
</div>
<div class="split last">
<label for="postcode">Post Code</label><input name="postcode" id="postcode" type="text" value="">
</div>
</div>
<div class="field clearfix">
<div class="split btn-submit">
<input id="search" name="search" alt="Search" type="submit" value="Search" class="button button-primary" title="search"><span class="a_hide">ext search</span></input>
</div>
<noscript>
<input value="yes" name="javascriptoff" id="javascriptoff" type="hidden">
<div class="split btn-submit-nonjs">
<input name="nonjsSubmit" type="submit" alt="Search" value="Search" class="button button-primary" title="search"><span class="a_hide">ext search</span></input>
</div>
<div></div>
</noscript>
<div style="display:none" id="no-result">
<p>No Branch Found as per your search criteria</p>
</div>
<div id="branch-finder-results-container">
<hr>
</div>
</div>
</form>这是我到目前为止的代码：
from scrapy.crawler import CrawlerProcess
from scrapy.utils.response import open_in_browser
from scrapy.http import FormRequest
import scrapy
class HalifaxSpider(scrapy.Spider):
name = "halifax"
start_urls = [
"https://www.halifax.co.uk/branchfinder/search.asp"
]
def parse(self, response):
print(response.text)
yield FormRequest.from_response(response, formname='branch-finder-search-form', formdata={"postcode": "EH"}, callback=self.open_in_browser)
def open_in_browser(self, response):
print(response.text)
open_in_browser(response)
crawler = CrawlerProcess()
crawler.crawl(HalifaxSpider)
crawler.start()我还尝试传入参数clickdata={"name"：" search "}，以防搜索按钮没有被单击，但我得到了相同的结果。
一般来说，我对网络抓取是个新手，所以我真的被困住了，不知道从哪里开始。我已经阅读了一些文档，但关于填写各种不同类型的表单的信息或教程并不多，所以我不知所措。
有人能帮帮忙吗？
回答 1
Stack Overflow用户
回答已采纳
发布于 2019-05-12 00:53:27
查看以下获取所需数据的方法。Headers在这里扮演着重要的角色，所以一定要包含它(我已经添加了它)。
import scrapy
import urllib
from scrapy.crawler import CrawlerProcess
class HalifaxSpider(scrapy.Spider):
name = "halifax"
url = "https://www.halifax.co.uk/asp_includes/branch-finder/branch-finder.asp?"
def start_requests(self):
formdata = {
'street': '',
'town': '',
'postcode': 'EH',
'searchType': 'branch'
}
req_url = f'{self.url}{urllib.parse.urlencode(formdata)}'
yield scrapy.Request(req_url,callback=self.parse)
def parse(self,response):
print(response.css("h2#resultTitle1 ::text").getall())
if __name__ == "__main__":
crawler = CrawlerProcess({'USER_AGENT':'Mozilla/5.0'})
crawler.crawl(HalifaxSpider)
crawler.start()页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/56091385
复制相关文章
相似问题
腾讯云开发者
