如何用python抓取aspx页面？
如何用python抓取aspx页面？
提问于 2018-07-18 04:49:47
我正在尝试抓取这个网站的地址-- https://aca.accela.com/ALAMEDA/Cap/CapHome.aspx?module=Building&TabName=Building
下面是HTML的一部分：
<tr class="ACA_TabRow_Odd ACA_TabRow_Odd_FontSize">
<td class="ACA_AlignLeftOrRightTop"><input id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_CB_0" type="checkbox" name="ctl00$PlaceHolderMain$dgvPermitList$gdvPermitList$ctl02$CB_0" onclick="Check(this,'ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_hfSaveSelectedItems');" class="aca_gridview_checkbox" title="To select one record in current page."></td><td class="ACA_AlignLeftOrRightTop" style="width:60px;">
<div style="margin-right: 5px;">
<span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblUpdatedTime" css="ACA_SmLabel ACA_SmLabel_FontSize">07/10/2018</span>
</div>
</td><td class="ACA_AlignLeftOrRightTop" style="width:110px;">
<div>
<strong>
</strong>
<a id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_hlPermitNumber" href="/ALAMEDA/Cap/CapDetail.aspx?Module=Building&TabName=Building&capID1=18SPV&capID2=00000&capID3=00039&agencyCode=ALAMEDA&IsToShowInspection=">
<strong>
<span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblPermitNumber1">SPV18-0037</span></strong>
</a>
<input type="hidden" id="RecordId" value="18SPV-00000-00039">
</div>
</td><td class="ACA_AlignLeftOrRightTop" style="width:130px;">
<div class="ACA_CapListStyle">
<span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblType">Solar Photovoltaic System Residential</span>
</div>
</td><td class="ACA_AlignLeftOrRightTop" style="width:130px;">
<div class="ACA_CapListStyle">
<span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblDescription">INSTALL A 5.12 KW SOLAR SYSTEM, ROOFTOP, FLUSH MOUNT, 16 PANELS (BLDG)</span>
</div>
</td><td class="ACA_AlignLeftOrRightTop" style="width:130px;">
<div class="ACA_CapListStyle">
<span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblAddress">1623 CLINTON AVE, ALAMEDA CA 94501</span>
</div>
</td><td class="ACA_AlignLeftOrRightTop" style="width:100px;">
<div style="white-space: nowrap;" class="ACA_CapListStyle">
<div id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_panelStatus">
<span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblStatus">Plan Review</span>
</div>
<div id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_panelbtnRenewalDetail">
</div>
</div>
</td><td class="ACA_AlignLeftOrRightTop" style="width:100px;">
<div style="white-space: nowrap;" class="ACA_CapListStyle">
<div id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_Panel2">
</div>
<div id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_Panel3">
</div>
<div id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_Panel4">
</div>
<div id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_Panel5">
</div>
</div>
</td><td class="ACA_AlignLeftOrRightTop" style="width:110px;">
<div class="ACA_CapListStyle">
<span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblShortNote">INSTALL A 5.12 KW SOLAR SYSTEM, ROOFTOP, FLUSH MOU</span>
</div>
</td><td class="ACA_Hide">
<div class="ACA_CapListStyle">
<span id="ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl02_lblPermitAddress">1623 CLINTON AVE, ALAMEDA CA 94501</span>
</div>
</td>
</tr>我正在尝试从td列中获取地址。这是我的python代码-
import bs4
from bs4 import BeautifulSoup
try:
from urllib.request import urlopen
except ImportError:
from urllib2 import urlopen
import os
from lxml import html
import requests
import csv
import sys
for page_no in range(2, 50):
curr_page = str(page_no)
if page_no < 9 :
data = {
'ctl00$ScriptManager1': 'ctl00$PlaceHolderMain$dgvPermitList$updatePanel|ctl00$PlaceHolderMain$dgvPermitList$gdvPermitList$ctl13$ctl0'+curr_page,
'ctl00$PlaceHolderMain$generalSearchForm$ddlGSPermitType': 'Building/Solar Photovoltaic System/Residential/NA',
'ctl00$PlaceHolderMain$generalSearchForm$txtGSStartDate': '01/01/2008',
'ctl00$PlaceHolderMain$generalSearchForm$txtGSEndDate': '07/17/2018'
}
else :
data = {
'ctl00$ScriptManager1': 'ctl00$PlaceHolderMain$dgvPermitList$updatePanel|ctl00$PlaceHolderMain$dgvPermitList$gdvPermitList$ctl13$ctl1'+curr_page,
'ctl00$PlaceHolderMain$generalSearchForm$ddlGSPermitType': 'Building/Solar Photovoltaic System/Residential/NA',
'ctl00$PlaceHolderMain$generalSearchForm$txtGSStartDate': '01/01/2008',
'ctl00$PlaceHolderMain$generalSearchForm$txtGSEndDate': '07/17/2018'
}
page = requests.get('https://aca.accela.com/ALAMEDA/Cap/CapHome.aspx?module=Building&TabName=Building', data = data)
soup = BeautifulSoup(page.text, 'html.parser')
print('Page ', page_no)
#print(soup)
address = soup.find_all('tbody')
print(address)当我尝试打印address变量时，它只返回一个空字符串。当我尝试打印soup变量时，它返回网站的整个代码，而不是我们单击search时找到的数据。有没有人能告诉我哪里出了问题？
回答 1
Stack Overflow用户
发布于 2018-07-18 05:01:55
find_all的方法签名为find_all(name,attrs,recursive,string,limit,**kwargs)
您将name作为span传入，但随后尝试使用字符串作为关键字参数id，这是不正确的。你可能想要的是
id = 'ctl00_PlaceHolderMain_dgvPermitList_gdvPermitList_ctl09_lblAddress'
作为你的第二个论点。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/51389981
复制相关文章
相似问题