blocks|key|1286427|text|因为条件if+len(next_page)<1总是假的。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1286428|例如，我尝试了url+page=99999999999999999999999，它给出了第13页，这是最后一页。|1286429|您可以尝试的是检查“下一页”按钮是否已禁用。|1286430|📷|atomic|1286431|​|1286432|entityMap|0|LINK|mutability|MUTABLE|url|https://monentreprise.bj/page/annonces?Company_page=99999999999999999999999999999999999&ajax=annonces-list|1|IMAGE|IMMUTABLE|imageUrl|https://developer.qcloudimg.com/http-save/yehe-900000/cd5579bfa8356f2d55254c18c3e72a8f.png|imageAlt^0|4|J|0|B|S|0|0|0|0|1|1|0|0^^$0|@$1|2|3|4|5|6|7|12|8|@$9|13|A|14|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|15|8|@]|D|@$9|16|A|17|1|18]]|E|$]]|$1|H|3|I|5|6|7|19|8|@]|D|@]|E|$]]|$1|J|3|K|5|L|7|1A|8|@]|D|@$9|1B|A|1C|1|1D]]|E|$]]|$1|M|3|N|5|6|7|1E|8|@]|D|@]|E|$]]|$1|O|3|-4|5|6|7|1F|8|@]|D|@]|E|$]]]|P|$Q|$5|R|S|T|E|$U|V]]|W|$5|X|S|Y|E|$Z|10|11|-4]]]]

Because the condition <code>if len(next_page)&lt;1</code> is always False.
For instance I tried the url <a href="https://monentreprise.bj/page/annonces?Company_page=99999999999999999999999999999999999&amp;ajax=annonces-list" rel="nofollow noreferrer">monentreprise.bj/page/annonces?Company_page=99999999999999999999999</a> and it gives the page 13 which is the last page
What you could try maybe is checking if the &quot;next page&quot; button is disabled
<a href="https://i.stack.imgur.com/ccgFz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ccgFz.png" alt="enter image description here" /></a>

blocks|key|1272317|text|这里有几个问题：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1272318|//*[@id='yw3']/li[12]/a不是next分页按钮的正确定位器。|ordered-list-item|offset|length|style|CODE|1272319|这里到达的最后一个页面的更好的指示是验证这个基于css_locator的元素.pagination+.next是否包含disabled类。|1272320|在单击“下一页”按钮之前，必须将页面向下滚动。|1272321|单击分页按钮后，必须添加延迟。否则这是行不通的。
这段代码适用于我：|1272322|from+selenium+import+webdriver
from+selenium.webdriver.common.by+import+By
from+selenium.webdriver.support.ui+import+WebDriverWait
from+selenium.webdriver.support+import+expected_conditions+as+EC
import+time

driver+=+webdriver.Chrome()
my_url+=+"https://monentreprise.bj/page/annonces"
driver.get(my_url)
next_page_parent+=+'.pagination+.next'
next_page_parent_arrow+=+'.pagination+.next+a'
while+True:
++++driver.execute_script("window.scrollTo(0,+document.body.scrollHeight)")
++++time.sleep(0.5)
++++parent+=+driver.find_element(By.CSS_SELECTOR,next_page_parent)
++++class_name+=+parent.get_attribute("class")
++++if+"disabled"+in+class_name:
++++++++print("No+more+pages")
++++++++break
++++else:
++++++++WebDriverWait(driver,+10).until(EC.element_to_be_clickable((By.CSS_SELECTOR,+next_page_parent_arrow))).click()
++++++++time.sleep(1.5)
++++++++print('ok')|code-block|syntax|javascript|1272323|产出如下：|1272324|ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
No+more+pages|1272325|entityMap^0|0|0|N|P|4|0|12|H|1N|8|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Z|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|10|8|@$E|11|F|12|G|H]|$E|13|F|14|G|H]]|9|@]|A|$]]|$1|I|3|J|5|D|7|15|8|@$E|16|F|17|G|H]|$E|18|F|19|G|H]]|9|@]|A|$]]|$1|K|3|L|5|D|7|1A|8|@]|9|@]|A|$]]|$1|M|3|N|5|D|7|1B|8|@]|9|@]|A|$]]|$1|O|3|P|5|Q|7|1C|8|@]|9|@]|A|$R|S]]|$1|T|3|U|5|6|7|1D|8|@]|9|@]|A|$]]|$1|V|3|W|5|Q|7|1E|8|@]|9|@]|A|$R|S]]|$1|X|3|-4|5|6|7|1F|8|@]|9|@]|A|$]]]|Y|$]]

There are several issues here:
<ol>
<li><code>//*[@id='yw3']/li[12]/a</code> is not a correct locator for the <code>next</code> pagination button.</li>
<li>The better indication for the last page reached state here will be to validate if this css_locator based element <code>.pagination .next</code> contains <code>disabled</code> class.</li>
<li>You have to scroll the page down before clicking the next page button</li>
<li>You have to add a delay after clicking on the pagination button. Otherwise this will not work. 
This code worked for me:</li>
</ol>
<pre class="lang-py prettyprint-override"><code>from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

driver = webdriver.Chrome()
my_url = &quot;https://monentreprise.bj/page/annonces&quot;
driver.get(my_url)
next_page_parent = '.pagination .next'
next_page_parent_arrow = '.pagination .next a'
while True:
 driver.execute_script(&quot;window.scrollTo(0, document.body.scrollHeight)&quot;)
 time.sleep(0.5)
 parent = driver.find_element(By.CSS_SELECTOR,next_page_parent)
 class_name = parent.get_attribute(&quot;class&quot;)
 if &quot;disabled&quot; in class_name:
 print(&quot;No more pages&quot;)
 break
 else:
 WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, next_page_parent_arrow))).click()
 time.sleep(1.5)
 print('ok')
</code></pre>
The output is:
<pre><code>ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
No more pages
</code></pre>

The amount of data(number of pages) on the site keeps changing and I need to scrape all the pages looping through the pagination.
Website: <code>https://monentreprise.bj/page/annonces</code>
Code I tried:
<pre><code>xpath= &quot;//*[@id='yw3']/li[12]/a&quot; 
while True:
 next_page = driver.find_elements(By.XPATH,xpath)
 if len(next_page) &lt; 1:
 print(&quot;No more pages&quot;)
 break
 else:
 WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, xpath))).click()
 print('ok')
</code></pre>
<code>ok</code> is printed continuously

How to stop the selenium webdriver after reaching the last page while scraping the website?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

网站上的数据量(页数)一直在变化，我需要在分页过程中刮掉所有的页面。网址：https://monentreprise.bj/page/annonces我试过的代码：xpath= "//*[@id='yw3']/li[12]/a"        while True:    next_page = driver.find...

问在抓取网站时，如何在到达最后一页后停止selenium？
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在抓取网站时，如何在到达最后一页后停止selenium？EN