blocks|key|2633686|text|您正在处理这个问题，因为您使用了错误的链接来获取文章链接。所以我没有做太多的修改，并想出了这个代码(请注意，我删除了bs4模块，因为它不再需要了)|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2633687|import+requests

search+=+"edith+cowan"

headers+=+{
++++'User-Agent':+'Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/63.0.3239.132+Safari/537.36'}

json+=+{"scholarly_search":{"from":0,"size":"10","_source":{"excludes":["referenced_by_patent_hash","referenced_by_patent","reference"]},"query":{"bool":{"must":[{"query_string":{"query":f"\"{search}\"","fields":["title","abstract","default"],"default_operator":"and"}}],"must_not":[{"terms":{"publication_type":["unknown"]}}],"filter":[]}},"highlight":{"pre_tags":["<span+class=\"highlight\">"],"post_tags":[""],"fields":{"title":{}},"number_of_fragments":0},"sort":[{"referenced_by_patent_count":{"order":"desc"}}]},"view":"scholar"}

req+=+requests.post("https://www.lens.org/lens/api/multi/search?request_cache=true",+headers+=+headers,+json+=+json).json()

links+=+[]
for+x+in+req["query_result"]["hits"]["hits"]:
++++links.append("https://www.lens.org/lens/scholar/article/{}/main".format(x["_source"]["record_lens_id"]))|code-block|syntax|javascript|2633688|search变量等于您正在搜索的术语(在本例中为"edith+cowan")。链接存储在links变量中。|offset|length|style|CODE|2633689|2633690|编辑:+How+I+it|BOLD|2633691|因此，主要问题可能是我从哪里获得链接，以及我如何知道要在json变量中包含哪些内容。为此，我使用了一个简单的HTML拦截器(在我的例子中是)。|2633692|这个工具告诉我，当您访问这个URL+(您在问题中用来向其发送请求的那个)时，您的浏览器会向https://www.lens.org/lens/api/multi/search?request_cache=true发送一个post请求，然后它会检索该current+page的所有信息。json变量burp套件还向您显示发送了哪些包，因此我将它们复制粘贴到json变量中。|2633693|为了更好的可视化，这是它在burp套件中的外观：|2633694|​|2633695|📷|atomic|2633696|2633697|2633698|编辑:扫描所有页面|2633699|为了扫描所有页面，您可以使用以下脚本：|2633700|import+requests

search+=+"edith+cowan"+#Change+this+to+the+term+you+are+searching+for
r_to_show+=+100+#This+is+the+number+of+articles+per+page+(I+strongly+recommend+leaving+it+at+100)

headers+=+{
++++'User-Agent':+'Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/63.0.3239.132+Safari/537.36'}

json+=+{"scholarly_search":{"from":0,"size":f"{r_to_show}","_source":{"excludes":["referenced_by_patent_hash","referenced_by_patent","reference"]},"query":{"bool":{"must":[{"query_string":{"query":f"\"{search}\"","fields":["title","abstract","default"],"default_operator":"and"}}],"must_not":[{"terms":{"publication_type":["unknown"]}}],"filter":[]}},"highlight":{"pre_tags":["<span+class=\"highlight\">"],"post_tags":[""],"fields":{"title":{}},"number_of_fragments":0},"sort":[{"referenced_by_patent_count":{"order":"desc"}}]},"view":"scholar"}

req+=+requests.post("https://www.lens.org/lens/api/multi/search?request_cache=true",+headers+=+headers,+json+=+json).json()

links+=+[]+#links+are+stored+here
count+=+0

#link_before+and+link_after+helps+determine+when+to+stop+going+to+the+next+page+
link_before+=+0
link_after+=+0

while+True:
++++json["scholarly_search"]["from"]+%2B=+r_to_show
++++if+count+>+0:
++++++++req+=+requests.post("https://www.lens.org/lens/api/multi/search?request_cache=true",+headers+=+headers,+json+=+json).json()+
++++for+x+in+req["query_result"]["hits"]["hits"]:
++++++++links.append("https://www.lens.org/lens/scholar/article/{}/main".format(x["_source"]["record_lens_id"]))
++++count+%2B=+1
++++link_after+=+len(links)
++++if+link_after+==+link_before:
++++++++break
++++link_before+=+len(links)
++++print(f"page+{count}+done,+links+recorder+{len(links)}")+|2633701|我在代码中添加了一些注释，使其更易于理解。|2633702|entityMap|0|LINK|mutability|MUTABLE|url|https://www.lens.org/lens/scholar/search/results?q=%2522edith%2520cowan|1|https://www.lens.org/lens/api/multi/search?request_cache=true|2|https://www.lens.org/lens/scholar/search/results?q=%2522edith%2520cowan|3|IMAGE|IMMUTABLE|imageUrl|https://ask.qcloudimg.com/http-save/yehe-900000/2253890e80a990c2dcfb48858d47b0f8.png|imageAlt^0|0|0|0|6|O|D|18|5|0|0|0|C|0|S|4|0|4X|4|E|3|0|19|1P|1|3G|C|2|0|0|0|0|1|3|0|0|0|0|9|0|0|0|0^^$0|@$1|2|3|4|5|6|7|1T|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|1U|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|1V|8|@$I|1W|J|1X|K|L]|$I|1Y|J|1Z|K|L]|$I|20|J|21|K|L]]|9|@]|A|$]]|$1|M|3|-4|5|6|7|22|8|@]|9|@]|A|$]]|$1|N|3|O|5|6|7|23|8|@$I|24|J|25|K|P]]|9|@]|A|$]]|$1|Q|3|R|5|6|7|26|8|@$I|27|J|28|K|L]]|9|@]|A|$]]|$1|S|3|T|5|6|7|29|8|@$I|2A|J|2B|K|L]]|9|@$I|2C|J|2D|1|2E]|$I|2F|J|2G|1|2H]|$I|2I|J|2J|1|2K]]|A|$]]|$1|U|3|V|5|6|7|2L|8|@]|9|@]|A|$]]|$1|W|3|X|5|6|7|2M|8|@]|9|@]|A|$]]|$1|Y|3|Z|5|10|7|2N|8|@]|9|@$I|2O|J|2P|1|2Q]]|A|$]]|$1|11|3|X|5|6|7|2R|8|@]|9|@]|A|$]]|$1|12|3|-4|5|6|7|2S|8|@]|9|@]|A|$]]|$1|13|3|14|5|6|7|2T|8|@$I|2U|J|2V|K|P]]|9|@]|A|$]]|$1|15|3|16|5|6|7|2W|8|@]|9|@]|A|$]]|$1|17|3|18|5|D|7|2X|8|@]|9|@]|A|$E|F]]|$1|19|3|1A|5|6|7|2Y|8|@]|9|@]|A|$]]|$1|1B|3|-4|5|6|7|2Z|8|@]|9|@]|A|$]]]|1C|$1D|$5|1E|1F|1G|A|$1H|1I]]|1J|$5|1E|1F|1G|A|$1H|1K]]|1L|$5|1E|1F|1G|A|$1H|1M]]|1N|$5|1O|1F|1P|A|$1Q|1R|1S|-4]]]]

You are dealing with this problem due to the fact that you were using the wrong link to get the article links. So I didn't quite a few changes and came up with this code (Do note that I removed the bs4 module as it is not needed anymore):

<pre><code>import requests

search = "edith cowan"

headers = {
 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'}

json = {"scholarly_search":{"from":0,"size":"10","_source":{"excludes":["referenced_by_patent_hash","referenced_by_patent","reference"]},"query":{"bool":{"must":[{"query_string":{"query":f"\"{search}\"","fields":["title","abstract","default"],"default_operator":"and"}}],"must_not":[{"terms":{"publication_type":["unknown"]}}],"filter":[]}},"highlight":{"pre_tags":["&lt;span class=\"highlight\"&gt;"],"post_tags":["&lt;/span&gt;"],"fields":{"title":{}},"number_of_fragments":0},"sort":[{"referenced_by_patent_count":{"order":"desc"}}]},"view":"scholar"}

req = requests.post("https://www.lens.org/lens/api/multi/search?request_cache=true", headers = headers, json = json).json()

links = []
for x in req["query_result"]["hits"]["hits"]:
 links.append("https://www.lens.org/lens/scholar/article/{}/main".format(x["_source"]["record_lens_id"]))
</code></pre>

The <code>search</code> variable is equal to the term that you are searching for (in your case <code>"edith cowan"</code>). The links are stored in the <code>links</code> variable.

<hr>

Edit: How I did it

So the main question might be where did I get the link from and how did I know what to include in the <code>json</code> variable. For this, I used a simple HTML intercepter (in my case <a href="https://portswigger.net/burp" rel="nofollow noreferrer">burp suite community edition</a>). 

This tool showed me that when you visit this <a href="https://www.lens.org/lens/scholar/search/results?q=%22edith%20cowan" rel="nofollow noreferrer">URL</a> (the one that you used in your question to send the req to) your browser sends a post request to <a href="https://www.lens.org/lens/api/multi/search?request_cache=true" rel="nofollow noreferrer">https://www.lens.org/lens/api/multi/search?request_cache=true</a> which then retrieves all the info of the <a href="https://www.lens.org/lens/scholar/search/results?q=%22edith%20cowan" rel="nofollow noreferrer">current page</a>. What concerns the json variable burp suite also shows you what packets are sent, so I copy-pasted them into the <code>json</code> variable. 

for better visualisation, this is how it looked inside of burp suite:

<a href="https://i.stack.imgur.com/4hWJ6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4hWJ6.png" alt="Burp Suite screenshot"></a>

<hr>

Edit: Scanning all the pages

In order to scan all the pages, you can use the following script:

<pre><code>import requests

search = "edith cowan" #Change this to the term you are searching for
r_to_show = 100 #This is the number of articles per page (I strongly recommend leaving it at 100)

headers = {
 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'}

json = {"scholarly_search":{"from":0,"size":f"{r_to_show}","_source":{"excludes":["referenced_by_patent_hash","referenced_by_patent","reference"]},"query":{"bool":{"must":[{"query_string":{"query":f"\"{search}\"","fields":["title","abstract","default"],"default_operator":"and"}}],"must_not":[{"terms":{"publication_type":["unknown"]}}],"filter":[]}},"highlight":{"pre_tags":["&lt;span class=\"highlight\"&gt;"],"post_tags":["&lt;/span&gt;"],"fields":{"title":{}},"number_of_fragments":0},"sort":[{"referenced_by_patent_count":{"order":"desc"}}]},"view":"scholar"}

req = requests.post("https://www.lens.org/lens/api/multi/search?request_cache=true", headers = headers, json = json).json()

links = [] #links are stored here
count = 0

#link_before and link_after helps determine when to stop going to the next page 
link_before = 0
link_after = 0

while True:
 json["scholarly_search"]["from"] += r_to_show
 if count &gt; 0:
 req = requests.post("https://www.lens.org/lens/api/multi/search?request_cache=true", headers = headers, json = json).json() 
 for x in req["query_result"]["hits"]["hits"]:
 links.append("https://www.lens.org/lens/scholar/article/{}/main".format(x["_source"]["record_lens_id"]))
 count += 1
 link_after = len(links)
 if link_after == link_before:
 break
 link_before = len(links)
 print(f"page {count} done, links recorder {len(links)}") 
</code></pre>

I added a few comments to the code to make it more understandable.

I am trying to scrape <a href="https://www.lens.org/lens/scholar/search/results?q=%22edith%20cowan%22" rel="nofollow noreferrer">this page</a> and to get the URL of the title of each article which is an 'h3' 'a' element e.g. the first result is a link with text "Functional annotation of a full-length mouse cDNA collection" that links to <a href="https://www.lens.org/lens/scholar/article/021-211-986-094-957/main" rel="nofollow noreferrer">this page</a>.

All that my search returns is '[]'

My code is as follows:

<pre><code>import requests
from bs4 import BeautifulSoup
req = requests.get('https://www.lens.org/lens/scholar/search/results?q="edith%20cowan"')
soup = BeautifulSoup(req.content, "html5lib")
article_links = soup.select('h3 a')
print(article_links)
</code></pre>

Where am I going wrong?

Trouble finding content from python beautiful scraping

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我试图抓取并获取每篇文章标题的URL，它是一个'h3‘'a’元素，例如，第一个结果是一个链接，带有文本"Functional annotation of a full-length with collection“，它链接到。我搜索返回的所有内容都是'[]‘我的代码如下：import requestsfrom bs4 ...

难以从python漂亮的抓取中找到内容-腾讯云开发者社区-腾讯云

问难以从python漂亮的抓取中找到内容
EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问难以从python漂亮的抓取中找到内容EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问难以从python漂亮的抓取中找到内容
EN