blocks|key|999617|text|安装Firebug，研究站点的javascript发出的请求，并在您的程序中模拟它们。有可能只需要发出一个请求，结果就会以JSON这样的形式出现|type|unstyled|depth|inlineStyleRanges|entityRanges|data|999618|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

Install Firebug, study the requests that are made by the site's javascript and mimic them in your program. Chances are there is a single request that needs to be made and the resulds would come in some nice form like JSON

blocks|key|999663|text|除非搜索提供商给你一个可以使用的API+(无论是通过反向通道协议还是公开可用的协议)，否则你所做的任何事情都不可能持续很长时间。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|999664|你可能会不遗余力地欺骗网站，让他们相信你是一个普通的网站用户。然后，他们会对他们的网站的工作方式做一些小的改变(因为他们不知道有人在以你的方式使用它)，突然你的黑客攻击就不起作用了。过一段时间，他们可能会注意到你正在以这种方式使用它们，并检测到你的使用情况并完全阻止它。|999665|基本上，除非他们给你一个API，否则你基本上是在偷窃，并且应该期待得到所有应得的礼貌……没有。|999666|以免你认为我在评判你，我会让你知道我是凭经验说的;)|999667|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|J|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|K|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|L|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|M|8|@]|9|@]|A|$]]|$1|H|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|I|$]]

Unless the search provider gives you an API to work with (either via backchannel agreement or a publicly available one), then nothing you do will be likely to work for very long. 

You may go to great pains to fool the site into believing you are an ordinary website user. Then, they will make some minor change to how their site works (because they have no idea someone is using it in the fashion you are) and all of a sudden your hack won't work. Sometime later, they may notice that you are using them in this fashion, and detect your usage and flat out block it.

Basically, unless they give you an API, you are essentially stealing, and should expect to receive all the courtesy that deserves... none.

Lest you think I am judging you, I'll let you know I speak from experience ;)

blocks|key|999710|text|Javascript处理http请求几乎就像浏览器一样，一旦你弄清楚它们是什么，你可以尝试在perl/python/等中重新创建它们。使用Firefox%2BFirebug，你可以在“Net”面板中看到请求。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|999711|你可能需要考虑的是用户代理字符串，cookie，有时返回的数据是由Javascript运行/解释的，等等。也许你选择的语言有一个不错的httpbrowser类可以使用？|999712|999713|只要看一看，搜索IBM，从Firebug中获取post数据，将换行符替换为'&‘，并将其放在请求url之后：|999714|[http://bcode.bloomberg.com/sym/dwr/call/plaincall/searchMgr.search.dwr?callCount=1&windowName=&c0-scriptName=searchMgr&c0-methodName=search&c0-id=0&c0-e1=string:ibm&c0-e2=string:&c0-e3=number:100&c0-e4=number:0&c0-e5=boolean:false&c0-param0=Object_SearchCriteria:{search:reference:c0-e1,%2520filter:reference:c0-e2,%2520limit:reference:c0-e3&,%2520start:reference:c0-e4,%2520allSources:reference:c0-e5}&batchId=4&page=%252Fsym%252F&httpSessionId=&scriptSessionId=FBC68693A4E1BC08D6E0DDFBDF6D0860]|code-block|syntax|javascript|999715|但它会返回|999716|throw+'allowScriptTagRemoting+is+false.';
//#DWR-REPLY
if+(window.dwr)+dwr.engine.remote.handleBatchException({+name:'java.lang.SecurityException',+message:'GET+Disallowed'+});
else+if+(window.parent.dwr)+window.parent.dwr.engine.remote.handleBatchException({+name:'java.lang.SecurityException',+message:'GET+Disallowed'+});|999717|而且没有数据..所以看起来你必须编写一个post请求的脚本。看看他们的限制和指导方针，也许你应该联系一下，问问有没有公共API？|999718|entityMap^0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|T|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|U|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|V|8|@]|9|@]|A|$]]|$1|E|3|F|5|6|7|W|8|@]|9|@]|A|$]]|$1|G|3|H|5|I|7|X|8|@]|9|@]|A|$J|K]]|$1|L|3|M|5|6|7|Y|8|@]|9|@]|A|$]]|$1|N|3|O|5|I|7|Z|8|@]|9|@]|A|$J|K]]|$1|P|3|Q|5|6|7|10|8|@]|9|@]|A|$]]|$1|R|3|-4|5|6|7|11|8|@]|9|@]|A|$]]]|S|$]]

Javascript does http requests almost just like a browser does, once you figure out what they are you can try to re-create them in perl/python/etc.
With Firefox+Firebug you can see the requests in the 'Net' panel.

Things you might have to take into account are user-agent string, cookies, the fact that sometimes the returned data is meant to be run/interpreted by Javascript etc. Maybe your language of choice has a nice httpbrowser class you can use?

<hr>

Just took a look, searching for IBM, took the post data from Firebug, replaced newlines with '&amp;' and put it after the request url:

<pre><code>[http://bcode.bloomberg.com/sym/dwr/call/plaincall/searchMgr.search.dwr?callCount=1&amp;windowName=&amp;c0-scriptName=searchMgr&amp;c0-methodName=search&amp;c0-id=0&amp;c0-e1=string:ibm&amp;c0-e2=string:&amp;c0-e3=number:100&amp;c0-e4=number:0&amp;c0-e5=boolean:false&amp;c0-param0=Object_SearchCriteria:{search:reference:c0-e1,%20filter:reference:c0-e2,%20limit:reference:c0-e3&amp;,%20start:reference:c0-e4,%20allSources:reference:c0-e5}&amp;batchId=4&amp;page=%2Fsym%2F&amp;httpSessionId=&amp;scriptSessionId=FBC68693A4E1BC08D6E0DDFBDF6D0860]
</code></pre>

but it returns

<pre><code>throw 'allowScriptTagRemoting is false.';
//#DWR-REPLY
if (window.dwr) dwr.engine.remote.handleBatchException({ name:'java.lang.SecurityException', message:'GET Disallowed' });
else if (window.parent.dwr) window.parent.dwr.engine.remote.handleBatchException({ name:'java.lang.SecurityException', message:'GET Disallowed' });
</code></pre>

and no data.. So it looks like you have to script a post request. Looking at their restrictions and guidelines, maybe you should just get in touch and ask if there's a public API?

There's a search site whose search results are generated dynamically by javascript. So the user enters a query, and the site displays the content on the page, without refreshing. 

I need to grab those search results programmatically (say from a Java program or a perl/python script). 

So ideally, I can launch my program with 100 queries as user inputs, and then the program would hit that website with each query and spit out on my screen all the search results as returned by the website.

The obvious problem is that the site is in javasript instead of simple HTML, so sending a URL request and parsing the resultant output is not going to work (as the source code for this page is always just a bunch of reference to various .js files).

Given the above conditions, what are my options?

Best way to grab website content externally

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

有一个搜索网站，它的搜索结果是由javascript动态生成的。因此，用户输入查询，站点在页面上显示内容，而不进行刷新。我需要以编程方式获取这些搜索结果(比如从Java程序或perl/python脚本)。因此，理想情况下，我可以使用100个查询作为用户输入启动我的程序，然后程序会在每个查询中访问该网站，并在我的屏幕上显...

问从外部抓取网站内容的最佳方式
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问从外部抓取网站内容的最佳方式EN