blocks|key|1706492|text|在Unix/Linux机器上，你可以只运行'wget‘，但如果你正在编写一个跨平台的客户端，这并不是一个真正的选择。当然，这是假设您并不真的想对从下载到命中磁盘这段时间内下载的数据做太多操作。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1706493|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

On a Unix/Linux box you could just run 'wget' but this is not really an option if you're writing a cross-platform client. Of course this assumes that you don't really want to do much with the data you download between the point of downloading it and it hitting the disk.

blocks|key|1491459|text|下面是一些使用Java的URL类测试过的代码。不过，我建议在处理异常或将它们向上传递到调用堆栈方面做得比我在这里做得更好。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1491460|public+static+void+main(String[]+args)+{
++++URL+url;
++++InputStream+is+=+null;
++++BufferedReader+br;
++++String+line;

++++try+{
++++++++url+=+new+URL("http://stackoverflow.com/");
++++++++is+=+url.openStream();++//+throws+an+IOException
++++++++br+=+new+BufferedReader(new+InputStreamReader(is));

++++++++while+((line+=+br.readLine())+!=+null)+{
++++++++++++System.out.println(line);
++++++++}
++++}+catch+(MalformedURLException+mue)+{
+++++++++mue.printStackTrace();
++++}+catch+(IOException+ioe)+{
+++++++++ioe.printStackTrace();
++++}+finally+{
++++++++try+{
++++++++++++if+(is+!=+null)+is.close();
++++++++}+catch+(IOException+ioe)+{
++++++++++++//+nothing+to+see+here
++++++++}
++++}
}|code-block|syntax|javascript|1491461|entityMap|0|LINK|mutability|MUTABLE|url|http://java.sun.com/javase/6/docs/api/java/net/URL.html^0|C|3|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$G|H]]|$1|I|3|-4|5|6|7|V|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

Here's some tested code using Java's <a href="http://java.sun.com/javase/6/docs/api/java/net/URL.html" rel="noreferrer">URL</a> class. I'd recommend do a better job than I do here of handling the exceptions or passing them up the call stack, though.

<pre><code>public static void main(String[] args) {
 URL url;
 InputStream is = null;
 BufferedReader br;
 String line;

 try {
 url = new URL("http://stackoverflow.com/");
 is = url.openStream(); // throws an IOException
 br = new BufferedReader(new InputStreamReader(is));

 while ((line = br.readLine()) != null) {
 System.out.println(line);
 }
 } catch (MalformedURLException mue) {
 mue.printStackTrace();
 } catch (IOException ioe) {
 ioe.printStackTrace();
 } finally {
 try {
 if (is != null) is.close();
 } catch (IOException ioe) {
 // nothing to see here
 }
 }
}
</code></pre>

blocks|key|1491559|text|Bill的回答非常好，但您可能希望对请求执行一些操作，如压缩或用户代理。下面的代码展示了如何对请求进行各种类型的压缩。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1491560|URL+url+=+new+URL(urlStr);
HttpURLConnection+conn+=+(HttpURLConnection)+url.openConnection();+//+Cast+shouldn't+fail
HttpURLConnection.setFollowRedirects(true);
//+allow+both+GZip+and+Deflate+(ZLib)+encodings
conn.setRequestProperty("Accept-Encoding",+"gzip,+deflate");
String+encoding+=+conn.getContentEncoding();
InputStream+inStr+=+null;

//+create+the+appropriate+stream+wrapper+based+on
//+the+encoding+type
if+(encoding+!=+null+&&+encoding.equalsIgnoreCase("gzip"))+{
++++inStr+=+new+GZIPInputStream(conn.getInputStream());
}+else+if+(encoding+!=+null+&&+encoding.equalsIgnoreCase("deflate"))+{
++++inStr+=+new+InflaterInputStream(conn.getInputStream(),
++++++new+Inflater(true));
}+else+{
++++inStr+=+conn.getInputStream();
}|code-block|syntax|javascript|1491561|要同时设置用户代理，请添加以下代码：|1491562|conn.setRequestProperty+(+"User-agent",+"my+agent+name");|1491563|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

Bill's answer is very good, but you may want to do some things with the request like compression or user-agents. The following code shows how you can various types of compression to your requests.

<pre><code>URL url = new URL(urlStr);
HttpURLConnection conn = (HttpURLConnection) url.openConnection(); // Cast shouldn't fail
HttpURLConnection.setFollowRedirects(true);
// allow both GZip and Deflate (ZLib) encodings
conn.setRequestProperty("Accept-Encoding", "gzip, deflate");
String encoding = conn.getContentEncoding();
InputStream inStr = null;

// create the appropriate stream wrapper based on
// the encoding type
if (encoding != null &amp;&amp; encoding.equalsIgnoreCase("gzip")) {
 inStr = new GZIPInputStream(conn.getInputStream());
} else if (encoding != null &amp;&amp; encoding.equalsIgnoreCase("deflate")) {
 inStr = new InflaterInputStream(conn.getInputStream(),
 new Inflater(true));
} else {
 inStr = conn.getInputStream();
}
</code></pre>

To also set the user-agent add the following code:

<pre><code>conn.setRequestProperty ( "User-agent", "my agent name");
</code></pre>

blocks|key|1491611|text|上面提到的所有方法都不会下载在浏览器中出现的网页文本。现在很多数据都是通过html页面中的脚本加载到浏览器中的。上面提到的技术都不支持脚本，它们只下载html文本。HTMLUNIT支持javascripts。因此，如果你想下载在浏览器中显示的网页文本，那么你应该使用HTMLUNIT。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1491612|entityMap|0|LINK|mutability|MUTABLE|url|http://htmlunit.sourceforge.net/^0|3P|8|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

All the above mentioned approaches do not download the web page text as it looks in the browser. these days a lot of data is loaded into browsers through scripts in html pages. none of above mentioned techniques supports scripts, they just downloads the html text only. HTMLUNIT supports the javascripts. so if you are looking to download the web page text as it looks in the browser then you should use <a href="http://htmlunit.sourceforge.net" rel="noreferrer">HTMLUNIT</a>.

blocks|key|1706734|text|Jetty具有HTTP客户端，可用于下载网页。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1706735|package+com.zetcode;

import+org.eclipse.jetty.client.HttpClient;
import+org.eclipse.jetty.client.api.ContentResponse;

public+class+ReadWebPageEx5+{

++++public+static+void+main(String[]+args)+throws+Exception+{

++++++++HttpClient+client+=+null;

++++++++try+{

++++++++++++client+=+new+HttpClient();
++++++++++++client.start();
++++++++++++
++++++++++++String+url+=+"http://example.com";

++++++++++++ContentResponse+res+=+client.GET(url);

++++++++++++System.out.println(res.getContentAsString());

++++++++}+finally+{

++++++++++++if+(client+!=+null)+{

++++++++++++++++client.stop();
++++++++++++}
++++++++}
++++}
}|code-block|syntax|javascript|1706736|该示例打印一个简单网页的内容。|1706737|在Reading+a+web+page+in+Java教程中，我写了6个使用URL、JSoup、HtmlCleaner、Apache+HttpClient、Jetty+HttpClient和HtmlUnit编程下载网页的例子。|offset|length|1706738|entityMap|0|LINK|mutability|MUTABLE|url|https://zetcode.com/java/readwebpage/^0|0|0|0|1|Q|0|0^^$0|@$1|2|3|4|5|6|7|U|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|V|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|W|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|X|8|@]|9|@$K|Y|L|Z|1|10]]|A|$]]|$1|M|3|-4|5|6|7|11|8|@]|9|@]|A|$]]]|N|$O|$5|P|Q|R|A|$S|T]]]]

Jetty has an HTTP client which can be use to download a web page.
<pre><code>package com.zetcode;

import org.eclipse.jetty.client.HttpClient;
import org.eclipse.jetty.client.api.ContentResponse;

public class ReadWebPageEx5 {

 public static void main(String[] args) throws Exception {

 HttpClient client = null;

 try {

 client = new HttpClient();
 client.start();
 
 String url = &quot;http://example.com&quot;;

 ContentResponse res = client.GET(url);

 System.out.println(res.getContentAsString());

 } finally {

 if (client != null) {

 client.stop();
 }
 }
 }
}
</code></pre>
The example prints the contents of a simple web page.
In a <a href="https://zetcode.com/java/readwebpage/" rel="nofollow noreferrer">Reading a web page in Java</a> tutorial I have written six examples of dowloading a web page programmaticaly in Java using URL, JSoup, HtmlCleaner, Apache HttpClient, Jetty HttpClient, and HtmlUnit.

blocks|key|1491758|text|你很可能需要从一个安全的网页(https协议)中提取代码。在下面的示例中，html文件将被保存到c:\temp\filename.html+Enjoy！|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1491759|import+java.io.BufferedReader;
import+java.io.BufferedWriter;
import+java.io.FileWriter;
import+java.io.InputStream;
import+java.io.InputStreamReader;
import+java.net.URL;

import+javax.net.ssl.HttpsURLConnection;

/**
+*+Get+the+Html+source+from+the+secure+url+
+*/
public+class+HttpsClientUtil+{
++++public+static+void+main(String[]+args)+throws+Exception+{
++++++++String+httpsURL+=+"https://stackoverflow.com";
++++++++String+FILENAME+=+"c:\\temp\\filename.html";
++++++++BufferedWriter+bw+=+new+BufferedWriter(new+FileWriter(FILENAME));
++++++++URL+myurl+=+new+URL(httpsURL);
++++++++HttpsURLConnection+con+=+(HttpsURLConnection)+myurl.openConnection();
++++++++con.setRequestProperty+(+"User-Agent",+"Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64;+rv:63.0)+Gecko/20100101+Firefox/63.0"+);
++++++++InputStream+ins+=+con.getInputStream();
++++++++InputStreamReader+isr+=+new+InputStreamReader(ins,+"Windows-1252");
++++++++BufferedReader+in+=+new+BufferedReader(isr);
++++++++String+inputLine;

++++++++//+Write+each+line+into+the+file
++++++++while+((inputLine+=+in.readLine())+!=+null)+{
++++++++++++System.out.println(inputLine);
++++++++++++bw.write(inputLine);
++++++++}
++++++++in.close();+
++++++++bw.close();
++++}
}|code-block|syntax|javascript|1491760|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

You'd most likely need to extract code from a secure web page (https protocol). In the following example, the html file is being saved into c:\temp\filename.html Enjoy!

<pre><code>import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URL;

import javax.net.ssl.HttpsURLConnection;

/**
 * &lt;b&gt;Get the Html source from the secure url &lt;/b&gt;
 */
public class HttpsClientUtil {
 public static void main(String[] args) throws Exception {
 String httpsURL = "https://stackoverflow.com";
 String FILENAME = "c:\\temp\\filename.html";
 BufferedWriter bw = new BufferedWriter(new FileWriter(FILENAME));
 URL myurl = new URL(httpsURL);
 HttpsURLConnection con = (HttpsURLConnection) myurl.openConnection();
 con.setRequestProperty ( "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0" );
 InputStream ins = con.getInputStream();
 InputStreamReader isr = new InputStreamReader(ins, "Windows-1252");
 BufferedReader in = new BufferedReader(isr);
 String inputLine;

 // Write each line into the file
 while ((inputLine = in.readLine()) != null) {
 System.out.println(inputLine);
 bw.write(inputLine);
 }
 in.close(); 
 bw.close();
 }
}
</code></pre>

blocks|key|1491805|text|要使用NIO.2强大的Files.copy(InputStream+in，路径目标)来完成此操作：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1491806|URL+url+=+new+URL(+"http://download.me/"+);
Files.copy(+url.openStream(),+Paths.get("downloaded.html"+)+);|code-block|syntax|javascript|1491807|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

To do so using NIO.2 powerful Files.copy(InputStream in, Path target):

<pre><code>URL url = new URL( "http://download.me/" );
Files.copy( url.openStream(), Paths.get("downloaded.html" ) );
</code></pre>

I would like to be able to fetch a web page's html and save it to a <code>String</code>, so I can do some processing on it. Also, how could I handle various types of compression.

How would I go about doing that using Java?

How do you Programmatically Download a Webpage in Java

Windows

Linux

Java

Unix

我希望能够获取网页的html并将其保存到String，这样我就可以对其进行一些处理。另外，我如何处理各种类型的压缩。我该如何使用Java来做这件事呢？

问如何在Java中以编程方式下载网页
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在Java中以编程方式下载网页EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在Java中以编程方式下载网页
EN