httpclient参数配置
序
这里简单解释一下httpclient一些关键参数的配置
超时时间
final RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(SOCKET_TIMEOUT)
.setConnectTimeout(CONNECTION_TIMEOUT)
.setCookieSpec(CookieSpecs.IGNORE_COOKIES).build();这里设置了socket_timeout以及connection_timeout
KeepAliveStrategy
httpclient默认提供了一个策略
httpclient-4.5.3-sources.jar!/org/apache/http/impl/client/DefaultConnectionKeepAliveStrategy.java
@Contract(threading = ThreadingBehavior.IMMUTABLE)
public class DefaultConnectionKeepAliveStrategy implements ConnectionKeepAliveStrategy {
public static final DefaultConnectionKeepAliveStrategy INSTANCE = new DefaultConnectionKeepAliveStrategy();
@Override
public long getKeepAliveDuration(final HttpResponse response, final HttpContext context) {
Args.notNull(response, "HTTP response");
final HeaderElementIterator it = new BasicHeaderElementIterator(
response.headerIterator(HTTP.CONN_KEEP_ALIVE));
while (it.hasNext()) {
final HeaderElement he = it.nextElement();
final String param = he.getName();
final String value = he.getValue();
if (value != null && param.equalsIgnoreCase("timeout")) {
try {
return Long.parseLong(value) * 1000;
} catch(final NumberFormatException ignore) {
}
}
}
return -1;
}
}默认的话，是从response里头读timeout参数的，没有读到则设置为-1，这个代表无穷，这样设置是有点问题了，如果是https链接的话，则可能会经常报
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)HTTP规范没有确定一个持久连接可能或应该保持活动多长时间。一些HTTP服务器使用非标准的头部信息Keep-Alive来告诉客户端它们想在服务器端保持连接活动的周期秒数。如果这个信息可用，HttClient就会利用这个它。如果头部信息Keep-Alive在响应中不存在，HttpClient假设连接无限期的保持活动。然而许多现实中的HTTP服务器配置了在特定不活动周期之后丢掉持久连接来保存系统资源，往往这是不通知客户端的。
这里可以在此基础上重写一个，这里设置为5秒
ConnectionKeepAliveStrategy keepAliveStrategy = new DefaultConnectionKeepAliveStrategy() {
@Override
public long getKeepAliveDuration(final HttpResponse response, final HttpContext context) {
long keepAlive = super.getKeepAliveDuration(response, context);
if (keepAlive == -1) {
keepAlive = 5000;
}
return keepAlive;
}
};Connection eviction policy
一个经典的阻塞 I/O 模型的主要缺点是网络套接字仅当 I/O 操作阻塞时才可以响应 I/O 事件。当一个连接被释放返回管理器时，它可以被保持活动状态而却不能监控套接字的状态和响应任何 I/O 事件。如果连接在服务器端关闭，那么客户端连接也不能去侦测连接状态中的变化和关闭本端的套接字去作出适当响应。 HttpClient 通过测试连接是否是过时的来尝试去减轻这个问题，这已经不再有效了，因为它已经在服务器端关闭了，之前使用执行 HTTP 请求的连接。过时的连接检查也并不是 100% 的稳定，反而对每次请求执行还要增加10到30毫秒的开销。唯一可行的而不涉及到每个对空闲连接的套接字模型线程解决方案，是使用专用的监控线程来收回因为长时间不活动而被认为是过期的连接。监控线程可以周期地调用 ClientConnectionManager#closeExpiredConnections()方法来关闭所有过期的连接，从连接池中收回关闭的连接。它也可以选择性调用ClientConnectionManager#closeIdleConnections()方法来关闭所有已经空 闲超过给定时间周期的连接。
官方提供了一个实例
public static class IdleConnectionMonitorThread extends Thread {
private final HttpClientConnectionManager connMgr;
private volatile boolean shutdown;
public IdleConnectionMonitorThread(HttpClientConnectionManager connMgr) {
super();
this.connMgr = connMgr;
}
@Override
public void run() {
try {
while (!shutdown) {
synchronized (this) {
wait(5000);
// Close expired connections
connMgr.closeExpiredConnections();
// Optionally, close connections
// that have been idle longer than 30 sec
connMgr.closeIdleConnections(30, TimeUnit.SECONDS);
}
}
} catch (InterruptedException ex) {
// terminate
}
}
public void shutdown() {
shutdown = true;
synchronized (this) {
notifyAll();
}
}
}在spring cloud netflix zuul中则提供了类似的定时器，只不过参数是写死的 spring-cloud-netflix-core/1.2.6.RELEASE/spring-cloud-netflix-core-1.2.6.RELEASE-sources.jar!/org/springframework/cloud/netflix/zuul/filters/route/SimpleHostRoutingFilter.java
private final Timer connectionManagerTimer = new Timer(
"SimpleHostRoutingFilter.connectionManagerTimer", true);
@PostConstruct
private void initialize() {
this.httpClient = newClient();
SOCKET_TIMEOUT.addCallback(this.clientloader);
CONNECTION_TIMEOUT.addCallback(this.clientloader);
this.connectionManagerTimer.schedule(new TimerTask() {
@Override
public void run() {
if (SimpleHostRoutingFilter.this.connectionManager == null) {
return;
}
SimpleHostRoutingFilter.this.connectionManager.closeExpiredConnections();
}
}, 30000, 5000);
}每30秒清理一次失效的connection，但是这里的closeExpiredConnections是依赖connTimeToLive参数以及keep alive strategy设置的。 httpcore-4.4.6-sources.jar!/org/apache/http/pool/AbstractConnPool.java
/**
* Closes expired connections and evicts them from the pool.
*/
public void closeExpired() {
final long now = System.currentTimeMillis();
enumAvailable(new PoolEntryCallback<T, C>() {
@Override
public void process(final PoolEntry<T, C> entry) {
if (entry.isExpired(now)) {
entry.close();
}
}
});
}httpcore-4.4.6-sources.jar!/org/apache/http/pool/PoolEntry.java
public synchronized boolean isExpired(final long now) {
return now >= this.expiry;
}
public PoolEntry(final String id, final T route, final C conn,
final long timeToLive, final TimeUnit tunit) {
super();
Args.notNull(route, "Route");
Args.notNull(conn, "Connection");
Args.notNull(tunit, "Time unit");
this.id = id;
this.route = route;
this.conn = conn;
this.created = System.currentTimeMillis();
this.updated = this.created;
if (timeToLive > 0) {
this.validityDeadline = this.created + tunit.toMillis(timeToLive);
} else {
this.validityDeadline = Long.MAX_VALUE;
}
this.expiry = this.validityDeadline;
}
public synchronized void updateExpiry(final long time, final TimeUnit tunit) {
Args.notNull(tunit, "Time unit");
this.updated = System.currentTimeMillis();
final long newExpiry;
if (time > 0) {
newExpiry = this.updated + tunit.toMillis(time);
} else {
newExpiry = Long.MAX_VALUE;
}
this.expiry = Math.min(newExpiry, this.validityDeadline);
}如果是-1，则这里expiry为无穷大。那么closeExpiredConnections其实是无效的。但是还依赖一个keep alive strategy，它会去设置updateExpiry httpclient-4.5.3-sources.jar!/org/apache/http/impl/execchain/MainClientExec.java 主要看httpclient-4.5.3-sources.jar!/org/apache/http/impl/execchain/ConnectionHolder.java
public void setValidFor(final long duration, final TimeUnit tunit) {
synchronized (this.managedConn) {
this.validDuration = duration;
this.tunit = tunit;
}
}
private void releaseConnection(final boolean reusable) {
if (this.released.compareAndSet(false, true)) {
synchronized (this.managedConn) {
if (reusable) {
this.manager.releaseConnection(this.managedConn,
this.state, this.validDuration, this.tunit);
} else {
try {
this.managedConn.close();
log.debug("Connection discarded");
} catch (final IOException ex) {
if (this.log.isDebugEnabled()) {
this.log.debug(ex.getMessage(), ex);
}
} finally {
this.manager.releaseConnection(
this.managedConn, null, 0, TimeUnit.MILLISECONDS);
}
}
}
}
}这里设置了validDuration，会传给releaseConnection httpclient-4.5.3-sources.jar!/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.java
public void releaseConnection(
final HttpClientConnection managedConn,
final Object state,
final long keepalive, final TimeUnit tunit) {
Args.notNull(managedConn, "Managed connection");
synchronized (managedConn) {
final CPoolEntry entry = CPoolProxy.detach(managedConn);
if (entry == null) {
return;
}
final ManagedHttpClientConnection conn = entry.getConnection();
try {
if (conn.isOpen()) {
final TimeUnit effectiveUnit = tunit != null ? tunit : TimeUnit.MILLISECONDS;
entry.setState(state);
entry.updateExpiry(keepalive, effectiveUnit);
if (this.log.isDebugEnabled()) {
final String s;
if (keepalive > 0) {
s = "for " + (double) effectiveUnit.toMillis(keepalive) / 1000 + " seconds";
} else {
s = "indefinitely";
}
this.log.debug("Connection " + format(entry) + " can be kept alive " + s);
}
}
} finally {
this.pool.release(entry, conn.isOpen() && entry.isRouteComplete());
if (this.log.isDebugEnabled()) {
this.log.debug("Connection released: " + format(entry) + formatStats(entry.getRoute()));
}
}
}
}这里去updateExpiry，相当于更新了timeToLive
connTimeToLive
这里设置的是http连接池中connection的存活时间 httpclient-4.5.3-sources.jar!/org/apache/http/impl/client/HttpClientBuilder.java
/**
* Sets maximum time to live for persistent connections
* <p>
* Please note this value can be overridden by the {@link #setConnectionManager(
* org.apache.http.conn.HttpClientConnectionManager)} method.
* </p>
*
* @since 4.4
*/
public final HttpClientBuilder setConnectionTimeToLive(final long connTimeToLive, final TimeUnit connTimeToLiveTimeUnit) {
this.connTimeToLive = connTimeToLive;
this.connTimeToLiveTimeUnit = connTimeToLiveTimeUnit;
return this;
}httpclient-4.5.3-sources.jar!/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.java
public PoolingHttpClientConnectionManager(
final Registry<ConnectionSocketFactory> socketFactoryRegistry,
final HttpConnectionFactory<HttpRoute, ManagedHttpClientConnection> connFactory,
final DnsResolver dnsResolver) {
this(socketFactoryRegistry, connFactory, null, dnsResolver, -1, TimeUnit.MILLISECONDS);
}
public PoolingHttpClientConnectionManager(
final Registry<ConnectionSocketFactory> socketFactoryRegistry,
final HttpConnectionFactory<HttpRoute, ManagedHttpClientConnection> connFactory,
final SchemePortResolver schemePortResolver,
final DnsResolver dnsResolver,
final long timeToLive, final TimeUnit tunit) {
this(
new DefaultHttpClientConnectionOperator(socketFactoryRegistry, schemePortResolver, dnsResolver),
connFactory,
timeToLive, tunit
);
}默认构造的话，timeToLive是-1，如果默认是-1的话，则不失效
maxTotal 以及 defaultMaxPerRoute
4.0的ThreadSafeClientConnManager，在4.2版本的时候被废弃了，默认使用PoolingHttpClientConnectionManager。这个manager有两个重要的参数，一个是maxTotal，一个是defaultMaxPerRoute。
每个默认的实现对每个给定路由将会创建不超过2个的并发连接，而总共也不会超过 20 个连接。
this.pool = new CPool(new InternalConnectionFactory(this.configData, connFactory), 2, 20, timeToLive, tunit);对于很多真实的应用程序，这个限制也证明很大的制约，特别是他们在服务中使用 HTTP 作为 传输协议。连接限制，也可以使用 HTTP 参数来进行调整。 spring cloud netflix zuul 里头默认配置是总共200连接，每个route不超过20个连接
this.connectionManager = new PoolingHttpClientConnectionManager(registry);
this.connectionManager.setMaxTotal(this.hostProperties.getMaxTotalConnections()); // 200
this.connectionManager.setDefaultMaxPerRoute(this.hostProperties.getMaxPerRouteConnections()); //20HttpRoute
httpclient-4.5.3-sources.jar!/org/apache/http/conn/routing/HttpRoute.java HttpRoute对象是immutable的，包含的数据有目标主机、本地地址、代理链、是否tunnulled、是否layered、是否是安全路由。
@Contract(threading = ThreadingBehavior.IMMUTABLE)
public final class HttpRoute implements RouteInfo, Cloneable {
/** The target host to connect to. */
private final HttpHost targetHost;
/**
* The local address to connect from.
* {@code null} indicates that the default should be used.
*/
private final InetAddress localAddress;
/** The proxy servers, if any. Never null. */
private final List<HttpHost> proxyChain;
/** Whether the the route is tunnelled through the proxy. */
private final TunnelType tunnelled;
/** Whether the route is layered. */
private final LayerType layered;
/** Whether the route is (supposed to be) secure. */
private final boolean secure;
//...
}IdleConnectionEvictor
httpclient-4.5.3-sources.jar!/org/apache/http/impl/client/IdleConnectionEvictor.java
public IdleConnectionEvictor(
final HttpClientConnectionManager connectionManager,
final ThreadFactory threadFactory,
final long sleepTime, final TimeUnit sleepTimeUnit,
final long maxIdleTime, final TimeUnit maxIdleTimeUnit) {
this.connectionManager = Args.notNull(connectionManager, "Connection manager");
this.threadFactory = threadFactory != null ? threadFactory : new DefaultThreadFactory();
this.sleepTimeMs = sleepTimeUnit != null ? sleepTimeUnit.toMillis(sleepTime) : sleepTime;
this.maxIdleTimeMs = maxIdleTimeUnit != null ? maxIdleTimeUnit.toMillis(maxIdleTime) : maxIdleTime;
this.thread = this.threadFactory.newThread(new Runnable() {
@Override
public void run() {
try {
while (!Thread.currentThread().isInterrupted()) {
Thread.sleep(sleepTimeMs);
connectionManager.closeExpiredConnections();
if (maxIdleTimeMs > 0) {
connectionManager.closeIdleConnections(maxIdleTimeMs, TimeUnit.MILLISECONDS);
}
}
} catch (final Exception ex) {
exception = ex;
}
}
});
}builder默认会构造一个IdleConnectionEvictor httpclient-4.5.3-sources.jar!/org/apache/http/impl/client/HttpClientBuilder.java
if (!this.connManagerShared) {
if (closeablesCopy == null) {
closeablesCopy = new ArrayList<Closeable>(1);
}
final HttpClientConnectionManager cm = connManagerCopy;
if (evictExpiredConnections || evictIdleConnections) {
final IdleConnectionEvictor connectionEvictor = new IdleConnectionEvictor(cm,
maxIdleTime > 0 ? maxIdleTime : 10, maxIdleTimeUnit != null ? maxIdleTimeUnit : TimeUnit.SECONDS);
closeablesCopy.add(new Closeable() {
@Override
public void close() throws IOException {
connectionEvictor.shutdown();
}
});
connectionEvictor.start();
}
closeablesCopy.add(new Closeable() {
@Override
public void close() throws IOException {
cm.shutdown();
}
});
}如果没有指定maxIdleTime的话，但是有设置evictExpiredConnections的话，默认是10秒
public final HttpClientBuilder evictExpiredConnections() {
evictExpiredConnections = true;
return this;
}
public final HttpClientBuilder evictIdleConnections(final long maxIdleTime, final TimeUnit maxIdleTimeUnit) {
this.evictIdleConnections = true;
this.maxIdleTime = maxIdleTime;
this.maxIdleTimeUnit = maxIdleTimeUnit;
return this;
}小结
当getKeepAliveDuration为-1以及connTimeToLive为-1的时候，closeExpiredConnections方法其实是没有用的，查看debug日志会出现
o.a.http.impl.client.DefaultHttpClient 509 : Connection can be kept alive indefinitely当getKeepAliveDuration不为-1的话，假设是5s,则日志可能是这样的
o.a.http.impl.execchain.MainClientExec 285 : Connection can be kept alive for 5000 MILLISECONDSdoc
- httpcomponents-client-4.5.x-tutorial
- HttpClient 教程南磊译
- 使用httpclient必须知道的参数设置及代码写法、存在的风险
- Apache HttpComponents学习笔记（四）：HttpClient里的HttpRoute
- spring boot封装HttpClient