httpclient参数配置

这里简单解释一下httpclient一些关键参数的配置

超时时间

final RequestConfig requestConfig = RequestConfig.custom()
                .setSocketTimeout(SOCKET_TIMEOUT)
                .setConnectTimeout(CONNECTION_TIMEOUT)
                .setCookieSpec(CookieSpecs.IGNORE_COOKIES).build();

这里设置了socket_timeout以及connection_timeout

KeepAliveStrategy

httpclient默认提供了一个策略

httpclient-4.5.3-sources.jar!/org/apache/http/impl/client/DefaultConnectionKeepAliveStrategy.java

@Contract(threading = ThreadingBehavior.IMMUTABLE)
public class DefaultConnectionKeepAliveStrategy implements ConnectionKeepAliveStrategy {

    public static final DefaultConnectionKeepAliveStrategy INSTANCE = new DefaultConnectionKeepAliveStrategy();

    @Override
    public long getKeepAliveDuration(final HttpResponse response, final HttpContext context) {
        Args.notNull(response, "HTTP response");
        final HeaderElementIterator it = new BasicHeaderElementIterator(
                response.headerIterator(HTTP.CONN_KEEP_ALIVE));
        while (it.hasNext()) {
            final HeaderElement he = it.nextElement();
            final String param = he.getName();
            final String value = he.getValue();
            if (value != null && param.equalsIgnoreCase("timeout")) {
                try {
                    return Long.parseLong(value) * 1000;
                } catch(final NumberFormatException ignore) {
                }
            }
        }
        return -1;
    }

}

默认的话,是从response里头读timeout参数的,没有读到则设置为-1,这个代表无穷,这样设置是有点问题了,如果是https链接的话,则可能会经常报

Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:170)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
    at sun.security.ssl.InputRecord.read(InputRecord.java:503)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)

HTTP规范没有确定一个持久连接可能或应该保持活动多长时间。一些HTTP服务器使用非标准的头部信息Keep-Alive来告诉客户端它们想在服务器端保持连接活动的周期秒数。如果这个信息可用,HttClient就会利用这个它。如果头部信息Keep-Alive在响应中不存在,HttpClient假设连接无限期的保持活动。然而许多现实中的HTTP服务器配置了在特定不活动周期之后丢掉持久连接来保存系统资源,往往这是不通知客户端的。

这里可以在此基础上重写一个,这里设置为5秒

ConnectionKeepAliveStrategy keepAliveStrategy = new DefaultConnectionKeepAliveStrategy() {
            @Override
            public long getKeepAliveDuration(final HttpResponse response, final HttpContext context) {
                long keepAlive = super.getKeepAliveDuration(response, context);
                if (keepAlive == -1) {
                    keepAlive = 5000;
                }
                return keepAlive;
            }
        };

Connection eviction policy

一个经典的阻塞 I/O 模型的主要缺点是网络套接字仅当 I/O 操作阻塞时才可以响应 I/O 事件。当一个连接被释放返回管理器时,它可以被保持活动状态而却不能监控套接字的状态和响应任何 I/O 事件。如果连接在服务器端关闭,那么客户端连接也不能去侦测连接状态中的变化和关闭本端的套接字去作出适当响应。 HttpClient 通过测试连接是否是过时的来尝试去减轻这个问题,这已经不再有效了,因为它已经在服务器端关闭了,之前使用执行 HTTP 请求的连接。过时的连接检查也并不是 100% 的稳定,反而对每次请求执行还要增加10到30毫秒的开销。唯一可行的而不涉及到每个对空闲连接的套接字模型线程解决方案,是使用专用的监控线程来收回因为长时间不活动而被认为是过期的连接。监控线程可以周期地调用 ClientConnectionManager#closeExpiredConnections()方法来关闭所有过期的连接,从连接池中收回关闭的连接。它也可以选择性调用ClientConnectionManager#closeIdleConnections()方法来关闭所有已经空 闲超过给定时间周期的连接。

官方提供了一个实例

public static class IdleConnectionMonitorThread extends Thread {

    private final HttpClientConnectionManager connMgr;
    private volatile boolean shutdown;

    public IdleConnectionMonitorThread(HttpClientConnectionManager connMgr) {
        super();
        this.connMgr = connMgr;
    }

    @Override
    public void run() {
        try {
            while (!shutdown) {
                synchronized (this) {
                    wait(5000);
                    // Close expired connections
                    connMgr.closeExpiredConnections();
                    // Optionally, close connections
                    // that have been idle longer than 30 sec
                    connMgr.closeIdleConnections(30, TimeUnit.SECONDS);
                }
            }
        } catch (InterruptedException ex) {
            // terminate
        }
    }

    public void shutdown() {
        shutdown = true;
        synchronized (this) {
            notifyAll();
        }
    }

}

在spring cloud netflix zuul中则提供了类似的定时器,只不过参数是写死的 spring-cloud-netflix-core/1.2.6.RELEASE/spring-cloud-netflix-core-1.2.6.RELEASE-sources.jar!/org/springframework/cloud/netflix/zuul/filters/route/SimpleHostRoutingFilter.java

private final Timer connectionManagerTimer = new Timer(
            "SimpleHostRoutingFilter.connectionManagerTimer", true);

@PostConstruct
    private void initialize() {
        this.httpClient = newClient();
        SOCKET_TIMEOUT.addCallback(this.clientloader);
        CONNECTION_TIMEOUT.addCallback(this.clientloader);
        this.connectionManagerTimer.schedule(new TimerTask() {
            @Override
            public void run() {
                if (SimpleHostRoutingFilter.this.connectionManager == null) {
                    return;
                }
                SimpleHostRoutingFilter.this.connectionManager.closeExpiredConnections();
            }
        }, 30000, 5000);
    }

每30秒清理一次失效的connection,但是这里的closeExpiredConnections是依赖connTimeToLive参数以及keep alive strategy设置的。 httpcore-4.4.6-sources.jar!/org/apache/http/pool/AbstractConnPool.java

/**
     * Closes expired connections and evicts them from the pool.
     */
    public void closeExpired() {
        final long now = System.currentTimeMillis();
        enumAvailable(new PoolEntryCallback<T, C>() {

            @Override
            public void process(final PoolEntry<T, C> entry) {
                if (entry.isExpired(now)) {
                    entry.close();
                }
            }

        });
    }

httpcore-4.4.6-sources.jar!/org/apache/http/pool/PoolEntry.java

public synchronized boolean isExpired(final long now) {
        return now >= this.expiry;
    }
public PoolEntry(final String id, final T route, final C conn,
            final long timeToLive, final TimeUnit tunit) {
        super();
        Args.notNull(route, "Route");
        Args.notNull(conn, "Connection");
        Args.notNull(tunit, "Time unit");
        this.id = id;
        this.route = route;
        this.conn = conn;
        this.created = System.currentTimeMillis();
        this.updated = this.created;
        if (timeToLive > 0) {
            this.validityDeadline = this.created + tunit.toMillis(timeToLive);
        } else {
            this.validityDeadline = Long.MAX_VALUE;
        }
        this.expiry = this.validityDeadline;
    }
public synchronized void updateExpiry(final long time, final TimeUnit tunit) {
        Args.notNull(tunit, "Time unit");
        this.updated = System.currentTimeMillis();
        final long newExpiry;
        if (time > 0) {
            newExpiry = this.updated + tunit.toMillis(time);
        } else {
            newExpiry = Long.MAX_VALUE;
        }
        this.expiry = Math.min(newExpiry, this.validityDeadline);
    }

如果是-1,则这里expiry为无穷大。那么closeExpiredConnections其实是无效的。但是还依赖一个keep alive strategy,它会去设置updateExpiry httpclient-4.5.3-sources.jar!/org/apache/http/impl/execchain/MainClientExec.java 主要看httpclient-4.5.3-sources.jar!/org/apache/http/impl/execchain/ConnectionHolder.java

public void setValidFor(final long duration, final TimeUnit tunit) {
        synchronized (this.managedConn) {
            this.validDuration = duration;
            this.tunit = tunit;
        }
    }

    private void releaseConnection(final boolean reusable) {
        if (this.released.compareAndSet(false, true)) {
            synchronized (this.managedConn) {
                if (reusable) {
                    this.manager.releaseConnection(this.managedConn,
                            this.state, this.validDuration, this.tunit);
                } else {
                    try {
                        this.managedConn.close();
                        log.debug("Connection discarded");
                    } catch (final IOException ex) {
                        if (this.log.isDebugEnabled()) {
                            this.log.debug(ex.getMessage(), ex);
                        }
                    } finally {
                        this.manager.releaseConnection(
                                this.managedConn, null, 0, TimeUnit.MILLISECONDS);
                    }
                }
            }
        }
    }

这里设置了validDuration,会传给releaseConnection httpclient-4.5.3-sources.jar!/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.java

public void releaseConnection(
            final HttpClientConnection managedConn,
            final Object state,
            final long keepalive, final TimeUnit tunit) {
        Args.notNull(managedConn, "Managed connection");
        synchronized (managedConn) {
            final CPoolEntry entry = CPoolProxy.detach(managedConn);
            if (entry == null) {
                return;
            }
            final ManagedHttpClientConnection conn = entry.getConnection();
            try {
                if (conn.isOpen()) {
                    final TimeUnit effectiveUnit = tunit != null ? tunit : TimeUnit.MILLISECONDS;
                    entry.setState(state);
                    entry.updateExpiry(keepalive, effectiveUnit);
                    if (this.log.isDebugEnabled()) {
                        final String s;
                        if (keepalive > 0) {
                            s = "for " + (double) effectiveUnit.toMillis(keepalive) / 1000 + " seconds";
                        } else {
                            s = "indefinitely";
                        }
                        this.log.debug("Connection " + format(entry) + " can be kept alive " + s);
                    }
                }
            } finally {
                this.pool.release(entry, conn.isOpen() && entry.isRouteComplete());
                if (this.log.isDebugEnabled()) {
                    this.log.debug("Connection released: " + format(entry) + formatStats(entry.getRoute()));
                }
            }
        }
    }

这里去updateExpiry,相当于更新了timeToLive

connTimeToLive

这里设置的是http连接池中connection的存活时间 httpclient-4.5.3-sources.jar!/org/apache/http/impl/client/HttpClientBuilder.java

/**
     * Sets maximum time to live for persistent connections
     * <p>
     * Please note this value can be overridden by the {@link #setConnectionManager(
     *   org.apache.http.conn.HttpClientConnectionManager)} method.
     * </p>
     *
     * @since 4.4
     */
    public final HttpClientBuilder setConnectionTimeToLive(final long connTimeToLive, final TimeUnit connTimeToLiveTimeUnit) {
        this.connTimeToLive = connTimeToLive;
        this.connTimeToLiveTimeUnit = connTimeToLiveTimeUnit;
        return this;
    }

httpclient-4.5.3-sources.jar!/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.java

public PoolingHttpClientConnectionManager(
            final Registry<ConnectionSocketFactory> socketFactoryRegistry,
            final HttpConnectionFactory<HttpRoute, ManagedHttpClientConnection> connFactory,
            final DnsResolver dnsResolver) {
        this(socketFactoryRegistry, connFactory, null, dnsResolver, -1, TimeUnit.MILLISECONDS);
    }
public PoolingHttpClientConnectionManager(
            final Registry<ConnectionSocketFactory> socketFactoryRegistry,
            final HttpConnectionFactory<HttpRoute, ManagedHttpClientConnection> connFactory,
            final SchemePortResolver schemePortResolver,
            final DnsResolver dnsResolver,
            final long timeToLive, final TimeUnit tunit) {
        this(
            new DefaultHttpClientConnectionOperator(socketFactoryRegistry, schemePortResolver, dnsResolver),
            connFactory,
            timeToLive, tunit
        );
    }

默认构造的话,timeToLive是-1,如果默认是-1的话,则不失效

maxTotal 以及 defaultMaxPerRoute

4.0的ThreadSafeClientConnManager,在4.2版本的时候被废弃了,默认使用PoolingHttpClientConnectionManager。这个manager有两个重要的参数,一个是maxTotal,一个是defaultMaxPerRoute。

每个默认的实现对每个给定路由将会创建不超过2个的并发连接,而总共也不会超过 20 个连接。

 this.pool = new CPool(new InternalConnectionFactory(this.configData, connFactory), 2, 20, timeToLive, tunit);

对于很多真实的应用程序,这个限制也证明很大的制约,特别是他们在服务中使用 HTTP 作为 传输协议。连接限制,也可以使用 HTTP 参数来进行调整。 spring cloud netflix zuul 里头默认配置是总共200连接,每个route不超过20个连接

this.connectionManager = new PoolingHttpClientConnectionManager(registry);
            this.connectionManager.setMaxTotal(this.hostProperties.getMaxTotalConnections()); // 200
            this.connectionManager.setDefaultMaxPerRoute(this.hostProperties.getMaxPerRouteConnections()); //20

HttpRoute

httpclient-4.5.3-sources.jar!/org/apache/http/conn/routing/HttpRoute.java HttpRoute对象是immutable的,包含的数据有目标主机、本地地址、代理链、是否tunnulled、是否layered、是否是安全路由。

@Contract(threading = ThreadingBehavior.IMMUTABLE)
public final class HttpRoute implements RouteInfo, Cloneable {

    /** The target host to connect to. */
    private final HttpHost targetHost;

    /**
     * The local address to connect from.
     * {@code null} indicates that the default should be used.
     */
    private final InetAddress localAddress;

    /** The proxy servers, if any. Never null. */
    private final List<HttpHost> proxyChain;

    /** Whether the the route is tunnelled through the proxy. */
    private final TunnelType tunnelled;

    /** Whether the route is layered. */
    private final LayerType layered;

    /** Whether the route is (supposed to be) secure. */
    private final boolean secure;

    //...

}

IdleConnectionEvictor

httpclient-4.5.3-sources.jar!/org/apache/http/impl/client/IdleConnectionEvictor.java

public IdleConnectionEvictor(
            final HttpClientConnectionManager connectionManager,
            final ThreadFactory threadFactory,
            final long sleepTime, final TimeUnit sleepTimeUnit,
            final long maxIdleTime, final TimeUnit maxIdleTimeUnit) {
        this.connectionManager = Args.notNull(connectionManager, "Connection manager");
        this.threadFactory = threadFactory != null ? threadFactory : new DefaultThreadFactory();
        this.sleepTimeMs = sleepTimeUnit != null ? sleepTimeUnit.toMillis(sleepTime) : sleepTime;
        this.maxIdleTimeMs = maxIdleTimeUnit != null ? maxIdleTimeUnit.toMillis(maxIdleTime) : maxIdleTime;
        this.thread = this.threadFactory.newThread(new Runnable() {
            @Override
            public void run() {
                try {
                    while (!Thread.currentThread().isInterrupted()) {
                        Thread.sleep(sleepTimeMs);
                        connectionManager.closeExpiredConnections();
                        if (maxIdleTimeMs > 0) {
                            connectionManager.closeIdleConnections(maxIdleTimeMs, TimeUnit.MILLISECONDS);
                        }
                    }
                } catch (final Exception ex) {
                    exception = ex;
                }

            }
        });
    }

builder默认会构造一个IdleConnectionEvictor httpclient-4.5.3-sources.jar!/org/apache/http/impl/client/HttpClientBuilder.java

if (!this.connManagerShared) {
            if (closeablesCopy == null) {
                closeablesCopy = new ArrayList<Closeable>(1);
            }
            final HttpClientConnectionManager cm = connManagerCopy;

            if (evictExpiredConnections || evictIdleConnections) {
                final IdleConnectionEvictor connectionEvictor = new IdleConnectionEvictor(cm,
                        maxIdleTime > 0 ? maxIdleTime : 10, maxIdleTimeUnit != null ? maxIdleTimeUnit : TimeUnit.SECONDS);
                closeablesCopy.add(new Closeable() {

                    @Override
                    public void close() throws IOException {
                        connectionEvictor.shutdown();
                    }

                });
                connectionEvictor.start();
            }
            closeablesCopy.add(new Closeable() {

                @Override
                public void close() throws IOException {
                    cm.shutdown();
                }

            });
        }

如果没有指定maxIdleTime的话,但是有设置evictExpiredConnections的话,默认是10秒

public final HttpClientBuilder evictExpiredConnections() {
        evictExpiredConnections = true;
        return this;
    }
public final HttpClientBuilder evictIdleConnections(final long maxIdleTime, final TimeUnit maxIdleTimeUnit) {
        this.evictIdleConnections = true;
        this.maxIdleTime = maxIdleTime;
        this.maxIdleTimeUnit = maxIdleTimeUnit;
        return this;
    }

小结

当getKeepAliveDuration为-1以及connTimeToLive为-1的时候,closeExpiredConnections方法其实是没有用的,查看debug日志会出现

o.a.http.impl.client.DefaultHttpClient   509 : Connection can be kept alive indefinitely

当getKeepAliveDuration不为-1的话,假设是5s,则日志可能是这样的

o.a.http.impl.execchain.MainClientExec   285 : Connection can be kept alive for 5000 MILLISECONDS

doc

  • httpcomponents-client-4.5.x-tutorial
  • HttpClient 教程南磊译
  • 使用httpclient必须知道的参数设置及代码写法、存在的风险
  • Apache HttpComponents学习笔记(四):HttpClient里的HttpRoute
  • spring boot封装HttpClient

原文发布于微信公众号 - 码匠的流水账(geek_luandun)

原文发表时间:2017-08-21

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏Java与Android技术栈

基于RxJava2实现的简单图片爬虫

今年十月份以来,跟朋友尝试导入一些图片到tensorflow来生成模型,这就需要大量的图片。刚开始我只写了一个简单的HttpClient程序来抓取图片,后来为了...

16720
来自专栏潇涧技术专栏

Pury Project Analysis

Pury的源码:https://github.com/NikitaKozlov/Pury

9520
来自专栏移动端开发

Android学习--跨程序共享数据之内容提供其探究

      跨程序共享数据之内容提供器,这是个什么功能?看到这个名称的时候最能给我们提供信息的应该是“跨程序”这个词了,是的重点就是这个词,这个内容提供器的作用...

13430
来自专栏函数式编程语言及工具

Akka(43): Http:SSE-Server Sent Event - 服务端主推消息

   因为我了解Akka-http的主要目的不是为了有关Web-Server的编程,而是想实现一套系统集成的api,所以也需要考虑由服务端主动向客户端发送指令的...

27290
来自专栏Android开发指南

Android优化指南

53070
来自专栏向治洪

Universal-Image-Loader源码分析,及常用的缓存策略

讲到图片请求,主要涉及到网络请求,内存缓存,硬盘缓存等原理和4大引用的问题,概括起来主要有以下几个内容: 原理示意图     主体有三个,分别是UI,缓存模...

22290
来自专栏Flutter知识集

Flutter 实践 MVVM

在做Android或iOS开发时,经常会了解到MVC,MVP和MVVM。MVVM在移动端一度被非常推崇,虽然也有不少反对的声音,不过MVVM确实是不错的设计架构...

3.4K50
来自专栏JavaEdge

长文慎入-探索Java并发编程与高并发解决方案(更新中)1 基本概念2 CPU3 项目准备4线程安全性5发布对象7 AQS9 线程池10 死锁

50080
来自专栏我和未来有约会

如何在silverlihgt中使用右键

一般我们在silverlight中点击右键会出现如下的对话筐. ? ? 在flash中 其提供了一个可定制话的右键菜单系统.(ContextMenu) 这个...

21570
来自专栏Golang语言社区

Golang将时间戳转为字符串

在golang里面获取时间戳并不难。只要加载time包。然后time.Now().Unix(),就可以了,但接下来转成string就麻烦了 本来,加载strco...

33080

扫码关注云+社区

领取腾讯云代金券