专栏首页码匠的流水账httpclient参数配置

httpclient参数配置

这里简单解释一下httpclient一些关键参数的配置

超时时间

final RequestConfig requestConfig = RequestConfig.custom()
                .setSocketTimeout(SOCKET_TIMEOUT)
                .setConnectTimeout(CONNECTION_TIMEOUT)
                .setCookieSpec(CookieSpecs.IGNORE_COOKIES).build();

这里设置了socket_timeout以及connection_timeout

KeepAliveStrategy

httpclient默认提供了一个策略

httpclient-4.5.3-sources.jar!/org/apache/http/impl/client/DefaultConnectionKeepAliveStrategy.java

@Contract(threading = ThreadingBehavior.IMMUTABLE)
public class DefaultConnectionKeepAliveStrategy implements ConnectionKeepAliveStrategy {

    public static final DefaultConnectionKeepAliveStrategy INSTANCE = new DefaultConnectionKeepAliveStrategy();

    @Override
    public long getKeepAliveDuration(final HttpResponse response, final HttpContext context) {
        Args.notNull(response, "HTTP response");
        final HeaderElementIterator it = new BasicHeaderElementIterator(
                response.headerIterator(HTTP.CONN_KEEP_ALIVE));
        while (it.hasNext()) {
            final HeaderElement he = it.nextElement();
            final String param = he.getName();
            final String value = he.getValue();
            if (value != null && param.equalsIgnoreCase("timeout")) {
                try {
                    return Long.parseLong(value) * 1000;
                } catch(final NumberFormatException ignore) {
                }
            }
        }
        return -1;
    }

}

默认的话,是从response里头读timeout参数的,没有读到则设置为-1,这个代表无穷,这样设置是有点问题了,如果是https链接的话,则可能会经常报

Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:170)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
    at sun.security.ssl.InputRecord.read(InputRecord.java:503)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)

HTTP规范没有确定一个持久连接可能或应该保持活动多长时间。一些HTTP服务器使用非标准的头部信息Keep-Alive来告诉客户端它们想在服务器端保持连接活动的周期秒数。如果这个信息可用,HttClient就会利用这个它。如果头部信息Keep-Alive在响应中不存在,HttpClient假设连接无限期的保持活动。然而许多现实中的HTTP服务器配置了在特定不活动周期之后丢掉持久连接来保存系统资源,往往这是不通知客户端的。

这里可以在此基础上重写一个,这里设置为5秒

ConnectionKeepAliveStrategy keepAliveStrategy = new DefaultConnectionKeepAliveStrategy() {
            @Override
            public long getKeepAliveDuration(final HttpResponse response, final HttpContext context) {
                long keepAlive = super.getKeepAliveDuration(response, context);
                if (keepAlive == -1) {
                    keepAlive = 5000;
                }
                return keepAlive;
            }
        };

Connection eviction policy

一个经典的阻塞 I/O 模型的主要缺点是网络套接字仅当 I/O 操作阻塞时才可以响应 I/O 事件。当一个连接被释放返回管理器时,它可以被保持活动状态而却不能监控套接字的状态和响应任何 I/O 事件。如果连接在服务器端关闭,那么客户端连接也不能去侦测连接状态中的变化和关闭本端的套接字去作出适当响应。 HttpClient 通过测试连接是否是过时的来尝试去减轻这个问题,这已经不再有效了,因为它已经在服务器端关闭了,之前使用执行 HTTP 请求的连接。过时的连接检查也并不是 100% 的稳定,反而对每次请求执行还要增加10到30毫秒的开销。唯一可行的而不涉及到每个对空闲连接的套接字模型线程解决方案,是使用专用的监控线程来收回因为长时间不活动而被认为是过期的连接。监控线程可以周期地调用 ClientConnectionManager#closeExpiredConnections()方法来关闭所有过期的连接,从连接池中收回关闭的连接。它也可以选择性调用ClientConnectionManager#closeIdleConnections()方法来关闭所有已经空 闲超过给定时间周期的连接。

官方提供了一个实例

public static class IdleConnectionMonitorThread extends Thread {

    private final HttpClientConnectionManager connMgr;
    private volatile boolean shutdown;

    public IdleConnectionMonitorThread(HttpClientConnectionManager connMgr) {
        super();
        this.connMgr = connMgr;
    }

    @Override
    public void run() {
        try {
            while (!shutdown) {
                synchronized (this) {
                    wait(5000);
                    // Close expired connections
                    connMgr.closeExpiredConnections();
                    // Optionally, close connections
                    // that have been idle longer than 30 sec
                    connMgr.closeIdleConnections(30, TimeUnit.SECONDS);
                }
            }
        } catch (InterruptedException ex) {
            // terminate
        }
    }

    public void shutdown() {
        shutdown = true;
        synchronized (this) {
            notifyAll();
        }
    }

}

在spring cloud netflix zuul中则提供了类似的定时器,只不过参数是写死的 spring-cloud-netflix-core/1.2.6.RELEASE/spring-cloud-netflix-core-1.2.6.RELEASE-sources.jar!/org/springframework/cloud/netflix/zuul/filters/route/SimpleHostRoutingFilter.java

private final Timer connectionManagerTimer = new Timer(
            "SimpleHostRoutingFilter.connectionManagerTimer", true);

@PostConstruct
    private void initialize() {
        this.httpClient = newClient();
        SOCKET_TIMEOUT.addCallback(this.clientloader);
        CONNECTION_TIMEOUT.addCallback(this.clientloader);
        this.connectionManagerTimer.schedule(new TimerTask() {
            @Override
            public void run() {
                if (SimpleHostRoutingFilter.this.connectionManager == null) {
                    return;
                }
                SimpleHostRoutingFilter.this.connectionManager.closeExpiredConnections();
            }
        }, 30000, 5000);
    }

每30秒清理一次失效的connection,但是这里的closeExpiredConnections是依赖connTimeToLive参数以及keep alive strategy设置的。 httpcore-4.4.6-sources.jar!/org/apache/http/pool/AbstractConnPool.java

/**
     * Closes expired connections and evicts them from the pool.
     */
    public void closeExpired() {
        final long now = System.currentTimeMillis();
        enumAvailable(new PoolEntryCallback<T, C>() {

            @Override
            public void process(final PoolEntry<T, C> entry) {
                if (entry.isExpired(now)) {
                    entry.close();
                }
            }

        });
    }

httpcore-4.4.6-sources.jar!/org/apache/http/pool/PoolEntry.java

public synchronized boolean isExpired(final long now) {
        return now >= this.expiry;
    }
public PoolEntry(final String id, final T route, final C conn,
            final long timeToLive, final TimeUnit tunit) {
        super();
        Args.notNull(route, "Route");
        Args.notNull(conn, "Connection");
        Args.notNull(tunit, "Time unit");
        this.id = id;
        this.route = route;
        this.conn = conn;
        this.created = System.currentTimeMillis();
        this.updated = this.created;
        if (timeToLive > 0) {
            this.validityDeadline = this.created + tunit.toMillis(timeToLive);
        } else {
            this.validityDeadline = Long.MAX_VALUE;
        }
        this.expiry = this.validityDeadline;
    }
public synchronized void updateExpiry(final long time, final TimeUnit tunit) {
        Args.notNull(tunit, "Time unit");
        this.updated = System.currentTimeMillis();
        final long newExpiry;
        if (time > 0) {
            newExpiry = this.updated + tunit.toMillis(time);
        } else {
            newExpiry = Long.MAX_VALUE;
        }
        this.expiry = Math.min(newExpiry, this.validityDeadline);
    }

如果是-1,则这里expiry为无穷大。那么closeExpiredConnections其实是无效的。但是还依赖一个keep alive strategy,它会去设置updateExpiry httpclient-4.5.3-sources.jar!/org/apache/http/impl/execchain/MainClientExec.java 主要看httpclient-4.5.3-sources.jar!/org/apache/http/impl/execchain/ConnectionHolder.java

public void setValidFor(final long duration, final TimeUnit tunit) {
        synchronized (this.managedConn) {
            this.validDuration = duration;
            this.tunit = tunit;
        }
    }

    private void releaseConnection(final boolean reusable) {
        if (this.released.compareAndSet(false, true)) {
            synchronized (this.managedConn) {
                if (reusable) {
                    this.manager.releaseConnection(this.managedConn,
                            this.state, this.validDuration, this.tunit);
                } else {
                    try {
                        this.managedConn.close();
                        log.debug("Connection discarded");
                    } catch (final IOException ex) {
                        if (this.log.isDebugEnabled()) {
                            this.log.debug(ex.getMessage(), ex);
                        }
                    } finally {
                        this.manager.releaseConnection(
                                this.managedConn, null, 0, TimeUnit.MILLISECONDS);
                    }
                }
            }
        }
    }

这里设置了validDuration,会传给releaseConnection httpclient-4.5.3-sources.jar!/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.java

public void releaseConnection(
            final HttpClientConnection managedConn,
            final Object state,
            final long keepalive, final TimeUnit tunit) {
        Args.notNull(managedConn, "Managed connection");
        synchronized (managedConn) {
            final CPoolEntry entry = CPoolProxy.detach(managedConn);
            if (entry == null) {
                return;
            }
            final ManagedHttpClientConnection conn = entry.getConnection();
            try {
                if (conn.isOpen()) {
                    final TimeUnit effectiveUnit = tunit != null ? tunit : TimeUnit.MILLISECONDS;
                    entry.setState(state);
                    entry.updateExpiry(keepalive, effectiveUnit);
                    if (this.log.isDebugEnabled()) {
                        final String s;
                        if (keepalive > 0) {
                            s = "for " + (double) effectiveUnit.toMillis(keepalive) / 1000 + " seconds";
                        } else {
                            s = "indefinitely";
                        }
                        this.log.debug("Connection " + format(entry) + " can be kept alive " + s);
                    }
                }
            } finally {
                this.pool.release(entry, conn.isOpen() && entry.isRouteComplete());
                if (this.log.isDebugEnabled()) {
                    this.log.debug("Connection released: " + format(entry) + formatStats(entry.getRoute()));
                }
            }
        }
    }

这里去updateExpiry,相当于更新了timeToLive

connTimeToLive

这里设置的是http连接池中connection的存活时间 httpclient-4.5.3-sources.jar!/org/apache/http/impl/client/HttpClientBuilder.java

/**
     * Sets maximum time to live for persistent connections
     * <p>
     * Please note this value can be overridden by the {@link #setConnectionManager(
     *   org.apache.http.conn.HttpClientConnectionManager)} method.
     * </p>
     *
     * @since 4.4
     */
    public final HttpClientBuilder setConnectionTimeToLive(final long connTimeToLive, final TimeUnit connTimeToLiveTimeUnit) {
        this.connTimeToLive = connTimeToLive;
        this.connTimeToLiveTimeUnit = connTimeToLiveTimeUnit;
        return this;
    }

httpclient-4.5.3-sources.jar!/org/apache/http/impl/conn/PoolingHttpClientConnectionManager.java

public PoolingHttpClientConnectionManager(
            final Registry<ConnectionSocketFactory> socketFactoryRegistry,
            final HttpConnectionFactory<HttpRoute, ManagedHttpClientConnection> connFactory,
            final DnsResolver dnsResolver) {
        this(socketFactoryRegistry, connFactory, null, dnsResolver, -1, TimeUnit.MILLISECONDS);
    }
public PoolingHttpClientConnectionManager(
            final Registry<ConnectionSocketFactory> socketFactoryRegistry,
            final HttpConnectionFactory<HttpRoute, ManagedHttpClientConnection> connFactory,
            final SchemePortResolver schemePortResolver,
            final DnsResolver dnsResolver,
            final long timeToLive, final TimeUnit tunit) {
        this(
            new DefaultHttpClientConnectionOperator(socketFactoryRegistry, schemePortResolver, dnsResolver),
            connFactory,
            timeToLive, tunit
        );
    }

默认构造的话,timeToLive是-1,如果默认是-1的话,则不失效

maxTotal 以及 defaultMaxPerRoute

4.0的ThreadSafeClientConnManager,在4.2版本的时候被废弃了,默认使用PoolingHttpClientConnectionManager。这个manager有两个重要的参数,一个是maxTotal,一个是defaultMaxPerRoute。

每个默认的实现对每个给定路由将会创建不超过2个的并发连接,而总共也不会超过 20 个连接。

 this.pool = new CPool(new InternalConnectionFactory(this.configData, connFactory), 2, 20, timeToLive, tunit);

对于很多真实的应用程序,这个限制也证明很大的制约,特别是他们在服务中使用 HTTP 作为 传输协议。连接限制,也可以使用 HTTP 参数来进行调整。 spring cloud netflix zuul 里头默认配置是总共200连接,每个route不超过20个连接

this.connectionManager = new PoolingHttpClientConnectionManager(registry);
            this.connectionManager.setMaxTotal(this.hostProperties.getMaxTotalConnections()); // 200
            this.connectionManager.setDefaultMaxPerRoute(this.hostProperties.getMaxPerRouteConnections()); //20

HttpRoute

httpclient-4.5.3-sources.jar!/org/apache/http/conn/routing/HttpRoute.java HttpRoute对象是immutable的,包含的数据有目标主机、本地地址、代理链、是否tunnulled、是否layered、是否是安全路由。

@Contract(threading = ThreadingBehavior.IMMUTABLE)
public final class HttpRoute implements RouteInfo, Cloneable {

    /** The target host to connect to. */
    private final HttpHost targetHost;

    /**
     * The local address to connect from.
     * {@code null} indicates that the default should be used.
     */
    private final InetAddress localAddress;

    /** The proxy servers, if any. Never null. */
    private final List<HttpHost> proxyChain;

    /** Whether the the route is tunnelled through the proxy. */
    private final TunnelType tunnelled;

    /** Whether the route is layered. */
    private final LayerType layered;

    /** Whether the route is (supposed to be) secure. */
    private final boolean secure;

    //...

}

IdleConnectionEvictor

httpclient-4.5.3-sources.jar!/org/apache/http/impl/client/IdleConnectionEvictor.java

public IdleConnectionEvictor(
            final HttpClientConnectionManager connectionManager,
            final ThreadFactory threadFactory,
            final long sleepTime, final TimeUnit sleepTimeUnit,
            final long maxIdleTime, final TimeUnit maxIdleTimeUnit) {
        this.connectionManager = Args.notNull(connectionManager, "Connection manager");
        this.threadFactory = threadFactory != null ? threadFactory : new DefaultThreadFactory();
        this.sleepTimeMs = sleepTimeUnit != null ? sleepTimeUnit.toMillis(sleepTime) : sleepTime;
        this.maxIdleTimeMs = maxIdleTimeUnit != null ? maxIdleTimeUnit.toMillis(maxIdleTime) : maxIdleTime;
        this.thread = this.threadFactory.newThread(new Runnable() {
            @Override
            public void run() {
                try {
                    while (!Thread.currentThread().isInterrupted()) {
                        Thread.sleep(sleepTimeMs);
                        connectionManager.closeExpiredConnections();
                        if (maxIdleTimeMs > 0) {
                            connectionManager.closeIdleConnections(maxIdleTimeMs, TimeUnit.MILLISECONDS);
                        }
                    }
                } catch (final Exception ex) {
                    exception = ex;
                }

            }
        });
    }

builder默认会构造一个IdleConnectionEvictor httpclient-4.5.3-sources.jar!/org/apache/http/impl/client/HttpClientBuilder.java

if (!this.connManagerShared) {
            if (closeablesCopy == null) {
                closeablesCopy = new ArrayList<Closeable>(1);
            }
            final HttpClientConnectionManager cm = connManagerCopy;

            if (evictExpiredConnections || evictIdleConnections) {
                final IdleConnectionEvictor connectionEvictor = new IdleConnectionEvictor(cm,
                        maxIdleTime > 0 ? maxIdleTime : 10, maxIdleTimeUnit != null ? maxIdleTimeUnit : TimeUnit.SECONDS);
                closeablesCopy.add(new Closeable() {

                    @Override
                    public void close() throws IOException {
                        connectionEvictor.shutdown();
                    }

                });
                connectionEvictor.start();
            }
            closeablesCopy.add(new Closeable() {

                @Override
                public void close() throws IOException {
                    cm.shutdown();
                }

            });
        }

如果没有指定maxIdleTime的话,但是有设置evictExpiredConnections的话,默认是10秒

public final HttpClientBuilder evictExpiredConnections() {
        evictExpiredConnections = true;
        return this;
    }
public final HttpClientBuilder evictIdleConnections(final long maxIdleTime, final TimeUnit maxIdleTimeUnit) {
        this.evictIdleConnections = true;
        this.maxIdleTime = maxIdleTime;
        this.maxIdleTimeUnit = maxIdleTimeUnit;
        return this;
    }

小结

当getKeepAliveDuration为-1以及connTimeToLive为-1的时候,closeExpiredConnections方法其实是没有用的,查看debug日志会出现

o.a.http.impl.client.DefaultHttpClient   509 : Connection can be kept alive indefinitely

当getKeepAliveDuration不为-1的话,假设是5s,则日志可能是这样的

o.a.http.impl.execchain.MainClientExec   285 : Connection can be kept alive for 5000 MILLISECONDS

doc

  • httpcomponents-client-4.5.x-tutorial
  • HttpClient 教程南磊译
  • 使用httpclient必须知道的参数设置及代码写法、存在的风险
  • Apache HttpComponents学习笔记(四):HttpClient里的HttpRoute
  • spring boot封装HttpClient

本文分享自微信公众号 - 码匠的流水账(geek_luandun)

原文出处及转载信息见文内详细说明,如有侵权,请联系 yunjia_community@tencent.com 删除。

原始发表时间:2017-08-21

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

我来说两句

0 条评论
登录 后参与评论

相关文章

  • 聊聊flink KeyedStream的intervalJoin操作

    flink-streaming-java_2.11-1.7.0-sources.jar!/org/apache/flink/streaming/api/data...

    codecraft
  • 聊聊flink KeyedStream的intervalJoin操作

    flink-streaming-java_2.11-1.7.0-sources.jar!/org/apache/flink/streaming/api/data...

    codecraft
  • 聊聊spring cloud consul的TtlScheduler

    spring-cloud-consul-discovery-2.1.2.RELEASE-sources.jar!/org/springframework/clo...

    codecraft
  • 聊聊spring cloud consul的TtlScheduler

    spring-cloud-consul-discovery-2.1.2.RELEASE-sources.jar!/org/springframework/clo...

    codecraft
  • [第27 期]彻底分清Javascript forEach & map

    JavaScript 中,数组的遍历我们肯定都不陌生,最常见的两个便是forEach 和 map。

    用户6900878
  • 使用这个,自定义AlertDialog在你手里都不是问题

    Xiaolei123
  • Spring5.0源码深度解析之SpringBean的Aop通知调用链源码分析

    这里就是那五个通知后面还有个TransactionInterceptor,后面会分析它

    须臾之余
  • 使命必达: 深入剖析WCF的可靠会话[实例篇](内含美女图片,定力差者慎入)

    通过前面一系列的博文(《WCF 并发(Concurrency)的本质》、《并发中的同步》、《实践重于理论》、《并发与实例上下文模式》、《回调与并发》、《Conc...

    蒋金楠
  • Flutter基础widgets教程-Chip篇

    青年码农

扫码关注云+社区

领取腾讯云代金券