ServiceStack IOException:使用Redis C#客户端“现有连接被远程主机强制关闭”
ServiceStack IOException:使用Redis C#客户端“现有连接被远程主机强制关闭”
提问于 2012-10-30 03:32:45
我们有以下设置：
RackspaceCloud 8 8GB实例上Ubuntu Linux 12.04LTE上的Redis 2.6，设置如下：
daemonize yes
pidfile /var/run/redis_6379.pid
port 6379
timeout 300
loglevel notice
logfile /var/log/redis_6379.log
databases 16
save 900 1
save 300 10
save 60 10000
rdbcompression yes
dbfilename dump.rdb
dir /var/redis/6379
requirepass PASSWORD
maxclients 10000
maxmemory 7gb
maxmemory-policy allkeys-lru
maxmemory-samples 3
appendonly no
slowlog-log-slower-than 10000
slowlog-max-len 128
activerehashing yes我们的应用服务器托管在RackSpace托管中，并通过公网IP连接到Redis (为了避免设置RackSpace连接，这是皇家皮塔)，我们通过要求Redis连接密码来提供一些安全性。我手动将unix文件描述符限制增加到10240，10k的最大连接数应该可以提供足够的余量。正如您从上面的设置文件中看到的，我将内存使用量限制在7 7GB，以留出一些RAM余量。
我们使用ServiceStack C# Redis驱动程序。我们使用以下web.config设置：
<RedisConfig suffix="">
<Primary password="PASSWORD" host="HOST" port="6379" maxReadPoolSize="50" maxWritePoolSize="50"/>
</RedisConfig> 我们有一个PooledRedisClientManager单例，每个AppPool创建一次，如下所示：
private static PooledRedisClientManager _clientManager;
public static PooledRedisClientManager ClientManager
{
get
{
if (_clientManager == null)
{
try
{
var poolConfig = new RedisClientManagerConfig
{
MaxReadPoolSize = RedisConfig.Config.Primary.MaxReadPoolSize,
MaxWritePoolSize = RedisConfig.Config.Primary.MaxWritePoolSize,
};
_clientManager = new PooledRedisClientManager(new List<string>() { RedisConfig.Config.Primary.ToHost() }, null, poolConfig);
}
catch (Exception e)
{
log.Fatal("Could not spin up Redis", e);
CacheFailed = DateTime.Now;
}
}
return _clientManager;
}
}我们获取一个连接并执行如下的put/get操作：
using (var client = ClientManager.GetClient())
{
client.Set<T>(region + key, value);
}代码似乎大部分都是有效的。假设我们有大约20个AppPools，50-100个读客户端和50-100个写客户端，我们预计最多有2000-4000个连接到Redis服务器。然而，我们总是在错误日志中看到以下异常，通常是几百个这样的异常聚集在一起，一个小时内什么都没有，然后一次又一次，临时的。
System.IO.IOException: Unable to read data from the transport connection:
An existing connection was forcibly closed by the remote host.
---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at
System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags) at
System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace
- at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) at System.IO.BufferedStream.ReadByte() at
ServiceStack.Redis.RedisNativeClient.ReadLine() in C:\src\ServiceStack.Redis\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 85 at
ServiceStack.Redis.RedisNativeClient.SendExpectData(Byte[][] cmdWithBinaryArgs) in C:\src\ServiceStack.Redis\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 355 at
ServiceStack.Redis.RedisNativeClient.GetBytes(String key) in C:\src\ServiceStack.Redis\src\ServiceStack.Redis\RedisNativeClient.cs:line 404 at ServiceStack.Redis.RedisClient.GetValue(String key) in C:\src\ServiceStack.Redis\src\ServiceStack.Redis\RedisClient.cs:line 185 at ServiceStack.Redis.RedisClient.Get[T](String key) in C:\src\ServiceStack.Redis\src\ServiceStack.Redis\RedisClient.ICacheClient.cs:line 32 at DataPeaks.NoSQL.RedisCacheClient.Get[T](String key) in c:\dev\base\branches\currentversion\DataPeaks\DataPeaks.NoSQL\RedisCacheClient.cs:line 96我们已经尝试了Redis Server超时为0(即无连接超时)，超时为24小时，以及介于两者之间，但没有运气。Googling和Stackoverflowing并没有带来真正的答案，一切似乎都表明我们至少在代码上做了正确的事情。
我们的感觉是，我们经常会遇到持续的网络延迟问题，而不是Rackspace Hosted和Rackspace Cloud，这会导致TCP连接变得陈旧。我们可以通过实现客户端连接超时来解决这个问题，问题是我们是否也需要服务器端超时。但这只是一种感觉，我们不能100%确定我们在正确的轨道上。
想法？
编辑:我偶尔也会看到以下错误：
ServiceStack.Redis.RedisException: Unable to Connect: sPort: 65025 ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.Send(IList`1 buffers, SocketFlags socketFlags) at ServiceStack.Redis.RedisNativeClient.FlushSendBuffer() in C:\src\ServiceStack.Redis\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 273 at ServiceStack.Redis.RedisNativeClient.SendCommand(Byte[][] cmdWithBinaryArgs) in C:\src\ServiceStack.Redis\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 203 --- End of inner exception stack trace --- at ServiceStack.Redis.RedisNativeClient.CreateConnectionError() in C:\src\ServiceStack.Redis\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 165 at ServiceStack.Redis.RedisNativeClient.SendExpectData(Byte[][] cmdWithBinaryArgs) in C:\src\ServiceStack.Redis\src\ServiceStack.Redis\RedisNativeClient_Utils.cs:line 355 at ServiceStack.Redis.RedisNativeClient.GetBytes(String key) in C:\src\ServiceStack.Redis\src\ServiceStack.Redis\RedisNativeClient.cs:line 404 at ServiceStack.Redis.RedisClient.GetValue(String key) in C:\src\ServiceStack.Redis\src\ServiceStack.Redis\RedisClient.cs:line 185 at ServiceStack.Redis.RedisClient.Get[T](String key) in C:\src\ServiceStack.Redis\src\ServiceStack.Redis\RedisClient.ICacheClient.cs:line 32 at DataPeaks.NoSQL.RedisCacheClient.Get[T](String key) in c:\dev\base\branches\currentversion\DataPeaks\DataPeaks.NoSQL\RedisCacheClient.cs:line 96我认为这是没有在客户端处理的服务器端连接超时的直接结果。看起来我们真的需要处理客户端连接超时。
回答 2
Stack Overflow用户
回答已采纳
发布于 2013-11-20 23:47:19
我们认为，在仔细阅读Redis文档并找到以下好处后，我们找到了根本原因(http://redis.io/topics/persistence)：
RDB needs to fork() often in order to persist on disk using a child process.
Fork() can be time consuming if the dataset is big, and may result in Redis
to stop serving clients for some millisecond or even for one second if the
dataset is very big and the CPU performance not great. AOF also needs to fork()
but you can tune how often you want to rewrite your logs without any trade-off
on durability.我们关闭了RDB持久性，从那以后就再也没有看到过连接中断。
Stack Overflow用户
发布于 2012-11-01 23:12:57
似乎将服务器超时从0设置为300可以缓解连接失败的问题。仍然看到一些糟糕的连接，但这可能是因为PooledRedisClientManager没有正确检查从GetClient()调用的GetInActiveWriteClient ()的连接状态。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/13128363
复制相关文章
相似问题