blocks|key|2162405|text|我相信Smart+Thread+Pool可以做到。它的ThreadPool类是实例化的，因此您应该能够根据需要创建和管理单独的站点特定实例。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|2162406|entityMap|0|LINK|mutability|MUTABLE|url|http://www.codeplex.com/smartthreadpool^0|3|H|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

I believe <a href="http://www.codeplex.com/smartthreadpool" rel="noreferrer">Smart Thread Pool</a> can do this. It's ThreadPool class is instantiated so you should be able to create and manage your separate site specific instances as you require.

blocks|key|248889|text|Ami+bar写了一个很好的可以实例化的Smart线程池。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|248890|看一看here|offset|length|248891|entityMap|0|LINK|mutability|MUTABLE|url|http://www.codeproject.com/KB/threads/smartthreadpool.aspx^0|0|3|4|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|O|8|@]|9|@$D|P|E|Q|1|R]]|A|$]]|$1|F|3|-4|5|6|7|S|8|@]|9|@]|A|$]]]|G|$H|$5|I|J|K|A|$L|M]]]]

Ami bar wrote an excellent Smart thread pool that can be instantiated.

take a look <a href="http://www.codeproject.com/KB/threads/smartthreadpool.aspx" rel="nofollow noreferrer">here</a>

blocks|key|5195820|text|问问乔恩·斯基特：http://www.yoda.arachsys.com/csharp/miscutil/|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|5195821|如果你想要大量并行运行的任务，Parallel+extensions+for+.Net+(TPL)实际上应该工作得更好。|5195822|entityMap|0|LINK|mutability|MUTABLE|url|http://www.yoda.arachsys.com/csharp/miscutil/|1|http://msdn.microsoft.com/en-us/magazine/cc163340.aspx^0|9|19|0|0|F|Y|1|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@$A|Q|B|R|1|S]]|C|$]]|$1|D|3|E|5|6|7|T|8|@]|9|@$A|U|B|V|1|W]]|C|$]]|$1|F|3|-4|5|6|7|X|8|@]|9|@]|C|$]]]|G|$H|$5|I|J|K|C|$L|M]]|N|$5|I|J|K|C|$L|O]]]]

Ask Jon Skeet: <a href="http://www.yoda.arachsys.com/csharp/miscutil/" rel="nofollow noreferrer">http://www.yoda.arachsys.com/csharp/miscutil/</a>

<a href="http://msdn.microsoft.com/en-us/magazine/cc163340.aspx" rel="nofollow noreferrer">Parallel extensions for .Net (TPL)</a> should actually work much better if you want a large number of parallel running tasks.

blocks|key|3082349|text|使用BlockingCollection可以用作线程的队列。下面是它的一个实现。更新于2018-04-23:|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|entityRanges|data|3082350|public+class+WorkerPool<T>+:+IDisposable
{
++++BlockingCollection<T>+queue+=+new+BlockingCollection<T>();
++++List<Task>+taskList;
++++private+CancellationTokenSource+cancellationToken;
++++int+maxWorkers;
++++private+bool+wasShutDown;

++++int+waitingUnits;

++++public+WorkerPool(CancellationTokenSource+cancellationToken,+int+maxWorkers)
++++{
++++++++this.cancellationToken+=+cancellationToken;
++++++++this.maxWorkers+=+maxWorkers;
++++++++this.taskList+=+new+List<Task>();
++++}
++++public+void+enqueue(T+value)
++++{
++++++++queue.Add(value);
++++++++waitingUnits%2B%2B;
++++}
++++//call+to+signal+that+there+are+no+more+item
++++public+void+CompleteAdding()
++++{
++++++++queue.CompleteAdding();++++++++++
++++}

++++//create+workers+and+put+then+running
++++public+void+startWorkers(Action<T>+worker)
++++{
++++++++for+(int+i+=+0;+i+<+maxWorkers;+i%2B%2B)
++++++++{
++++++++++++taskList.Add(new+Task(()+=>
++++++++++++{
++++++++++++++++string+myname+=+"worker+"+%2B+Guid.NewGuid().ToString();

++++++++++++++++try
++++++++++++++++{
++++++++++++++++++++while+(!cancellationToken.IsCancellationRequested)
++++++++++++++++++++{+++++++++++++++++++++
++++++++++++++++++++++++var+value+=+queue.Take();
++++++++++++++++++++++++waitingUnits--;
++++++++++++++++++++++++worker(value);
++++++++++++++++++++}
++++++++++++++++}
++++++++++++++++catch+(Exception+ex)+when+(ex+is+InvalidOperationException)++//throw+when+collection+is+closed+with++CompleteAdding+method.+No+pretty+way+to+do+this.
++++++++++++++++{
++++++++++++++++++++//do+nothing
++++++++++++++++}
++++++++++++}));
++++++++}

++++++++foreach+(var+task+in+taskList)
++++++++{
++++++++++++task.Start();
++++++++}
++++}

++++//wait+for+all+workers+to+be+finish+their+jobs
++++public+void+await()
++++{
++++++++while+(waitingUnits+>0+%7C%7C+!queue.IsAddingCompleted)
++++++++++++Thread.Sleep(100);

++++++++shutdown();
++++}

++++private+void+shutdown()
++++{
++++++++wasShutDown+=+true;
++++++++Task.WaitAll(taskList.ToArray());++++++++++++
++++}

++++//case+something+bad+happen+dismiss+all+pending+work
++++public+void+Dispose()
++++{
++++++++if+(!wasShutDown)
++++++++{
++++++++++++queue.CompleteAdding();
++++++++++++shutdown();
++++++++}
++++}
}|code-block|syntax|javascript|3082351|然后像这样使用：|3082352|WorkerPool<int>+workerPool+=+new+WorkerPool<int>(new+CancellationTokenSource(),+5);

workerPool.startWorkers(value+=>
{
++++log.Debug(value);
});
//enqueue+all+the+work
for+(int+i+=+0;+i+<+100;+i%2B%2B)
{
++++workerPool.enqueue(i);
}
//Signal+no+more+work
workerPool.CompleteAdding();

//wait+all+pending+work+to+finish
workerPool.await();|3082353|只需创建新的WorkPool对象，您就可以拥有尽可能多的投票。|3082354|entityMap^0|14|E|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@$9|T|A|U|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|V|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|W|8|@]|D|@]|E|$]]|$1|M|3|N|5|H|7|X|8|@]|D|@]|E|$I|J]]|$1|O|3|P|5|6|7|Y|8|@]|D|@]|E|$]]|$1|Q|3|-4|5|6|7|Z|8|@]|D|@]|E|$]]]|R|$]]

Using BlockingCollection can be used as a queue for the threads.
Here is an implementation of it.
Updated at 2018-04-23:

<pre><code>public class WorkerPool&lt;T&gt; : IDisposable
{
 BlockingCollection&lt;T&gt; queue = new BlockingCollection&lt;T&gt;();
 List&lt;Task&gt; taskList;
 private CancellationTokenSource cancellationToken;
 int maxWorkers;
 private bool wasShutDown;

 int waitingUnits;

 public WorkerPool(CancellationTokenSource cancellationToken, int maxWorkers)
 {
 this.cancellationToken = cancellationToken;
 this.maxWorkers = maxWorkers;
 this.taskList = new List&lt;Task&gt;();
 }
 public void enqueue(T value)
 {
 queue.Add(value);
 waitingUnits++;
 }
 //call to signal that there are no more item
 public void CompleteAdding()
 {
 queue.CompleteAdding(); 
 }

 //create workers and put then running
 public void startWorkers(Action&lt;T&gt; worker)
 {
 for (int i = 0; i &lt; maxWorkers; i++)
 {
 taskList.Add(new Task(() =&gt;
 {
 string myname = "worker " + Guid.NewGuid().ToString();

 try
 {
 while (!cancellationToken.IsCancellationRequested)
 { 
 var value = queue.Take();
 waitingUnits--;
 worker(value);
 }
 }
 catch (Exception ex) when (ex is InvalidOperationException) //throw when collection is closed with CompleteAdding method. No pretty way to do this.
 {
 //do nothing
 }
 }));
 }

 foreach (var task in taskList)
 {
 task.Start();
 }
 }

 //wait for all workers to be finish their jobs
 public void await()
 {
 while (waitingUnits &gt;0 || !queue.IsAddingCompleted)
 Thread.Sleep(100);

 shutdown();
 }

 private void shutdown()
 {
 wasShutDown = true;
 Task.WaitAll(taskList.ToArray()); 
 }

 //case something bad happen dismiss all pending work
 public void Dispose()
 {
 if (!wasShutDown)
 {
 queue.CompleteAdding();
 shutdown();
 }
 }
}
</code></pre>

Then use like this:

<pre><code>WorkerPool&lt;int&gt; workerPool = new WorkerPool&lt;int&gt;(new CancellationTokenSource(), 5);

workerPool.startWorkers(value =&gt;
{
 log.Debug(value);
});
//enqueue all the work
for (int i = 0; i &lt; 100; i++)
{
 workerPool.enqueue(i);
}
//Signal no more work
workerPool.CompleteAdding();

//wait all pending work to finish
workerPool.await();
</code></pre>

You can have as many polls has you like simply creating new WorkPool objects.

blocks|key|250411|text|这里有一个免费的nuget库：CodeFluentRuntimeClient有一个可以重用的CustomThreadPool类。它非常可配置，您可以更改池线程优先级、数量、COM单元状态、偶数名称(用于调试)以及区域性。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|250412|entityMap|0|LINK|mutability|MUTABLE|url|https://nuget.org/packages/CodeFluentRuntimeClient^0|F|N|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

This free nuget library here: <a href="https://nuget.org/packages/CodeFluentRuntimeClient" rel="nofollow">CodeFluentRuntimeClient</a> has a CustomThreadPool class that you can reuse. It's very configurable, you can change pool threads priority, number, COM apartment state, even name (for debugging), and also culture.

blocks|key|248916|text|另一种方法是使用Dataflow+Pipeline。我添加了这些后来的答案，因为我发现数据流是解决这类问题的一种更好的方法，即拥有多个线程池的问题。它们提供了一种更灵活、更结构化的方法，并且可以轻松地垂直扩展。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|248917|您可以将代码拆分成一个或多个块，然后链接到数据流，然后让数据流引擎根据CPU和内存可用性分配线程|248918|我建议分成3个模块，一个用于准备对网站页面的查询，一个用于访问网站页面，最后一个用于分析数据。这样，慢速块(get)可以分配更多的线程来进行补偿。|248919|下面是数据流设置的样子：|248920|var+linkOptions+=+new+DataflowLinkOptions+{+PropagateCompletion+=+true+};

prepareBlock.LinkTo(get,+linkOptions);
getBlock.LinkTo(analiseBlock,+linkOptions);|code-block|syntax|javascript|248921|数据将从prepareBlock流向getBlock，然后流向analiseBlock。块之间的接口可以是任何类，只要相同即可。请参阅Dataflow+Pipeline上的完整示例|style|CODE|248922|使用数据流的过程如下所示：|248923|+while+...{
++++...
++++prepareBlock.Post(...);+//to+send+data+to+the+pipeline
+}
+prepareBlock.Complete();+//when+done
+analiseBlock.Completion.Wait(cancellationTokenSource.Token);+//to+wait+for+all+queues+to+empty+or+cancel+|248924|entityMap|0|LINK|mutability|MUTABLE|url|https://docs.microsoft.com/en-us/dotnet/standard/parallel-programming/walkthrough-creating-a-dataflow-pipeline|1^0|8|H|0|0|0|0|0|0|4|C|I|8|V|C|1V|H|1|0|0|0^^$0|@$1|2|3|4|5|6|7|15|8|@]|9|@$A|16|B|17|1|18]]|C|$]]|$1|D|3|E|5|6|7|19|8|@]|9|@]|C|$]]|$1|F|3|G|5|6|7|1A|8|@]|9|@]|C|$]]|$1|H|3|I|5|6|7|1B|8|@]|9|@]|C|$]]|$1|J|3|K|5|L|7|1C|8|@]|9|@]|C|$M|N]]|$1|O|3|P|5|6|7|1D|8|@$A|1E|B|1F|Q|R]|$A|1G|B|1H|Q|R]|$A|1I|B|1J|Q|R]]|9|@$A|1K|B|1L|1|1M]]|C|$]]|$1|S|3|T|5|6|7|1N|8|@]|9|@]|C|$]]|$1|U|3|V|5|L|7|1O|8|@]|9|@]|C|$M|N]]|$1|W|3|-4|5|6|7|1P|8|@]|9|@]|C|$]]]|X|$Y|$5|Z|10|11|C|$12|13]]|14|$5|Z|10|11|C|$12|13]]]]

Another approach is to use a <a href="https://docs.microsoft.com/en-us/dotnet/standard/parallel-programming/walkthrough-creating-a-dataflow-pipeline" rel="nofollow noreferrer">Dataflow Pipeline</a>. I added these later answer because i find Dataflows a much better approach for these kind of problem, the problem of having several thread pools. They provide a more flexible and structured approach and can easily scale vertically.

You can broke your code into one or more blocks, link then with Dataflows and let then the Dataflow engine allocate threads according to CPU and memory availability

I suggest to broke into 3 blocks, one for preparing the query to the site page , one access site page, and the last one to Analise the data. 
This way the slow block (get) may have more threads allocated to compensate.

Here how would look like the Dataflow setup:

<pre><code>var linkOptions = new DataflowLinkOptions { PropagateCompletion = true };

prepareBlock.LinkTo(get, linkOptions);
getBlock.LinkTo(analiseBlock, linkOptions);
</code></pre>

Data will flow from <code>prepareBlock</code> to <code>getBlock</code> and then to <code>analiseBlock</code>.
The interfaces between blocks can be any class, just have to bee the same. See the full example on <a href="https://docs.microsoft.com/en-us/dotnet/standard/parallel-programming/walkthrough-creating-a-dataflow-pipeline" rel="nofollow noreferrer">Dataflow Pipeline</a>

Using the Dataflow would be something like this:

<pre><code> while ...{
 ...
 prepareBlock.Post(...); //to send data to the pipeline
 }
 prepareBlock.Complete(); //when done
 analiseBlock.Completion.Wait(cancellationTokenSource.Token); //to wait for all queues to empty or cancel 
</code></pre>

What is the most recommended .NET custom threadpool that can have separate instances i.e more than one threadpool per application? 
I need an unlimited queue size (building a crawler), and need to run a separate threadpool in parallel for each site I am crawling.

Edit : 
I need to mine these sites for information as fast as possible, using a separate threadpool for each site would give me the ability to control the number of threads working on each site at any given time. (no more than 2-3)

Thanks
Roey

.NET Custom Threadpool with separate instances

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

什么是最推荐的.NET自定义线程池，可以有单独的实例，即每个应用程序多个线程池？我需要一个无限的队列大小(建立一个爬虫)，并需要运行我正在爬行的每个网站在并行单独的线程池。编辑:我需要尽可能快地挖掘这些站点的信息，使用每个站点的单独线程池将使我能够控制在任何给定时间在每个站点上工作的线程数量。(不超过2-3个)谢谢罗伊

问具有独立实例的.NET自定义线程池
EN

回答 6

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问具有独立实例的.NET自定义线程池EN

回答 6

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问具有独立实例的.NET自定义线程池
EN