Windows上使用Python的并发性/并行性
Windows上使用Python的并发性/并行性
提问于 2018-09-10 00:37:59
我开发了一个简单的程序来解决八皇后问题。现在我想用不同的元参数做更多的测试，所以我想让它变得更快。我经历了几次分析迭代，能够显着缩短运行时间，但我相信只有部分计算同时进行才能使其更快。我尝试使用multiprocessing和concurrent.futures模块，但它并没有大大改善运行时，在某些情况下甚至减慢了执行速度。这只是给出一些背景。
我能够想出类似的代码结构，其中顺序版本优于并发版本。
import numpy as np
import concurrent.futures
import math
import time
import multiprocessing
def is_prime(n):
if n % 2 == 0:
return False
sqrt_n = int(math.floor(math.sqrt(n)))
for i in range(3, sqrt_n + 1, 2):
if n % i == 0:
return False
return True
def generate_data(seed):
np.random.seed(seed)
numbers = []
for _ in range(5000):
nbr = np.random.randint(50000, 100000)
numbers.append(nbr)
return numbers
def run_test_concurrent(numbers):
print("Concurrent test")
start_tm = time.time()
chunk = len(numbers)//3
primes = None
with concurrent.futures.ProcessPoolExecutor(max_workers=3) as pool:
primes = list(pool.map(is_prime, numbers, chunksize=chunk))
print("Time: {:.6f}".format(time.time() - start_tm))
print("Number of primes: {}\n".format(np.sum(primes)))
def run_test_sequential(numbers):
print("Sequential test")
start_tm = time.time()
primes = [is_prime(nbr) for nbr in numbers]
print("Time: {:.6f}".format(time.time() - start_tm))
print("Number of primes: {}\n".format(np.sum(primes)))
def run_test_multiprocessing(numbers):
print("Multiprocessing test")
start_tm = time.time()
chunk = len(numbers)//3
primes = None
with multiprocessing.Pool(processes=3) as pool:
primes = list(pool.map(is_prime, numbers, chunksize=chunk))
print("Time: {:.6f}".format(time.time() - start_tm))
print("Number of primes: {}\n".format(np.sum(primes)))
def main():
nbr_trails = 5
for trail in range(nbr_trails):
numbers = generate_data(trail*10)
run_test_concurrent(numbers)
run_test_sequential(numbers)
run_test_multiprocessing(numbers)
print("--\n")
if __name__ == '__main__':
main()当我在我的机器上运行它时-Windows7，四核英特尔酷睿i5，我得到了以下输出：
Concurrent test
Time: 2.006006
Number of primes: 431
Sequential test
Time: 0.010000
Number of primes: 431
Multiprocessing test
Time: 1.412003
Number of primes: 431
--
Concurrent test
Time: 1.302003
Number of primes: 447
Sequential test
Time: 0.010000
Number of primes: 447
Multiprocessing test
Time: 1.252003
Number of primes: 447
--
Concurrent test
Time: 1.280002
Number of primes: 446
Sequential test
Time: 0.010000
Number of primes: 446
Multiprocessing test
Time: 1.250002
Number of primes: 446
--
Concurrent test
Time: 1.260002
Number of primes: 446
Sequential test
Time: 0.010000
Number of primes: 446
Multiprocessing test
Time: 1.250002
Number of primes: 446
--
Concurrent test
Time: 1.282003
Number of primes: 473
Sequential test
Time: 0.010000
Number of primes: 473
Multiprocessing test
Time: 1.260002
Number of primes: 473
--我的问题是，我是否可以通过在Windows上与Python 3.6.4 |Anaconda, Inc.|同时运行它来提高速度。我在SO (Why is creating a new process more expensive on Windows than Linux?)上读到，在Windows上创建新进程代价很高。有什么可以做的来加快速度吗？我是不是漏掉了什么明显的东西？
我也只尝试创建了一次Pool，但似乎没有多大帮助。
编辑：
原始代码结构或多或少如下所示：
我的代码结构大体上是这样的：
class Foo(object):
def g() -> int:
# function performing simple calculations
# single function call is fast (~500 ms)
pass
def run(self):
nbr_processes = multiprocessing.cpu_count() - 1
with multiprocessing.Pool(processes=nbr_processes) as pool:
foos = get_initial_foos()
solution_found = False
while not solution_found:
# one iteration
chunk = len(foos)//nbr_processes
vals = list(pool.map(Foo.g, foos, chunksize=chunk))
foos = modify_foos()其中foos具有1000元素。不可能提前知道算法收敛的速度以及执行了多少次迭代，可能是数千次。
回答 1
Stack Overflow用户
发布于 2018-09-10 00:51:54
在UNIX变体中，进程要轻量级得多。Windows进程很繁重，而且启动时间要长得多。线程是在windows上执行多处理的推荐方式。你也可以关注这个主题：Why is creating a new process more expensive on Windows than Linux?
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/52246574
复制相关文章
相似问题