python cpu密集运算为何要使用多进程?

  • 回答 (0)
  • 关注 (0)
  • 查看 (13)

由于GIL的存在python 的多线程在cpu密集情况下基本没用,反而效率降低。但是为什么使用多进程可以有效利用处理器的多个核心?

多进程时每个进程不也是分时占用cpu的吗?多进程时每个进程只有一个主线程,也只占用处理器一个核心,但是为什么代码测试多进程可以占满所有cpu核心呢?

代码:

#!python3
import time
import os, sys, getopt
from threading import Thread
from multiprocessing import Process
import logging, coloredlogs
logger = logging.getLogger('TIMER')
formatter = logging.Formatter('%(asctime)s %(msecs)03d : %(message)s')
logging.basicConfig(level=logging.DEBUG,
                        format='%(asctime)s %(filename)s[line:%(lineno)d] %(levelname)s %(message)s',
                        #  datefmt='%a, %d %b %Y %H:%M:%S'
                    )
ProcessNum = 1 
def loop():
    start =time.process_time()
    mr = 300000000
    pr = 0
    for i in range(0,mr):
        pr = i
    end = time.process_time()
    logger.warning('pid {} coast time: {}'.format(os.getpid(),str(end-start)[:5] ))


def muti_process():
    for i in range(ProcessNum):
        t = Process(target=loop)
        t.start()
        logger.warning('start.... muti_process')

def muti_threads():
    for i in range(1):
        t = Thread(target=loop)
        t.start()
        logger.warning('start.... muti_threads')

if __name__ == '__main__':
    muti_process()
    #  muti_threads()

修改ProcessNum = 1,2,3,4 测试得到如下结果:

ProcessNum = 1

2019-05-11 21:18:03,469 process.py[line:29] WARNING start.... muti_process
2019-05-11 21:18:14,419 process.py[line:22] WARNING pid 3849 coast time: 10.89

ProcessNum = 2

2019-05-11 21:18:39,443 process.py[line:29] WARNING start.... muti_process
2019-05-11 21:18:39,445 process.py[line:29] WARNING start.... muti_process
2019-05-11 21:18:50,638 process.py[line:22] WARNING pid 3856 coast time: 11.14
2019-05-11 21:18:50,644 process.py[line:22] WARNING pid 3857 coast time: 11.15

ProcessNum =3

2019-05-11 21:19:01,319 process.py[line:29] WARNING start.... muti_process
2019-05-11 21:19:01,321 process.py[line:29] WARNING start.... muti_process
2019-05-11 21:19:01,324 process.py[line:29] WARNING start.... muti_process
2019-05-11 21:19:17,286 process.py[line:22] WARNING pid 3864 coast time: 15.61
2019-05-11 21:19:17,415 process.py[line:22] WARNING pid 3863 coast time: 15.78
2019-05-11 21:19:17,466 process.py[line:22] WARNING pid 3862 coast time: 15.82

ProcessNum =4

2019-05-11 21:19:28,140 process.py[line:29] WARNING start.... muti_process
2019-05-11 21:19:28,143 process.py[line:29] WARNING start.... muti_process
2019-05-11 21:19:28,147 process.py[line:29] WARNING start.... muti_process
2019-05-11 21:19:28,157 process.py[line:29] WARNING start.... muti_process
2019-05-11 21:19:48,927 process.py[line:22] WARNING pid 3867 coast time: 19.68
2019-05-11 21:19:49,049 process.py[line:22] WARNING pid 3870 coast time: 19.68
2019-05-11 21:19:49,085 process.py[line:22] WARNING pid 3869 coast time: 19.65
2019-05-11 21:19:49,092 process.py[line:22] WARNING pid 3868 coast time: 19.64

代码运行环境,操作系统: osx Mojave ,CPU :2.7G core i5(物理双核 支持超线程), python: Python 3.7.1

当 ProcessNum = 1 时可以看到 运行loop中的代码一个进程需要几乎11秒的时间。

当 ProcessNum = 2 时, 每个进程运行的时间也为11秒多一点,总时间也只花了大约11秒,也就是说 2个进程应该是并行运行的??

问题就是这里的2个进程可以同时运行在同一个处理器的不同核心上?

当 ProcessNum = 3 时总时间为 15.7秒 ,这相当于 3个进程运行在2个核心上,每个进程花了10秒左右。

当 ProcessNum = 4 时总时间为 19.6秒 ,这相当于 4个进程运行在2个核心上,每个进程花了10秒左右。

为何会产生这样的测试结果?

用户5380430用户5380430提问于

可能回答问题的人

  • 腾讯游戏云

    腾讯 · 产品团队 (已认证)

    309 粉丝1 提问9 回答
  • Hyman Wang

    腾讯云 · 高级产品经理 (已认证)

    5 粉丝0 提问16 回答
  • stephenqi

    腾讯云 · 资深产品经理 (已认证)

    2 粉丝0 提问1 回答
  • 樱桃子

    腾讯云 · 产品经理 (已认证)

    2 粉丝2 提问1 回答
  • Harker

    0 粉丝0 提问11 回答

扫码关注云+社区

领取腾讯云代金券