blocks|key|3258655|text|我上次使用Python已经有很长一段时间了，但我认为问题出在语句for+line+in+proc.stdout上，该语句在迭代之前读取整个输入。解决方案是改用readline()：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|3258656|#filters+output
import+subprocess
proc+=+subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
while+True:
++line+=+proc.stdout.readline()
++if+not+line:
++++break
++#the+real+code+does+filtering+here
++print+"test:",+line.rstrip()|code-block|syntax|javascript|3258657|当然，您仍然需要处理子进程的缓冲。|3258658|注意:除了预读缓冲区之外，使用迭代器的解决方案应该等同于使用readline()，但是(或者正是因为这个原因)提议的更改确实对我产生了不同的结果(+Windows+XP上的Python2.5)。|3258659|entityMap^0|W|N|27|A|0|0|0|U|A|0^^$0|@$1|2|3|4|5|6|7|Q|8|@$9|R|A|S|B|C]|$9|T|A|U|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|V|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|W|8|@]|D|@]|E|$]]|$1|M|3|N|5|6|7|X|8|@$9|Y|A|Z|B|C]]|D|@]|E|$]]|$1|O|3|-4|5|6|7|10|8|@]|D|@]|E|$]]]|P|$]]

It's been a long time since I last worked with Python, but I think the problem is with the statement <code>for line in proc.stdout</code>, which reads the entire input before iterating over it. The solution is to use <code>readline()</code> instead:

<pre><code>#filters output
import subprocess
proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
while True:
 line = proc.stdout.readline()
 if not line:
 break
 #the real code does filtering here
 print "test:", line.rstrip()
</code></pre>

Of course you still have to deal with the subprocess' buffering.

Note: <a href="http://docs.python.org/library/stdtypes.html#file.next" rel="noreferrer">according to the documentation</a> the solution with an iterator should be equivalent to using <code>readline()</code>, except for the read-ahead buffer, but (or exactly because of this) the proposed change did produce different results for me (Python 2.5 on Windows XP).

blocks|key|3044785|text|实际上，如果你整理了迭代器，那么缓冲现在可能就是你的问题了。您可以告诉该子进程中的python不要缓冲其输出。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3044786|proc+=+subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)|code-block|syntax|javascript|3044787|变成了|3044788|proc+=+subprocess.Popen(['python','-u',+'fake_utility.py'],stdout=subprocess.PIPE)|3044789|当我从python内部调用python时，我需要它。|3044790|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|Q|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|R|8|@]|9|@]|A|$E|F]]|$1|K|3|L|5|6|7|S|8|@]|9|@]|A|$]]|$1|M|3|-4|5|6|7|T|8|@]|9|@]|A|$]]]|N|$]]

Indeed, if you sorted out the iterator then buffering could now be your problem. You could tell the python in the sub-process not to buffer its output.

<pre><code>proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
</code></pre>

becomes 

<pre><code>proc = subprocess.Popen(['python','-u', 'fake_utility.py'],stdout=subprocess.PIPE)
</code></pre>

I have needed this when calling python from within python.

blocks|key|3258665|text|您希望将这些额外的参数传递给subprocess.Popen|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|3258666|bufsize=1,+universal_newlines=True|code-block|syntax|javascript|3258667|然后，您可以按照示例中的方式进行迭代。(使用Python+3.5测试)|3258668|entityMap^0|E|G|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@$9|P|A|Q|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|R|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|S|8|@]|D|@]|E|$]]|$1|M|3|-4|5|6|7|T|8|@]|D|@]|E|$]]]|N|$]]

You want to pass these extra parameters to <code>subprocess.Popen</code>:

<pre><code>bufsize=1, universal_newlines=True
</code></pre>

Then you can iterate as in your example. (Tested with Python 3.5)

blocks|key|3258677|text|有点晚了，但我很惊讶没有看到我认为是最简单的解决方案：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3258678|import+io
import+subprocess

proc+=+subprocess.Popen(["prog",+"arg"],+stdout=subprocess.PIPE)
for+line+in+io.TextIOWrapper(proc.stdout,+encoding="utf-8"):++#+or+another+encoding
++++#+do+something+with+line|code-block|syntax|javascript|3258679|(这需要Python+3。)|3258680|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

Bit late to the party, but was surprised not to see what I think is the simplest solution here:

<pre><code>import io
import subprocess

proc = subprocess.Popen(["prog", "arg"], stdout=subprocess.PIPE)
for line in io.TextIOWrapper(proc.stdout, encoding="utf-8"): # or another encoding
 # do something with line
</code></pre>

(This requires Python 3.)

blocks|key|3044814|text|下面是我在Python+2和Python+3+(2.7.12和3.6.1)上对Rámulo答案的修改：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3044815|import+os
import+subprocess

process+=+subprocess.Popen(command,+stdout=subprocess.PIPE)
while+True:
++line+=+process.stdout.readline()
++if+line+!=+'':
++++os.write(1,+line)
++else:
++++break|code-block|syntax|javascript|3044816|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

The following modification of Rômulo's answer works for me on Python 2 and 3 (2.7.12 and 3.6.1):

<pre class="lang-py prettyprint-override"><code>import os
import subprocess

process = subprocess.Popen(command, stdout=subprocess.PIPE)
while True:
 line = process.stdout.readline()
 if line != '':
 os.write(1, line)
 else:
 break
</code></pre>

blocks|key|3044818|text|source，我在python3上试过了，它起作用了|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|3044819|def+output_reader(proc):
++++for+line+in+iter(proc.stdout.readline,+b''):
++++++++print('got+line:+{0}'.format(line.decode('utf-8')),+end='')


def+main():
++++proc+=+subprocess.Popen(['python',+'fake_utility.py'],
++++++++++++++++++++++++++++stdout=subprocess.PIPE,
++++++++++++++++++++++++++++stderr=subprocess.STDOUT)

++++t+=+threading.Thread(target=output_reader,+args=(proc,))
++++t.start()

++++try:
++++++++time.sleep(0.2)
++++++++import+time
++++++++i+=+0

++++++++while+True:
++++++++print+(hex(i)*512)
++++++++i+%2B=+1
++++++++time.sleep(0.5)
++++finally:
++++++++proc.terminate()
++++++++try:
++++++++++++proc.wait(timeout=0.2)
++++++++++++print('==+subprocess+exited+with+rc+=',+proc.returncode)
++++++++except+subprocess.TimeoutExpired:
++++++++++++print('subprocess+did+not+terminate+in+time')
++++t.join()|code-block|syntax|javascript|3044820|entityMap|0|LINK|mutability|MUTABLE|url|https://eli.thegreenplace.net/2017/interacting-with-a-long-running-child-process-in-python/^0|0|6|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$G|H]]|$1|I|3|-4|5|6|7|V|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

I tried this with python3 and it worked, <a href="https://eli.thegreenplace.net/2017/interacting-with-a-long-running-child-process-in-python/" rel="nofollow noreferrer">source</a>

<pre><code>def output_reader(proc):
 for line in iter(proc.stdout.readline, b''):
 print('got line: {0}'.format(line.decode('utf-8')), end='')


def main():
 proc = subprocess.Popen(['python', 'fake_utility.py'],
 stdout=subprocess.PIPE,
 stderr=subprocess.STDOUT)

 t = threading.Thread(target=output_reader, args=(proc,))
 t.start()

 try:
 time.sleep(0.2)
 import time
 i = 0

 while True:
 print (hex(i)*512)
 i += 1
 time.sleep(0.5)
 finally:
 proc.terminate()
 try:
 proc.wait(timeout=0.2)
 print('== subprocess exited with rc =', proc.returncode)
 except subprocess.TimeoutExpired:
 print('subprocess did not terminate in time')
 t.join()
</code></pre>

blocks|key|307937|text|您还可以读取没有循环的行。在python3.6中工作。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|307938|import+os
import+subprocess

process+=+subprocess.Popen(command,+stdout=subprocess.PIPE)
list_of_byte_strings+=+process.stdout.readlines()|code-block|syntax|javascript|307939|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

You can also read lines w/o loop. Works in python3.6.

<pre><code>import os
import subprocess

process = subprocess.Popen(command, stdout=subprocess.PIPE)
list_of_byte_strings = process.stdout.readlines()
</code></pre>

blocks|key|310997|text|允许实时、逐行地同时迭代+stderr+和的函数|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|CODE|entityRanges|data|310998|如果您需要同时获取stdout和stderr的输出流，可以使用以下函数。|310999|该函数使用队列将两个Popen管道合并为一个迭代器。|311000|在这里，我们创建函数read_popen_pipes()|311001|from+queue+import+Queue,+Empty
from+concurrent.futures+import+ThreadPoolExecutor


def+enqueue_output(file,+queue):
++++for+line+in+iter(file.readline,+''):
++++++++queue.put(line)
++++file.close()


def+read_popen_pipes(p):

++++with+ThreadPoolExecutor(2)+as+pool:
++++++++q_stdout,+q_stderr+=+Queue(),+Queue()

++++++++pool.submit(enqueue_output,+p.stdout,+q_stdout)
++++++++pool.submit(enqueue_output,+p.stderr,+q_stderr)

++++++++while+True:

++++++++++++if+p.poll()+is+not+None+and+q_stdout.empty()+and+q_stderr.empty():
++++++++++++++++break

++++++++++++out_line+=+err_line+=+''

++++++++++++try:
++++++++++++++++out_line+=+q_stdout.get_nowait()
++++++++++++except+Empty:
++++++++++++++++pass
++++++++++++try:
++++++++++++++++err_line+=+q_stderr.get_nowait()
++++++++++++except+Empty:
++++++++++++++++pass

++++++++++++yield+(out_line,+err_line)|code-block|syntax|javascript|311002|正在使用的read_popen_pipes()：|311003|import+subprocess+as+sp


with+sp.Popen(my_cmd,+stdout=sp.PIPE,+stderr=sp.PIPE,+text=True)+as+p:

++++for+out_line,+err_line+in+read_popen_pipes(p):

++++++++#+Do+stuff+with+each+line,+e.g.:
++++++++print(out_line,+end='')
++++++++print(err_line,+end='')

++++return+p.poll()+#+return+status-code|311004|entityMap^0|0|C|D|6|D|6|0|9|6|G|6|0|0|A|I|0|0|5|I|0|0^^$0|@$1|2|3|4|5|6|7|X|8|@$9|Y|A|Z|B|C]|$9|10|A|11|B|C]|$9|12|A|13|B|D]]|E|@]|F|$]]|$1|G|3|H|5|6|7|14|8|@$9|15|A|16|B|D]|$9|17|A|18|B|D]]|E|@]|F|$]]|$1|I|3|J|5|6|7|19|8|@]|E|@]|F|$]]|$1|K|3|L|5|6|7|1A|8|@$9|1B|A|1C|B|D]]|E|@]|F|$]]|$1|M|3|N|5|O|7|1D|8|@]|E|@]|F|$P|Q]]|$1|R|3|S|5|6|7|1E|8|@$9|1F|A|1G|B|D]]|E|@]|F|$]]|$1|T|3|U|5|O|7|1H|8|@]|E|@]|F|$P|Q]]|$1|V|3|-4|5|6|7|1I|8|@]|E|@]|F|$]]]|W|$]]

A function that allows iterating over both <code>stdout</code> and <code>stderr</code> concurrently, in realtime, line by line

In case you need to get the output stream for both <code>stdout</code> and <code>stderr</code> at the same time, you can use the following function.

The function uses Queues to merge both Popen pipes into a single iterator.

Here we create the function <code>read_popen_pipes()</code>:

<pre class="lang-py prettyprint-override"><code>from queue import Queue, Empty
from concurrent.futures import ThreadPoolExecutor


def enqueue_output(file, queue):
 for line in iter(file.readline, ''):
 queue.put(line)
 file.close()


def read_popen_pipes(p):

 with ThreadPoolExecutor(2) as pool:
 q_stdout, q_stderr = Queue(), Queue()

 pool.submit(enqueue_output, p.stdout, q_stdout)
 pool.submit(enqueue_output, p.stderr, q_stderr)

 while True:

 if p.poll() is not None and q_stdout.empty() and q_stderr.empty():
 break

 out_line = err_line = ''

 try:
 out_line = q_stdout.get_nowait()
 except Empty:
 pass
 try:
 err_line = q_stderr.get_nowait()
 except Empty:
 pass

 yield (out_line, err_line)
</code></pre>

<code>read_popen_pipes()</code> in use:

<pre class="lang-py prettyprint-override"><code>import subprocess as sp


with sp.Popen(my_cmd, stdout=sp.PIPE, stderr=sp.PIPE, text=True) as p:

 for out_line, err_line in read_popen_pipes(p):

 # Do stuff with each line, e.g.:
 print(out_line, end='')
 print(err_line, end='')

 return p.poll() # return status-code
</code></pre>

blocks|key|308895|text|Pythont3.5向subprocess模块添加了方法run()和call()，这两个方法都返回一个CompletedProcess对象。这样，您就可以很好地使用proc.stdout.splitlines()了|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|308896|proc+=+subprocess.run(+comman,+shell=True,+capture_output=True,+text=True,+check=True+)
for+line+in+proc.stdout.splitlines():
+++print+"stdout:",+line|code-block|syntax|javascript|308897|另请参阅How+to+Execute+Shell+Commands+in+Python+Using+the+Subprocess+Run+Method|308898|entityMap|0|LINK|mutability|MUTABLE|url|https://linuxhint.com/execute_shell_python_subprocess_run_method/^0|B|A|S|5|Y|6|1F|G|2A|O|0|0|4|1Z|0|0^^$0|@$1|2|3|4|5|6|7|U|8|@$9|V|A|W|B|C]|$9|X|A|Y|B|C]|$9|Z|A|10|B|C]|$9|11|A|12|B|C]|$9|13|A|14|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|15|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|16|8|@]|D|@$9|17|A|18|1|19]]|E|$]]|$1|M|3|-4|5|6|7|1A|8|@]|D|@]|E|$]]]|N|$O|$5|P|Q|R|E|$S|T]]]]

Pythont 3.5 added the methods <code>run()</code> and <code>call()</code> to the <code>subprocess</code> module, both returning a <code>CompletedProcess</code> object. With this you are fine using <code>proc.stdout.splitlines()</code>:
<pre><code>proc = subprocess.run( comman, shell=True, capture_output=True, text=True, check=True )
for line in proc.stdout.splitlines():
 print &quot;stdout:&quot;, line
</code></pre>
See also <a href="https://linuxhint.com/execute_shell_python_subprocess_run_method/" rel="nofollow noreferrer">How to Execute Shell Commands in Python Using the Subprocess Run Method</a>

My python script uses subprocess to call a linux utility that is very noisy. I want to store all of the output to a log file and show some of it to the user. I thought the following would work, but the output doesn't show up in my application until the utility has produced a significant amount of output.

<pre><code>#fake_utility.py, just generates lots of output over time
import time
i = 0
while True:
 print hex(i)*512
 i += 1
 time.sleep(0.5)

#filters output
import subprocess
proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
for line in proc.stdout:
 #the real code does filtering here
 print "test:", line.rstrip()
</code></pre>

The behavior I really want is for the filter script to print each line as it is received from the subprocess. Sorta like what <a href="http://unixhelp.ed.ac.uk/CGI/man-cgi?tee" rel="noreferrer"><code>tee</code></a> does but with python code.

What am I missing? Is this even possible?

<hr>

Update:

If a <code>sys.stdout.flush()</code> is added to fake_utility.py, the code has the desired behavior in python 3.1. I'm using python 2.6. You would think that using <code>proc.stdout.xreadlines()</code> would work the same as py3k, but it doesn't.

<hr>

Update 2:

Here is the minimal working code.

<pre><code>#fake_utility.py, just generates lots of output over time
import sys, time
for i in range(10):
 print i
 sys.stdout.flush()
 time.sleep(0.5)

#display out put line by line
import subprocess
proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
#works in python 3.0+
#for line in proc.stdout:
for line in iter(proc.stdout.readline,''):
 print line.rstrip()
</code></pre>

read subprocess stdout line by line

日志文件

Windows

Python

我的python脚本使用子进程来调用一个非常嘈杂的linux实用程序。我希望将所有输出存储到一个日志文件中，并向用户显示其中的一部分。我原以为下面的命令会起作用，但是直到实用程序产生了大量的输出后，输出才会出现在我的应用程序中。#fake_utility.py, just generates lots of outpu...

问逐行读取子进程标准输出
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问逐行读取子进程标准输出EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问逐行读取子进程标准输出
EN