Home How can I asynchronously capture output from a long-running Python subprocess
Reply: 1

How can I asynchronously capture output from a long-running Python subprocess

Matthew Strawbridge
Matthew Strawbridge Published in 2017-12-06 10:08:58Z

For a test I want to start a long-running Python 3 script on Linux, capture some of its output, check that it's as expected and kill it again.

I'm just trying to get the general framework working for now.

When I run the following code, I would expect the output to include line = "0" -- the python script just prints out an increasing sequence of integers, one per second. Instead, that for loop is skipped completely.

import subprocess
from tempfile import NamedTemporaryFile
import time
import unittest

import asynchronousfilereader

class TestProcessSpawning(unittest.TestCase):
    def test_spawning_counter(self):
        counter_code = \
import time
i = 0
while True:
    i = i + 1

        with NamedTemporaryFile(mode="w", suffix='.py', delete=False)\
                as temp_file:
            file_name = temp_file.name

            # proc = subprocess.Popen(['ping', 'localhost'],
            #                         stdout=subprocess.PIPE, close_fds=True)
            proc = subprocess.Popen(['python3', file_name],
                                    stdout=subprocess.PIPE, close_fds=True)


            assert proc.returncode is None  # None => still running

            reader = asynchronousfilereader.AsynchronousFileReader(proc.stdout)

            time.sleep(3)  # give it a chance?

            for line in reader.readlines():
                print('line = "{}"'.format(line))
                break  # just grab first one


However, if I change ['python3', file_name] to ['ping', 'localhost'] I do get a line of output from ping (I'm running on Linux so ping keeps producing output until you stop it).

Any idea why this seems to work ok for other types of subprocess but not for python?


  • this code uses the asynchronousfilereader available from pip
  • if I just run python3 <temp_file_name> in a shell, the script runs as expected, printing 0, 1, 2, ...
  • the process return code of None after a few seconds suggests the process gets run (i.e. doesn't just crash)
  • I have tried running it through unbuffer as suggested here: the output did not appear
  • I tried passing bufsize=1 to Popen: the output did not appear
  • if I include the -v flag I get lots of output, including this, which suggests python is trying to run interactively even though I've provided a file name:

Python 3.5.2 (default, Nov 23 2017, 16:37:01)

[GCC 5.4.0 20160609] on linux

Type "help", "copyright", "credits" or "license" for more information.

Martijn Pieters
Martijn Pieters Reply to 2017-12-06 11:08:38Z

You need to flush the file you write:

with NamedTemporaryFile(mode="w", suffix='.py', delete=False) as temp_file:

Your program is too short to ever make it out of the buffer otherwise, and you are running an empty file.

I'd also make sure the program then flushes on every write; either run python3 -u for unbuffered stdout or use flush=True on every print. That way you don't have to add so many sleeps in your parent program.

With these changes, the unittest prints

line = "b'0\n'"

and several warnings about the subprocess still running and the stdout buffer still being open before completing.

You need to login account before you can post.

About| Privacy statement| Terms of Service| Advertising| Contact us| Help| Sitemap|
Processed in 0.307062 second(s) , Gzip On .

© 2016 Powered by mzan.com design MATCHINFO