Home How can I asynchronously capture output from a long-running Python subprocess
Reply: 0

How can I asynchronously capture output from a long-running Python subprocess

user2976 Published in July 18, 2018, 4:51 am

For a test I want to start a long-running Python 3 script on Linux, capture some of its output, check that it's as expected and kill it again.

I'm just trying to get the general framework working for now.

When I run the following code, I would expect the output to include line = "0" -- the python script just prints out an increasing sequence of integers, one per second. Instead, that for loop is skipped completely.

import subprocess
from tempfile import NamedTemporaryFile
import time
import unittest

import asynchronousfilereader

class TestProcessSpawning(unittest.TestCase):
    def test_spawning_counter(self):
        counter_code = \
import time
i = 0
while True:
    i = i + 1

        with NamedTemporaryFile(mode="w", suffix='.py', delete=False)\
                as temp_file:
            file_name = temp_file.name

            # proc = subprocess.Popen(['ping', 'localhost'],
            #                         stdout=subprocess.PIPE, close_fds=True)
            proc = subprocess.Popen(['python3', file_name],
                                    stdout=subprocess.PIPE, close_fds=True)


            assert proc.returncode is None  # None => still running

            reader = asynchronousfilereader.AsynchronousFileReader(proc.stdout)

            time.sleep(3)  # give it a chance?

            for line in reader.readlines():
                print('line = "{}"'.format(line))
                break  # just grab first one


However, if I change ['python3', file_name] to ['ping', 'localhost'] I do get a line of output from ping (I'm running on Linux so ping keeps producing output until you stop it).

Any idea why this seems to work ok for other types of subprocess but not for python?


  • this code uses the asynchronousfilereader available from pip
  • if I just run python3 <temp_file_name> in a shell, the script runs as expected, printing 0, 1, 2, ...
  • the process return code of None after a few seconds suggests the process gets run (i.e. doesn't just crash)
  • I have tried running it through unbuffer as suggested here: the output did not appear
  • I tried passing bufsize=1 to Popen: the output did not appear
  • if I include the -v flag I get lots of output, including this, which suggests python is trying to run interactively even though I've provided a file name:

Python 3.5.2 (default, Nov 23 2017, 16:37:01)

[GCC 5.4.0 20160609] on linux

Type "help", "copyright", "credits" or "license" for more information.

You need to login account before you can post.

About| Privacy statement| Terms of Service| Advertising| Contact us| Help| Sitemap|
Processed in 0.308317 second(s) , Gzip On .

© 2016 Powered by mzan.com design MATCHINFO