Handling blocked Process output stream

If you’re invoking external processes with Runtime.getRuntime().exec() or ProcessBuilder.start(), it is vitally important to handle the standard out and standard error streams. Otherwise, the parent process may block – it gets stuck without throwing an Exception to tell you why.

Invoking a process like this can cause problems:

Process p = Runtime.getRuntime().exec("./my_external_process.sh 5000");
boolean done = p.waitFor(60, TimeUnit.SECONDS);

Here I have an external process (my_external_process.sh) that writes 5000 lines to standard out (stdout). When I run this, waitFor waits 60 seconds for the process to complete and then returns false. The process did not complete within 60 seconds.

If I run my_external_process.sh 5000 from command line, it completes in a second or so. If I invoke it from Java with only 500 lines to stdout, it completes fine in less than a second. The issue here is not the timeout. I could remove the timeout and the program would never complete.

What’s blocking my process?

The Process class Javadoc explains the issue:

When my process writes to stdout, the output goes to an InputStream buffer ready to be read. If nothing reads it and it fills up, it will refuse to accept any further output from the process. This blocks the process from running.

How to prevent the process from blocking

We can prevent the process from blocking by reading from the process output streams. Confusingly, my process’s stdout and stderr streams are read in to my Java application and so have type InputStream. Some basic code to handle the stdout stream:

Process p = Runtime.getRuntime().exec("./my_external_process.sh 5000");
InputStream stdOut = p.getInputStream();

try (BufferedReader stdOutReader = new BufferedReader(new InputStreamReader(stdOut))) {
    while (stdOutReader.readLine() != null) {
        // Just reading the line is enough. We don't need to do anything else with it.
    }
}
boolean done = p.waitFor(60, TimeUnit.SECONDS);

The standard out (getInputStream()) is read and discarded. This is enough to prevent the InputStream buffer from filling up and blocking my application. If we want to do anything with the process’s output, just do something with the readLine() result:

String line;
while ((line = stdOutReader.readLine()) != null) {
    System.out.println(line);
}

How to handle stdout and stderr

We need to be a little more sophisticated to handle both stdout and stderr. A naive solution is to handle the stderr stream in the same way:

Process p = Runtime.getRuntime().exec("./my_external_process.sh 5000");
InputStream stdOut = p.getInputStream();
InputStream stdErr = p.getErrorStream();

try (BufferedReader stdOutReader = new BufferedReader(new InputStreamReader(stdOut))) {
    while (stdOutReader.readLine() != null) {
        // Just reading the line is enough. We don't need to do anything else with it.
    }
}
// THIS WON'T WORK!
try (BufferedReader stdErrReader = new BufferedReader(new InputStreamReader(stdErr))) {
    while (stdErrReader.readLine() != null) {
        // Just reading the line is enough. We don't need to do anything else with it.
    }
}
boolean done = p.waitFor(60, TimeUnit.SECONDS);

The problem here is that we handle the two streams sequentially. We handle the stdout stream until it’s exhausted and then we handle stderr. The stderr buffer will fill up before we start processing it and will cause the program to block while we’re still processing stdout.

Instead, we need to handle both streams in parallel, each in its own thread. Use an ExecutorService to execute the two tasks at once:

ExecutorService streamHandlers = Executors.newFixedThreadPool(2);

Process p = Runtime.getRuntime().exec("./my_external_process.sh -v 5000");
InputStream stdOut = p.getInputStream();
InputStream stdErr = p.getErrorStream();

streamHandlers.execute(() -> handleStream(stdOut));
streamHandlers.execute(() -> handleStream(stdErr));

boolean done = p.waitFor(60, TimeUnit.SECONDS);
if (done) {
    System.out.println("Process completed successfully");
} else {
    System.out.println("Process did not complete in 60 seconds");
}

The handleStream() method simply reads the stream:

private static void handleStream(InputStream inputStream) {
    try (BufferedReader stdOutReader = new BufferedReader(new InputStreamReader(inputStream))) {
        while (stdOutReader.readLine() != null) {
            // Just reading the line is enough. We don't need to do anything else with it.
        }
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
}

Combine the streams with ProcessBuilder

A simpler way to handle the two streams is to combine them. If you invoke the process with ProcessBuilder, you can combine (redirect) the error stream and input stream like this:

Process p = new ProcessBuilder("./my_external_process.sh", "-v", "5000")
        .redirectErrorStream(true)
        .start();
InputStream stdOut = p.getInputStream();

You can then process the combined stream without needing needing a separate thread:

try (BufferedReader stdOutReader = new BufferedReader(new InputStreamReader(stdOut))) {
    while (stdOutReader.readLine() != null) {
        // Just reading the line is enough. We don't need to do anything else with it.
    }
}

Full example code

For the full worked example demonstrating each of these methods, take a look at my process output stream repo on GitHub.