clojure.tools.build.api/process + ignore hangs when process has large output

Question

Similar to https://ask.clojure.org/index.php/11509/clojure-launch-doesnt-consume-inputstream-process-terminating

tools.build.api/process hangs when ignoring output and the process has large output. Here is curl downloading and printing 1mb of text:

$ clj -Sdeps '{:deps {io.github.clojure/tools.build {:git/tag "v0.7.5" :git/sha "34727f7"}}}'
Clojure 1.11.0-alpha3
user=> (require '[clojure.tools.build.api :as tb])

user=> (tb/process {:command-args ["curl" "https://gist.githubusercontent.com/khaykov/a6105154becce4c0530da38e723c2330/raw/41ab415ac41c93a198f7da5b47d604956157c5c3/gistfile1.txt"] :out :ignore})
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

I think instead of using PIPE, the ignore option should be using DISCARD:

https://docs.oracle.com/en/java/javase/16/docs/api/java.base/java/lang/ProcessBuilder.Redirect.html#DISCARD

One caveat: DISCARD is only available on Java 9 and newer. We could do resolve in a try/catch on the top level to see if DISCARD is available and then use that, if available and fall back on PIPE if it isn't. Alternatively, use a thread to consume output while the process is running or redirect to a temporary file.

I'll provide a patch, if desired, and when agreed upon the direction of the solution.

Answer 1

This is logged as https://clojure.atlassian.net/browse/TBUILD-1, haven't had a chance to think about it yet.

Comments

In addition, if the output is very large, the following will hang:

(defn get-history [_]
  (b/git-process {:git-args ["log" "--after=2025-01-01" "--pretty=format:%D%n%B"]}))

roklenarcic on Slack says:

this code is wrong:
(let [proc (.start pb)
          exit (.waitFor proc)
          out-str (when (= out :capture) (copy-stream (.getInputStream proc)))
          err-str (when (= err :capture) (copy-stream (.getErrorStream proc)))]
      (cond-> {:exit exit}
        out-str (assoc :out out-str)
        err-str (assoc :err err-str)))

you cannot await for process exit before setting up something that will drain the streams
that will cause process to block if the output is big enough

Alex says:

this process stuff (which was the precursor of the new 1.12 clojure.java.process api) definitely needs some clean up in this area. I actually thought I had done that but must have confused it with the work there