Compile parser ahead of time

Question

tools.cli/parse-opts accepts the args, the option-spec, and the additional options. It calls compile-option-specs and required-arguments to build the specs and req. Then it performs the validation with those specs and req on the provided args (along with the options). For a given application, the first two steps aren't going to change across calls.

I propose a new make-parse-opts-fn function that performs the compile-option-specs and required-arguments up front and returns a function that relies on the compiled specs and req. It could be used like this: (def compiled-parser (make-parse-opts-fn cli-options)).

Microbenchmarking with criterium shows a more than double increase in speed (tools.cli/parse-opts first, pre-compiled parser second):

; user=> (def cli-options
  [["-h" "--help" "This message"]
   [nil "--extra" "Output in extra format"
    :default false]
   ["-q" "--quiet" "Print no suggestions, only return exit code"
    :default false]])
#'user/cli-options

; user=> (bench (cli/parse-opts ["--quiet" "src"] cli-options :in-order true))
Evaluation count : 10962 in 6 samples of 1827 calls.
             Execution time mean : 65.021329 µs
    Execution time std-deviation : 5.276033 µs
   Execution time lower quantile : 58.658638 µs ( 2.5%)
   Execution time upper quantile : 69.724228 µs (97.5%)
                   Overhead used : 9.607933 ns
nil

; user=> (bench (compiled-parser ["--quiet" "src"] :in-order true))
Evaluation count : 24660 in 6 samples of 4110 calls.
             Execution time mean : 25.090846 µs
    Execution time std-deviation : 253.361821 ns
   Execution time lower quantile : 24.769079 µs ( 2.5%)
   Execution time upper quantile : 25.413286 µs (97.5%)
                   Overhead used : 9.607933 ns
nil

If the additional options are lifted into the make-parse-opts-fn as well (trading flexibility for speed), the difference is even more dramatic, providing roughly 10x speed increase over the existing tools.cli/parse-opts function:

; user=> (bench (compiled-parser-2 ["--quiet" "src"]))
Evaluation count : 73116 in 6 samples of 12186 calls.
             Execution time mean : 8.349923 µs
    Execution time std-deviation : 51.419864 ns
   Execution time lower quantile : 8.282266 µs ( 2.5%)
   Execution time upper quantile : 8.407998 µs (97.5%)
                   Overhead used : 9.607933 ns

Found 2 outliers in 6 samples (33.3333 %)
	low-severe	 1 (16.6667 %)
	low-mild	 1 (16.6667 %)
 Variance from outliers : 13.8889 % Variance is moderately inflated by outliers
nil

I can provide a patch for this if there's interest.

Comments

Can you explain the use case for parsing command-line arguments multiple times within a single run of an application?

I don't doubt your benchmarks showing that precompiling first and then parsing multiple times is faster than running the whole process multiple times -- but I don't understand why this would be useful.

---

The two use cases are "In-order processing of subcommands" as noted in the readme and running tests. I originally dove into this under the impression that precompiling would help single runs when AOT compiled but I don't think that's actually the case. However, this split marginally helps when running my test suite, wherein I'm calling code through the cli interface a bunch.

---

With "In-order processing of subcommands", that's still something that is generally only going to be performed once at startup, unless you're writing some sort of interactive shell (which may have a different syntax, no so amenable to tools.cli anyway).

I'd be happy to make private functions public so you could write your own wrapper for preprocessing/compilation, if that would help, but this doesn't feel like a use case that is compiling enough to bake it directly into the API.

---

That’s true it happens only once, I was thinking of processing subcommands and calling parse-opts multiple times in a row.

I think making the private functions called in parse-opts public would be nice, but I’m no longer convinced precomping is a good idea. Maybe there’s speed to be gained by performing the compilation and then copy-pasting the resulting data back into my code and writing a function of the second half of parse-opts that uses the manually compiled opts?

Yeah, making the compilation functions public (with intended privacy noted in the doc strings) would at least make it easier for me to experiment with this.

---

Noah, since you can call private functions (using #' to reference their Vars), would you mind going ahead and doing those experiments -- and if they show a worthwhile improvement, then I'll go ahead and make the relevant parts public. Thanks.

---

Hey Sean! After roughly 9 months of trying things out, the only thing that I think should be changed is I would like `cli/summarize` to be public. I pass in `:summary-fn identity` to `cli/parse-opts` to avoid building the summary string, only calling `(#'cli/summarize summary)` if the user requests the help docs. I can access it if it's private (as demonstrated), but that has the deref cost which I'd prefer to avoid.

---

Just kidding, `cli/summarize` is already public. I think the doc string made me think it was private and I never actually checked lol. I wish I knew how to read. :sob:

---

So... it sounds like TCLI-101 can be closed out with no action needed?

---

Yes, thank you.

Answer 1

Created jira as https://clojure.atlassian.net/browse/TCLI-101