Is this a bug in cl-format

Question

The following call to cl-format results in a Java level null pointer exception.

(cl-format false "
~@")

Of course that code sample is pretty meaningless. The real example which lead to this minimum test case was the following.

(cl-format false "~&~
                            item=~A~@
                            transitions=~A~@
                            lhs=~A~@
                            rhs=~A~@"
                      1
                      2
                      3
                      4
                      )

As near as I can tell the problem seems to be in dealing with the final ~@ where the clojure code seems to (I'm guessing) destructure the nil return value of extract-params
in the function compile-directive in cl_format.clj.

(defn- compile-directive [s offset]
  (let [[raw-params [rest offset]] (extract-params s offset)
        [_ [rest offset flags]] (extract-flags rest offset)
        directive (first rest)
        def (get directive-table (Character/toUpperCase ^Character directive))
        params (if def (map-params def (map translate-param raw-params) flags offset))]
    (if (not directive)
      (format-error "Format string ended in the middle of a directive" offset))
    (if (not def)
      (format-error (str "Directive \"" directive "\" is undefined") offset))
    [(struct compiled-directive ((:generator-fn def) params offset) def params offset)
     (let [remainder (subs rest 1) 
           offset (inc offset)
           trim? (and (= \newline (:directive def))
                      (not (:colon params)))
           trim-count (if trim? (prefix-count remainder [\space \tab]) 0)
           remainder (subs remainder trim-count)
           offset (+ offset trim-count)]
       [remainder offset])]))

Here is the stack trace I see.

1. Unhandled java.lang.NullPointerException
   (No message)

             cl_format.clj: 1717  clojure.pprint/compile-directive
             cl_format.clj: 1861  clojure.pprint/compile-format/fn
                  AFn.java:  154  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  665  clojure.core/apply
             utilities.clj:   37  clojure.pprint/consume
             cl_format.clj: 1845  clojure.pprint/compile-format
             cl_format.clj: 1845  clojure.pprint/compile-format
             cl_format.clj:   62  clojure.pprint/cl-format
             cl_format.clj:   27  clojure.pprint/cl-format
               RestFn.java:  521  clojure.lang.RestFn/invoke
                      REPL: 1391  clojure-rte.rte-core/eval24978
                      REPL: 1391  clojure-rte.rte-core/eval24978
             Compiler.java: 7176  clojure.lang.Compiler/eval
             Compiler.java: 7131  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj: 1973  clojure.core/with-bindings*
                  core.clj: 1973  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  414  clojure.main/repl/read-eval-print/fn
                  main.clj:  414  clojure.main/repl/read-eval-print
                  main.clj:  435  clojure.main/repl/fn
                  main.clj:  435  clojure.main/repl
                  main.clj:  345  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  202  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  201  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  834  java.lang.Thread/run

Answer 1

After a bit of research I now understand the error better. The format string given to cl-format may contain zero or more directives. Each directive begins with ~ and ends with a dispatch character such as % or a or s etc. between the ~ and the dispatch character several intervening things can be found. such as : or @ in either order, and potentially arguments such as ~2,4f or ~3,-4s or ~3,-4:@s. the : and @ are simply Boolean flags to pass to the dispatch function.

IN my case I gave it ~@ followed by end of string, so cl-format should ideally detect that end-of-string was reached before the dispatch character was found.

Here is the error I get in common-lisp

CL-USER> (format nil "~@")
error in FORMAT: String ended before directive was found
  ~@
  ^
   [Condition of type SB-FORMAT:FORMAT-ERROR]

Restarts:
 0: [RETRY] Retry SLIME REPL evaluation request.
 1: [*ABORT] Return to SLIME's top level.
 2: [ABORT] abort thread (#<THREAD "repl-thread" RUNNING {1004289BC3}>)

Backtrace:
  0: (SB-FORMAT::FORMAT-ERROR-AT* "~@" 0 "String ended before directive was found" NIL)
  1: (SB-FORMAT::FORMAT-ERROR-AT "~@" 0 "String ended before directive was found")
  2: ((FLET SB-FORMAT::GET-CHAR :IN SB-FORMAT::PARSE-DIRECTIVE))
  3: (SB-FORMAT::PARSE-DIRECTIVE "~@" 0 NIL)
  4: (SB-FORMAT::%TOKENIZE-CONTROL-STRING "~@" 0 2 NIL)
  5: (SB-FORMAT::TOKENIZE-CONTROL-STRING "~@")
  6: (SB-FORMAT::%FORMAT #<SB-IMPL::CHARACTER-STRING-OSTREAM {1004883B33}> "~@" NIL NIL)
  7: (FORMAT NIL "~@")
  8: (FORMAT NIL "~@") [more]
  9: (SB-INT:SIMPLE-EVAL-IN-LEXENV (FORMAT NIL "~@") #<NULL-LEXENV>)
 10: (EVAL (FORMAT NIL "~@"))

Comments

BTW it looks like the function `compile-directive` in `cl_format.clj` has code to detect `"Format string ended in the middle of a directive"` and call `format-error` with a helpful error message.   Unfortunately the arrival of the java error seems to prevent this code from being reached.

---

I believe the problem can be fixed with this patch to the `compile-directive` function.  Note that I'm using the `if` without `else` convention which the previous author has used.  The change simply detects whether the `directive` declared in the `let` was `nil` as opposed to a character.

```
(defn- compile-directive [s offset]
  (let [[raw-params [rest offset]] (extract-params s offset)
        [_ [rest offset flags]] (extract-flags rest offset)
        directive (first rest)
        def (if directive (get directive-table (Character/toUpperCase ^Character directive)))
        params (if def (map-params def (map translate-param raw-params) flags offset))]
    (if (not directive)
      (format-error "Format string ended in the middle of a directive" offset))
    (if (not def)
      (format-error (str "Directive \"" directive "\" is undefined") offset))
    [(struct compiled-directive ((:generator-fn def) params offset) def params offset)
     (let [remainder (subs rest 1)
           offset (inc offset)
           trim? (and (= \newline (:directive def))
                      (not (:colon params)))
           trim-count (if trim? (prefix-count remainder [\space \tab]) 0)
           remainder (subs remainder trim-count)
           offset (+ offset trim-count)]
       [remainder offset])]))
```

With this change I get a meaningful error message, which I believe was the original author's intent.

```
(cl-format nil "~:@")

Execution error at nrepl.middleware.interruptible-eval/evaluate$fn$fn (interruptible_eval.clj:87).
Format string ended in the middle of a directive
~:@
   ^

```

---

I tried to create a pull request with this simple patch.  However, I see that this project does not accept pull requests.  I tried to follow the instructions but I admit that I got lost.

---

I received and recorded your contributor agreement and have sent an invite to your email for jira. I've filed a ticket at https://clojure.atlassian.net/browse/CLJ-2653. You can follow the instructions at https://clojure.org/dev/developing_patches for making a patch if you like. Thanks!

---

Has this patch been rejected?  I ran into the same issue again today, and it seems the code in Clojure 1.11.1 still has the same bug.

Maybe I did something wrong in my patch?   

Can someone help?

---

We haven’t looked at it yet, sorry.

---

it is a very low risk fix. in my opinion