Clojure Q&A - Recent questions tagged clojure.test

Separate bindable reporting function from clojure.test/report

Tue, 27 Aug 2024 18:38:20 +0000

Set Up

clojure.test is built on the idea that a test will call an assertion function that directly calls a reporting function: is calls report (through do-report). The function report is a multimethod, allowing for adding or overriding keys to allow for implementing alternative reporting styles. It's also dynamic, meant to be rebound if an entirely new reporting system is to be used (TAP, JUnit, etc).

Problem

As discussed in the excellent kaocha source code, these two approaches don't work together, which leads to issues such as kaocha's workarounds (and monkey-patching), leiningen's monkey-patching, Cursive's test runner not always integrating with other tools due to binding report, and others.

Alessandra Sierra wrote about this issue as well when she attempted a follow-up to clojure.test called Lazytest (which I have resurrected over the last year). She ended up not finishing the library, and many other alternatives to Clojure test have not been as successful as one might hope either.

Given that clojure.test is here to stay, it seems prudent to make it easier to use and adapt without changing the fundamental api.

Potential Solution

Separating the "dynamic variable to be rebound" from "the default reporting function" would make it easier for libraries and tools to choose to extend report or replace it entirely, without worrying that they'll clash or run into each other. This can be accomplished by adding a new dynamic variable *reporter* to clojure.test that do-report calls, and defaulting *reporter* to clojure.test/report. (report would stay dynamic so as to not change working code.)

With such a change in place, workarounds such as Leiningen's monkey-patching could simply bind *report* with a new function and not worry about stepping on any other library's toes.

How to suppress/capture the output of defspec for use in clojure.test?

Tue, 06 Aug 2024 20:41:33 +0000

I've been a fan of test.check since the beginning. However, I've never found a convenient way to capture the output when using defspec. When used with clojure.test, one gets output that looks like:

Testing tst.tupelo.y64 {:result true, :num-tests 999, :seed 1722976402694, :time-elapsed-ms 53, :test-var "dospec-line-53"} {:result true, :num-tests 999, :seed 1722976402747, :time-elapsed-ms 36, :test-var "dospec-line-44"}

somewhat polluting the unit test output. How can one capture and/or suppress this output?

cljs.core/test does not work as the docstring describes

Tue, 27 Feb 2024 12:33:17 +0000

cljs.core/test does not work according to its docstring description.

The definition of cljs.core/test is

(defn test
  "test [v] finds fn at key :test in var metadata and calls it,
  presuming failure will throw exception"
  [v]
  (let [f (.-cljs$lang$test v)]
    (if f
      (do (f) :ok)
      :no-test)))

It's intended use, as far as I understand it, is as such (example copied from the documented Clojure version of the same function at ClojureDocs, https://clojuredocs.org/clojure.core/test#example-542692cac026201cdc326b8a ):

(defn my-function
  "this function adds two numbers"
  {:test #(do
            (assert (= (my-function 2 3) 5))
            (assert (= (my-function 4 4) 8)))}
  ([x y] (+ x y)))

(test #'my-function)  ;equal to (test (var my-function))
=> :ok

So, cljs.core/test enables functions to self-document their capabilities via tests made available in metadata under the :test key.

However, repeating the above in a CLJS environment yields a different result:

(defn my-function
  "this function adds two numbers"
  {:test #(do
            (assert (= (my-function 2 3) 5))
            (assert (= (my-function 4 4) 8)))}
  ([x y] (+ x y)))

(test #'my-function)
=> :no-test

The interesting thing is that, given that I evalated the above in a CLJS REPL, the metadata is available, as far as I can tell:

(meta #'my-function)
=>
{:ns cljs.user,
 :name my-function,
 :file "<cljs repl>",
 :end-column 18,
 :source "my-function",
 :column 1,
 :line 1,
 :end-line 1,
 :arglists ([x y]),
 :doc "this function adds two numbers", 
 :test #object[Function]}

This can be made even clearer by substituting the rather opaque function for a keyword:

(defn my-function
  "this function adds two numbers"
  {:test :just-a-keyword}
  ([x y] (+ x y)))

(meta #'my-function)
=>
{:ns cljs.user,
 :name my-function,
 :file "<cljs repl>",
 :end-column 18,
 :source "my-function",
 :column 1,
 :line 1,
 :end-line 1,
 :arglists ([x y]),
 :doc "this function adds two numbers", 
 :test :just-a-keyword}

I spoke to @theller and @hiredman about this on the Clojurians Slack, and they speculated that cljs.core/test has been repurposed to execute the body of deftests, since the two pieces seem to fit together. Observe:

(require '[cljs.test :as test])
(test/deftest foo (test/is (= 1 2)))
=> #'cljs.user/foo

(test foo)
FAIL in () (<NO_SOURCE_FILE>:1:28)
expected: (= 1 2)
  actual: (not (= 1 2))

:ok

The deftest body is executed and the keyword :ok is returned (regardless of the is result), as can also be gathered from reading definition of cljs.core/test.

If it is true that cljs.core/test has been repurposed in this manner, then two things are also true:

A) If it ever were the intent that cljs.core/test should enable testing :test assertions for defns, that no longer holds.
B) The docstring of cljs.core/test is somewhat misleading.

The first bit of the docstring, "test [v] finds fn at key :test in var metadata and calls it", may be technically correct, but mentions no connection with deftest, and therefore somewhat unhelpful.

The second bit of the docstring, ", presuming failure will throw exception", is certainly not correct when used for deftest - the cljs.test/is macro, which I think is fair to say that most deftests use, catches exceptions and reports instead. I think the original intent is that the execption is thrown, so :ok is not returned, but in this context, that construct fails.

The docstring seems to be a carbon copy of the Clojure version, so possibly, it warrants an update. Or maybe cljs.core/test is simply unused and redundant at this point?

Way to set clojure.test/load-tests when starting a REPL

Thu, 02 Feb 2023 14:13:28 +0000

Is there a way to set *load-tests* when starting the REPL?
Some options can be set using system properties, like compiler options, any chance for something similar for clojure.test?

Clojure Q&A - Recent questions tagged clojure.test

Separate bindable reporting function from clojure.test/report

How to suppress/capture the output of defspec for use in clojure.test?

cljs.core/test does not work as the docstring describes

Way to set clojure.test/*load-tests* when starting a REPL

Way to set clojure.test/load-tests when starting a REPL