Test output is misleading for `(is false)`

Question

Given the following code:

(deftest a-test
  (is false))

running the tests produces the following output:

FAIL in (a-test) (form-init1218676551285641018.clj:28)
expected: false
  actual: false

The behaviour is correct.

This error message is very confusing for a few reasons:

• There is no expect value in this case. I'm not using the (is (= expected actual))
• The "expected" and the "actual" as printed, are the same value, yet the test fails.

Answer 1

If you use a var or expression that produces false, it becomes a bit more obvious what is going on here:

user=> (def v false)
#'user/v
user=> (t/deftest foo (t/is v))
#'user/foo
user=> (foo)

FAIL in (foo) (NO_SOURCE_FILE:4)
expected: v
  actual: false
nil
user=> 

The expected: line is essentially an abbreviation for "I expected this expression to yield truthy" so in your example it is saying " I expected false to yield truthy" (but the actual value was false).

clojure.test could certainly produce better output -- much of the original work done on Jay Fields' Expectations library and Paul Stadig's clojure.test extension Humane Test Output are attempts to produce better error messages when assertions fail.

When I extracted my clojure.test-compatibility work from Expectations and turned it into a standalone library https://github.com/clojure-expectations/clojure-test one of the things I lost was the really nice failure messages that Expectations produced because I switched from Expectations' reporting engine to clojure.test's reporting engine. I've talked to Alex Miller about the possibility of clojure.test being split out of Clojure "core" at some future point so it could be worked on separately and I've expressed an interest in taking over maintenance of it... I don't know whether that will happen or what the timeline would be.

Answer 2

Do you have any examples of more useful (is ...) test expressions that produce similarly confusing output?

It seems unlikely that the Clojure developers would want to make changes to the clojure.test library if the only example is (is false).

Comments

No, that's the only case that I came across.