TL;DR
When instrumentation is turned on, spec validates a fspec that appears in the :args of another fspec by calling the actual function with generated arguments, which if it performs side effects could lead to unexpected and incorrect results.
The problem
If we have the following functions and specs
(defn create-generator []
(let [counter (atom [])]
(fn [& _]
(swap! counter inc))))
(s/def ::my-generator (s/fspec :args (s/cat :identifier-arguments (s/* string?))
:ret integer?))
(defn my-fn [generator]
(generator))
(s/fdef my-fn
:args (s/cat :my-context ::my-generator)
:ret string?)
Calling (my-fn (create-generator)) will return different results depending on instrumentation being enabled or not.
With instrumentation enabled, our function is being called 21 times more than the single invocation we'll be expecting (hint: this numbers comes from clojure.alpha.spec/fspec-iterations).
This happens because my-fn's spec has args, and in such case clojure.alpha.spec.test/instrument does the following:
If a var has an :args fn-spec, sets the var's root binding to a fn
that checks arg conformance (throwing an exception on failure) before
delegating to the original fn.
This means it must check if the function received as argument conforms the ::my-generator spec, which is itself an fspec. For that it calls the fspec reified conform* and that implementation runs a quick-check style test, generating random arguments based on the :args spec and calling the actual function. This can be observed by creating an exception in the function returned by create-generator, the stacktrace looks like this
'[[clojure.test_clojure.fspec_bug_test$create_generator$fn__5468 doInvoke "fspec_bug_test.clj" 4]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.core$apply invokeStatic "core.clj" 665]
[clojure.core$apply invoke "core.clj" 660]
[clojure.alpha.spec.impl$call_valid_QMARK_ invokeStatic "impl.clj" 1616]
[clojure.alpha.spec.impl$call_valid_QMARK_ invoke "impl.clj" 1612]
[clojure.alpha.spec.impl$validate_fn$fn__4117 invoke "impl.clj" 1627]
[clojure.lang.AFn applyToHelper "AFn.java" 154]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.core$apply invokeStatic "core.clj" 665]
[clojure.core$apply invoke "core.clj" 660]
[clojure.test.check.properties$apply_gen$fn__5403$fn__5404 invoke "properties.cljc" 31]
[clojure.test.check.properties$apply_gen$fn__5403 invoke "properties.cljc" 30]
[clojure.test.check.rose_tree$fmap invokeStatic "rose_tree.cljc" 77]
[clojure.test.check.rose_tree$fmap invoke "rose_tree.cljc" 73]
[clojure.test.check.generators$fmap$fn__4871 invoke "generators.cljc" 104]
[clojure.test.check.generators$gen_fmap$fn__4845 invoke "generators.cljc" 59]
[clojure.test.check.generators$call_gen invokeStatic "generators.cljc" 43]
[clojure.test.check.generators$call_gen invoke "generators.cljc" 39]
[clojure.test.check$quick_check invokeStatic "check.cljc" 211]
[clojure.test.check$quick_check doInvoke "check.cljc" 59]
[clojure.lang.RestFn invoke "RestFn.java" 425]
[clojure.lang.AFn applyToHelper "AFn.java" 156]
[clojure.lang.RestFn applyTo "RestFn.java" 132]
[clojure.core$apply invokeStatic "core.clj" 665]
[clojure.core$apply invoke "core.clj" 660]
[clojure.alpha.spec.gen$quick_check invokeStatic "gen.clj" 32]
[clojure.alpha.spec.gen$quick_check doInvoke "gen.clj" 30]
[clojure.lang.RestFn invoke "RestFn.java" 421]
[clojure.alpha.spec.impl$validate_fn invokeStatic "impl.clj" 1628]
[clojure.alpha.spec.impl$validate_fn invoke "impl.clj" 1623]
[clojure.alpha.spec.impl$fspec_impl$reify__4124 conform_STAR_ "impl.clj" 1646]
[clojure.alpha.spec$conform invokeStatic "spec.clj" 245]
[clojure.alpha.spec$conform invoke "spec.clj" 237]
[clojure.alpha.spec$conform invokeStatic "spec.clj" 241]
[clojure.alpha.spec$conform invoke "spec.clj" 237]
[clojure.alpha.spec.impl$dt invokeStatic "impl.clj" 219]
[clojure.alpha.spec.impl$dt invoke "impl.clj" 214]
[clojure.alpha.spec.impl$dt invokeStatic "impl.clj" 215]
[clojure.alpha.spec.impl$dt invoke "impl.clj" 214]
[clojure.alpha.spec.impl$deriv invokeStatic "impl.clj" 1426]
[clojure.alpha.spec.impl$deriv invoke "impl.clj" 1420]
[clojure.alpha.spec.impl$deriv invokeStatic "impl.clj" 1434]
[clojure.alpha.spec.impl$deriv invoke "impl.clj" 1420]
[clojure.alpha.spec.impl$re_conform invokeStatic "impl.clj" 1564]
[clojure.alpha.spec.impl$re_conform invoke "impl.clj" 1555]
[clojure.alpha.spec.impl$as_regex_spec$fn__3849 invoke "impl.clj" 1238]
[clojure.alpha.spec.protocols$eval1958$fn__2056$G__1939__2067 invoke "protocols.clj" 11]
[clojure.alpha.spec$conform invokeStatic "spec.clj" 245]
[clojure.alpha.spec$conform invoke "spec.clj" 237]
[clojure.alpha.spec$conform invokeStatic "spec.clj" 241]
[clojure.alpha.spec$conform invoke "spec.clj" 237]
[clojure.alpha.spec.test$spec_checking_fn$conform_BANG___4398 invoke "test.clj" 131]
[clojure.alpha.spec.test$spec_checking_fn$fn__4400 doInvoke "test.clj" 150]
[clojure.lang.RestFn invoke "RestFn.java" 408]
[clojure.test_clojure.fspec_bug_test$eval5473 invokeStatic "fspec_bug_test.clj" 2]
[clojure.test_clojure.fspec_bug_test$eval5473 invoke "fspec_bug_test.clj" 44]
...]
Of course, the problem can be reduced to the fspec implementation of conform* since by just calling s/valid? will show the behavior of the function being called, but this is uncommon, while having functions annotated and turning on instrumentation during tests is a common practice, that's why the example in this PR shows a concrete problematic scenario with the current implementation of fspec.
; This already shows the actual function being called, but is an unrealistic example
(s/valid? ::my-generator (fn [& args]
(println args)
1))
