Disable fspec validation during instrumentation

Question

Problem statement: Enable instrumentation, and invoke a speced function with a lambda. To validate the lambda, spec tests it with generative testing. This results in the lambda being invoked multiple times. If the lambda launches a missile, many missiles are now launched by spec. There are many scenarios in which this is not acceptable because it can for example crash the environment.

Current solutions:
- Don't spec the lambda. Disadvantage: Spec can't generate it in contexts where its spec is referred.
- Set fspec-iterations to 0. Disadvantage: Disables all validation of all lambdas.
- ???

Ideas:
- An fspec flag to disable the generative testing of its validation.

Answer 1

Comment made by: alexmiller

Another option that has been proposed for this is to make the instrumented function also wrap the function arg in instrumentation according to its spec.

Answer 2

Comment made by: lgs32a

@Alex Miller: Yes, I thought about this as well and believe it would be more consistent with how instrumentation works in regard to functions, i. e. they are checked at invocation time.

However I have not proposed it, because I don't see how we should do it. Spec would have to be able to generically replace all functions that are passed in any arg anywhere with instrumented ones, but also have to know which specs to use. How?

Answer 3

Comment made by: lgs32a

One possible approach would be to implement "descriptive walking" in spec as internal or even public enhancement.
A spec-walk feature would work like prewalk/postwalk, but it takes a spec and a value and invokes the user provided function with both a (sub)value and its corresponding (sub)spec. Instrument wrapper could then replace values that are fspeced with instrumented fns generically. Every spec that composes other specs would have to implement walking over its children and their specs as a new interface method.

Answer 4

Comment made by: alexmiller

Yes, would need something like this (see CLJ-2208 for ticket re spec walking).

Answer 5

Comment made by: lgs32a

@Alex Miller: CLJ-2208 won't do alone. We need to be able to generically walk/replace any given data structure using a spec describing its shape. Should I create a separate ticket and outline a few approaches to get things going? Or should that go here for now?

Answer 6

Reference: https://clojure.atlassian.net/browse/CLJ-2217 (reported by lgs32a)