Metadata for tuning on generated values

Question

It can be really useful for optimization of generative testing(with specs) if values outputted by generators included attempts and time as metadata.
Example:

(-> (s/exercise string? 1)
    first
    meta)
;; => {:time-ms 2, :attemts 1}

It can help in finding slowest generators which can benefit the most from customization.

Answer 1

_Comment made by: alexmiller_

I move this to test.check as that’s where the generators live.

Note that not every value can carry metadata (number, strings, etc) but it’s an interesting idea.

Answer 2

_Comment made by: jare_

> Note that not every value can carry metadata (number, strings, etc) but it’s an interesting idea.

It's not that big of a deal since the metadata will be directly useful mostly with recursive/nested specs for spec tooling devs. And a person that wants to use it for this can work around the limitation by wrapping primitives.

For the end-user API something similar to spec/explain(s/explain-data, s/explain-str..) would be great.
I'll call it s/complexity, the function takes a spec and returns (in the case of s/complexity-data) structure of the same shape as the original spec filled with generation complexity data (average time, attempts, mb the spec form...) It should handle too complex specs in a reasonable way i.e. Instead of throwing (as with generators) it preferably reports which part of the spec is too complex and what is individual complexity of its elements.

Answer 3

Comment made by: jare

Also s/complexity probably should have per node(spec element) timeout (mb configurable) since it's mostly debugging/profiling tool.

Answer 4

Comment made by: gfredericks

What does "attempts" mean here? Does it only apply to {{gen/such-that}} generators?

Answer 5

Comment made by: jare

Yeah I in the context of specs. How many times generator had to retry before it managed to satisfy spec.

Answer 6

Comment made by: gfredericks

I just noticed the {{:time-ms}} aspect of this, which would presumably apply to a lot more than just {{such-that}}.

I agree that being able to easily debug generator performance would be useful, but I don't think it's obvious how this approach applies to some of the combinators, like {{gen/fmap}} and {{gen/bind}}. I'd have an easier time considering a more concrete proposal, that includes details for those kinds of cases.

Answer 7

Comment made by: jare

bq. but I don't think it's obvious how this approach applies to some of the combinators, like gen/fmap

The metadata shouldn't be available from withing the function that {{gen/fmap}} applies. Instead the time spent inside the function and other performance stats should be assoc-ed with the {{gen/fmap}} inputs stats and then returned attached to the outputs. So the data will be carried by the context between nested {{gen/}} calls.

bq. The metadata shouldn't be available from withing the function that {{gen/fmap}} calls.

Alternatively you can allow to peak into the inputs stats but not alter it. Not sure about that. It may be too confusing.

Answer 8

_Comment made by: jare_

Actually if you allow peeking it might be used not only for debugging but also to alter the generation for performance.
For amortisation of the cumulative generator complexity, to meet some deadline by switching routes.

- - - - -

@gfredericks For some reason I can't add a new comments to any issue so I update the old one.

For:


(def g1 ...)

(def g2 (gen/fmap f g1))


 _complexity_ function should output



(complexit g1) => {:name <generator-name> :time-ms 10}

(complexit g2) => {:name :fmap :time-ms 5 :args [{:name <generator-name> :time-ms 10}]}


here _:time-ms 5_ is time spent inside _f_ function.

Answer 9

Comment made by: gfredericks

For example, let's say I have:

`
(def g1 ...)

(def g2 (gen/fmap f g1))
`

When I generate a value from {{g2}}, what kind of information will I get? Will it be about {{g1}}, or about {{f}}, or about both somehow?

Answer 10

Comment made by: gfredericks

(link: ~JAre) I believe your comment permissions are fixed.

Okay, I can see that it might be a well-defined concept. It would be a non-trivial amount of work, though, and I'd want to be sure that there wouldn't be a major performance difference. If there is, we'd probably want a way to have it off by default, and make sure that there's not a performance hit even when off.

Answer 11

Comment made by: jare

Yeah works now. Thx.

Answer 12

Reference: https://clojure.atlassian.net/browse/TCHECK-153 (reported by alex+import)