Welcome! Please see the About page for a little more info on how this works.

0 votes
in test.check by

Inspired in the analog feature that exists in Haskell's QuickCheck. Adds a {{classify}} fn that is intended to be used as a wrapper of {{prop/for-all}}, returning a property (a generator) appropriate for {{test.check/quick-check}} that will augment the result-map returned by the underlying property, adding the collected labels under the {{:labels}} key. Also triggers a new event {{:stats}} in the {{default-reporter-fn}} whose default implementation calls {{test.check.stats/print}}, printing the classification of trials with the following format:

12.7% :lt-30 14.5% :gte-30 29.1% :lt-30, :lt-20 43.6% :lt-30, :lt-10, :lt-20

(note that multiple labels might be assigned to some test cases)

I think it answers the question "How could we collect stats about the sorts of things generated?" from the test.check design page

13 Answers

0 votes
by

Comment made by: nberger

The patches here no longer apply after the introduction of reporter-fn.

While I'm working on new patches, I'd like to know other people's (and especially gfredericks) opinion on having this new :labels concept in test.check. Each trial's result map can optionally include some :labels that are included in the :complete report so the stats can be computed.

An alternative would be to implement the stats as an external library/reporter, but to make that possible we need the trials loop to keep arbitrary state from each trial's result-map where we could accumulate the labels. Or maybe we could make the (reporter-fn :type :trial ...) to include the entire result-map so the reporter-fn has the chance to "save" this state, but this way doesn't look very elegant...

0 votes
by

Comment made by: gfredericks

Things to think about:

  • The stats are only a function of the args, so is it necessary to wrap the entire property?
    -- haskell does this though
  • On the other hand, I like the idea of stats being a particular instance of a generic feature of the {{check}} namespace, but the current implementation doesn't quite achieve that since we have lots of label stuff in the {{check}} namespace
  • Buuuut it's hard to move it out because if {{check}} just exposed a generic reduce feature, then users going directly to {{quick-check}} would have to supply two things -- a wrapped property, and the proper reduce function
0 votes
by

Comment made by: nberger

Uploaded a new patch that takes advantage of the new reporter-fn mechanism.

In this new patch some stuff was simplified:

  • Removed the new * } var that was going to be used as a way to enable/disable printing of stats for each test. The default-reporter-fn will now print the stats only when there are labels present in the result-map. To not print the stats it's just a matter of using a different :reporter-fn that doesn't print the stats.
  • In the stats report only lines with labels are printed now. In the previous version there was a line with the percentage of trials with "No labels". I removed it now to make it less verbose.

Also docstrings were added, made it sure the stats work in clojurescript and other minor improvements.

0 votes
by

Comment made by: nberger

I'm thinking it would be nice to give more ways for assigning labels to trials. For example:

  1. {{(classify prop)}}: It just uses the vector of args as the label.
  2. {{(classify prop label-fn)}}: Obtains the label by applying a {{label-fn}} to the args. Example: {{(classify prop count)}} - assigns the count of the arg (a vector, string, etc) as the label
  3. {{(classify prop label-fn pred)}}: Applies {{label-fn}} only when {{pred}} yields a truthy value. Example: (classify prop count #(> (count %) 1)) - assigns the count of the arg as the label, but only when count is greater than 1.
  4. {{(classify prop pred)}}: Uses the vector of args as the label, but only when {{pred}} yields a truthy value. Example: (classify prop #(<= (count %) 1)) - assigns the count of the arg as the label, but only when count is lower or equal to 1.
  5. {{(classify prop pred label)}}: This is the current signature. Assigns label only when pred yields truthy

So I'm thinking about changing the signature to receive a map of {{(:pred :label-fn :label)}} possible keys. The three keys are optional. {{:label}} and {{:label-fn}} can't be both present at the same time.

From what I could understand, Haskell QuickCheck (https://hackage.haskell.org/package/QuickCheck-2.8.2/docs/Test-QuickCheck-Property.html) has different functions to provide similar alternatives:
{{(classify prop)}} would be similar to {{collect}} in Haskell QC
{{(classify prop label-fn)}} would be similar to {{label}} in Haskell QC
{{(classify prop label-fn pred)}} would be similar to {{classify}} in Haskell QC

What do you think @gfredericks?

0 votes
by

Comment made by: gfredericks

After discussing it, my hunch is that 2 and 5 are the most natural ones, maybe calling 2 {{collect}} to mirror the haskell version.

We talked about maybe treating {{nil}} as a flag for not labelling in {{collect}}, but I don't particularly like that idea I don't think.

0 votes
by

Comment made by: gfredericks

I should also not forget that this might overlap with changes to address the "Test Failure Feedback" issue on the confluence page: http://dev.clojure.org/display/design/test.check

0 votes
by

Comment made by: gfredericks

I should also not forget that this might overlap with changes to address the "Test Failure Feedback" issue on the confluence page: http://dev.clojure.org/display/design/test.check

0 votes
by

Comment made by: nberger

bq. We talked about maybe treating nil as a flag for not labelling in collect, but I don't particularly like that idea I don't think.

Perhaps we can use a namespaced keyword to signal that a label should be ignored? Something like {{clojure.test.check.stats/ignore}}. This way we can easily implement {{classify}} in terms of collect by creating a function that returns {{stats/ignore}} when pred doesn't match:

`
(defn collect
[prop label-fn]
(gen/fmap

(fn [{:keys [args] :as result-map}]
  (let [label (apply label-fn args)]
    (if (= ::ignore label)
      result-map
      (update result-map :labels conj label))))
prop))

(defn classify
[prop pred label]
(collect prop (fn [& args]

              (if (apply pred args)
                label
                ::ignore))))

`

Another option could be to add an extra arity in {{collect}}, to receive a flag about whether nil should be treated as a label or if it should be ignored. I like {{:stats/ignore}} more

0 votes
by

Comment made by: nberger

Replaced with new patch that adds both {{stats/classify}} & {{stats/collect}} as discussed. I think this new patch implements what was discussed, covering cases 2 & 5 with function names analog to the haskell impl, leaving out any special treatment for nil (it is a valid label) and not adding {{:stats/ignore}} as I suggested in a previous comment.

0 votes
by

Comment made by: nberger

Added a new patch TCHECK87-add-stats-feature-2.patch that is rebased on current master. I don't think it's the final version (I'd like to find a way to not pollute the main quickcheck loop with the labels stuff for example) but it's hopefully getting closer.

0 votes
by

Comment made by: nberger

Added new patch TCHECK87-add-stats-feature-3.patch rebased on current master with some squashed commits. Also fixes the print-stats test in ClojureScript.

0 votes
by
Reference: https://clojure.atlassian.net/browse/TCHECK-87 (reported by nberger)
0 votes
by

Is anything holding back this patch?

...