Welcome! Please see the About page for a little more info on how this works.

0 votes
in Spec by
h3. Problem

using {{clojure.spec}} in runtime border validation supporting multiple exchange formats is hard.

h3. Details

Currently in clojure.spec (alpha-14), conformers are attached to Spec instances at creation time and they are invoked on every conform. This is not very useful in system border validation, where conforming/coercion functions should be selected based on runtime data, e.g. the exchange format.


* a {{keyword?}} spec:
** with EDN, no coercion should be done (it can present Keywords)
** with JSON, String->Keyword coercion should be applied
** with String-based formats (CSV, query-params, ...), String->Keyword coercion should be applied

* a {{integer?}} spec:
** with EDN, no coercion should be done (it can present numbers)
** with JSON, no coercion should be done (it can present numbers)
** with String-based formats (CSV, query-params, ...), String->Long coercion should be applied

Here is a more complete example:

(s/def ::id integer?)
(s/def ::name string?)
(s/def ::title keyword?)
(s/def ::person (s/keys :opt [::id], :req-un [::name ::title]))

;; this is how we see the data over different exchange formats
(def edn-person {::id 1, :name "Tiina", :title :boss})
(def json-person {::id 1, :name "Tiina", :title "boss"})
(def string-person {::id "1", :name "Tiina", :title "boss"})

;; here's what we want
(def conformed-person edn-person)

To use this today, one needs to manually create new border specs with different conformers for all different exchange formats. Non-qualified keywords could be mapped in {{s/keys}} to work (e.g. {{::title}} => {{::title$JSON}}), but this wont work if fully qualified keys are exposed over the border (like {{::id}} in the example) - one can't register multiple, differently conforming version of the spec with same name.

h3. Suggestion

Support selective conforming in the Spec Protocol with a new 3-arity {{conform*}} and {{clojure.spec/conform}}, both taking a extra user-provided callback/visitor function. If the callback is provided, it's called from within the Specs {{conform*}} with the current spec as argument and it will return either {{nil}} or a 2-arity conformer function that should be used for the actual confrom.

Actual conforming-matcher implementations can be maintained in 3rd party libraries, like spec-tools[1].

Using it would look like this:

;; edn
(assert (= conformed-person (s/conform ::person edn-person)))
(assert (= conformed-person (s/conform ::person edn-person nil)))

;; json
(assert (= conformed-person (s/conform ::person json-person json-conforming-matcher)))

;; string
(assert (= conformed-person (s/conform ::person string-person string-conforming-matcher)))

h3. Alternative

Another option to support this would be to allow Specs to be extended with Protocols. 3rd party libs could have a new {{Conforming}} protocol with 3-arity {{conform}} and add implementations for it on all current specs. Currently this is not possible.

[1] https://github.com/metosin/spec-tools

16 Answers

0 votes

Comment made by: alexmiller

I don't think we are interested in turning spec into a transformation engine via conformers, so I suspect we are probably not interested. However, I'll leave it for Rich to assess.

0 votes

Comment made by: ikitommi

Currently, Plumatic Schema is the tool used at the borders. Now, people are starting to move to Spec and it would really bad for the Clojure Web Developement Story if one had to use two different modelling libraries for their apps. If Spec doesn't want to be a tranformation engine via conformers, I hope for the Alternative suggestion to allow 3rd parties to write this kind of extensions: exposing Specs as Records/Types instead of reified protocols would do the job?

0 votes

Comment made by: kenrestivo

I could see why the Clojure core developers might not want Spec to support this kind of coercion, but the practical reality is that someone will have to. If it isn't in Spec itself, it'll have to be done libraries built upon it like Tommi's.

The use case here is: I have a conf file that is YAML. I'm parsing the YAML using a Clojure library, turning it into a map. Now I have to validate the map, but YAML doesn't support keywords, for example, and the settings structure goes directly into Component/Mount/etc as part of the app state, so it makes sense to run s/conform on it as the first step in app startup after reading configuration. Add to this the possibility of other methods of merging in configuration (env vars, .properties files, etc) and this coercion will be necessary somewhere.

0 votes

Comment made by: ikitommi

Any news on assessing this? I would be happy to provide a patch or a link to a modified {{clojure.spec}} with samples on usage with the 3-arity conform in it. Some thinking aloud: http://www.metosin.fi/blog/clojure-spec-as-a-runtime-transformation-engine/

0 votes

Comment made by: alexmiller

Rich hasn't looked at it yet. My guess is still that we're not interested in this change. While I think some interesting problems are described in the post, I don't agree with most of the approaches being taken there.

0 votes

Comment made by: sbelak

Why not just use s/or (or s/alt) and then dispatch on the tag. Something like:

(s/def ::id (s/and (s/or :int integer?

                     :str string?)
               (s/conformer (fn [[tag x]]
                              (case tag
                                :int x
                                :str (Integer/parseInt x))))))


I use that pattern quite a bit in https://github.com/sbelak/huri and with a bit of syntactic sugar it works quite well.

0 votes

Comment made by: imre

Simon that will not work if you are trying to conform to specs from third parties though. One of the points of this suggestion is that third parties would be able to write their own conformers to existing specs without redefining those specs.

0 votes
_Comment made by: ikitommi_

Thanks for the comments. I would be happy to provide a patch / sample repo with the changed needed for this, in hope that it would help to decide if this could end up in the spec or not. What do you think?

Below is a sample of initial spec-integration into ring/http libs, using spec-tools. For now, one needs to wrap specs into spec records to enable the 3-arity conforming. This is boilerplate I would like to see removed. With this change, it should work out-of-box for all (3rd party) specs.

(require '[compojure.api.sweet :refer :all])
(require '[clojure.spec.alpha :as s])
(require '[spec-tools.core :as st])

;; to enable 3-arity conforming
(defn enum [values]
  (st/spec (s/and (st/spec keyword?) values)))

(s/def ::id int?)
(s/def ::name string?)
(s/def ::description string?)
(s/def ::size (enum #{:L :M :S}))
(s/def ::country (st/spec keyword?) ;; to enable 3-arity conforming
(s/def ::city string?)
(s/def ::origin (s/keys :req-un [::country ::city]))
(s/def ::new-pizza (st/spec (s/keys :req-un [::name ::size ::origin] :opt-un [::description])))
(s/def ::pizza (st/spec (s/keys :req [::id] :req-un [::name ::size ::origin] :opt-un [::description])))

;; emits a ring-handler with input & output validation (& swagger-docs)
;; select conforming based on request content-type (e.g. json/edn) + strip-extra keys from maps
(context "/spec" []
    {:coercion :spec
     :parameters {:body-params ::new-pizza}
     :responses {200 {:schema ::pizza}}
     :post {:handler (fn [{new-pizza :body-params}]
                       (ok (assoc new-pizza ::id 1))}}))
0 votes

Comment made by: ikitommi

Intended to create internal PR in my fork of clojure.spec, but ended up doing a real DUMMY PR for the actual repo. Well, here it is anyway:


Happy to finalize & create a patch into Jira if this goes any further.

0 votes
_Comment made by: ikitommi_

comments welcome. here's a sample test for it:

(deftest conforming-callback-test
  (let [string->int-conforming
        (fn [spec]
          (condp = spec
            int? (fn [_ x _]
                     (int? x) x
                     (string? x) (try
                                   (Long/parseLong x)
                                   (catch Exception _
                     :else ::s/invalid))
            :else nil))]

    (testing "no conforming callback"
      (is (= 1 (s/conform int? 1)))
      (is (= ::s/invalid (s/conform int? "1"))))

    (testing "with conforming callback"
      (is (= 1 (s/conform int? 1 string->int-conforming)))
      (is (= 1 (s/conform int? "1" string->int-conforming))))))
0 votes

Comment made by: ikitommi

initial work as patch.

0 votes

Comment made by: ikitommi

Any news on this?

0 votes

Comment made by: ikitommi

Related https://dev.clojure.org/jira/browse/CLJ-2251

0 votes

Comment made by: marco.m

Hello, any news ?

0 votes
_Comment made by: alexmiller_

We won’t look at this at least until we do the next batch of implementation changes. I continue to think we will most likely decline this.