Many apis (elasticsearch, github, s3, etc) have parts of the api
which, in usage, end up being used in an interative way. You make an
api call, and you use the result to make another api call, and so
on. This most often shows up in apis have some concept of pages of
results that you page through, and is very prevalent in http apis.
This appears to be such a common pattern that it would be great if
Clojure had in built support for it.
You may think Clojure already does have support for it, after all,
Clojure has `iterate`. In fact the docstring for `iterate`
specifically says the function you give it must be free of side
effects.
I propose adding a function `unfold` to clojure.core to support this
use case. `unfold` would return an implementation of ReduceInit. The
name `unfold` matches what would be a similar Haskell function
(
https://hackage.haskell.org/package/base-4.8.2.0/docs/Data-List.html#v:unfoldr)
and also matches the name for a similar function used in some existing
Clojure libraries
(
https://github.com/amalloy/useful/blob/develop/src/flatland/useful/seq.clj#L128-L147).
`unfold` in some ways looks like a combination of `take-while` and
`iterate`, except for the fact that `iterate` requires a pure
function. Another possible solution would be a version of `iterate`
that doesn't require a pure function.
It seems like given the use case I envision for `unfold`, a
non-caching reducible would be perfect. But that would leave those
that prefer seqs high and dry, so maybe at least some consideration
should be given to seqs.
Mailing list discussion is here
(
https://groups.google.com/forum/#!topic/clojure-dev/89RNvkLdYc4)
A sort of dummy api you might want to interact with would look something like
(import '(java.util UUID))
(def uuids (repeatedly 1000 #(UUID/randomUUID)))
(def uuid-index
(loop [uuids uuids
index {}]
(if (seq uuids)
(recur (rest uuids) (assoc index (first uuids) (rest uuids)))
index)))
(defn api
"pages through uuids, 10 at a time. a list-from of :start starts the listing"
[list-from]
(let [page (take 10 (if (= :start list-from)
uuids
(get uuid-index list-from)))]
{:page page
:next (last page)}))
given the above api, if you had an implementation of `unfold` that took a predicate that decided when to continue unfolding, a producer which given a value in a sequence produced the next value, and an initial value, you could do something like this:
(= uuids (into [] (mapcat :page) (unfold :next (comp api :next) (api :start))))
and the result would be true.
The equivilant take-while + iterate would be something like:
;; the halting condition is not strictly the same
(= uuids (into [] (mapcat :page) (take-while (comp seq :page) (iterate (comp api :next) (api :start)))))
