Is interop with clojure.lang.RT an anti-pattern in clojure? / consider adding iter to clojure.core

Question

I've seen several occurrences of code using clojure.lang.RT/iter to retrieve an iterator for a Clojure collection.

(iterator-seq (clojure.lang.RT/iter [1 2 3]))

See e.g.:

https://github.com/noprompt/meander/commit/d2310daaa485afb4e15ceda72aa57f97ea90f284

and:

https://github.com/wilkerlucio/cljc-misc/blob/bb3c8016cace18db5caa5fe0aa5df7a507935f8d/src/main/com/wsscode/misc/coll.cljc#L262

For compatibility of babashka with Clojure I would like to know if I should expose clojure.lang.RT. I'm not entirely convinced that it should be, as clojure.lang.RT is probably an implementation detail of Clojure.

Instead, maybe Clojure can expose the RT/iter method as a clojure.core function?

I "grasped" my local .m2 for usages of clojure.lang.RT and excluded clojure itself:

[clojure.lang.RT/loadClassForName 15]
[clojure.lang.RT/iter 5]
[clojure.lang.RT/loadLibrary 4]
[clojure.lang.RT/assoc 4]
[clojure.lang.RT/classForName 3]
[clojure.lang.RT/load 2]

Note that ClojureScript has an iter function:

$ plk
ClojureScript 1.10.597
cljs.user=> (iter [1 2 3])
#object[cljs.core.RangedIterator]
cljs.user=>

$ clj
Clojure 1.10.1
user=> (iter [1 2 3])
Syntax error compiling at (REPL:1:1).
Unable to resolve symbol: iter in this context

Answer 1

I agree with @borkdude, would be great to have a core function for this common item in the language, which would make it more portable.

Not as a big deal, but in CLJS there is also the transformer-iterator, which creates a new iterator by applying transducers to one. Would be nice to have feature parity on this as well.

Comments

Nice to have *why*? What is the problem at hand?

---

The `transformer-iterator` is more particular and could be avoided. I brought it because I ended up using it in my CLJS implementation. But thinking about it, an `eduction` works pretty much the same for my case, and I can stick with it.

---

eductions encapsulate the iteration and give you something safe to pass between threads so are definitely preferred.

Answer 2

RT should be considered internal implementation and should not be called directly.

Iterators, in general, are very un-clojurey. They are stateful and generally not concurrency friendly. Clojure leans on them internally when it can constrain their use inside some other call (transducer contexts in particular).

(iterator-seq (clojure.lang.RT/iter [1 2 3])) seems bad and (seq [1 2 3]) seems better in several ways. If there is some use case that makes is driving people to want to make iterators, I'm interested in that problem, but I don't see it here.

Comments

This was just a trivial example of creating an consuming an interator, not an actual useful example.  The projects I linked to in the original post needed this function. I'll ask the lib authors to clarify their use cases.

---

Neither of the two linked pieces of code above shows a use of iterator - they are both about building an iterator function neither of which is ever called in those projects. Who uses these? why?

---

The main case I need iterators is to create a custom map type.

The exact detail is how to implement a custom `keys` for the type.

The way Clojure does keys is by first triggering `seq` on the type, and also send the type as you can see here: https://github.com/clojure/clojure/blob/57f7c6ea164271d92cec5a12d22043c7029e3de2/src/jvm/clojure/lang/APersistentMap.java#L147-L154

This implementation requires the type to also implement `Iterable`, which must return an `iterator`.

This is where I mostly need the `iter` function that is only available in `RT`, because there is an underlying structure and I'm re-using its iterator.

The current solution to make it cross platform is:

```
(defn iterator
  "CLJC utility to get an iterator from the collection."
  [coll]
  #?(:clj
     (if (instance? java.lang.Iterable coll)
       (.iterator ^java.lang.Iterable coll)
       (.iterator (seq coll)))

     :cljs
     (iter coll)))
```

So although I don't really want to use an iterator, this case forces me to.

---

There may be reasons to reach into Clojure internals when implementing low-level types (and indeed that's why they are available and not locked away in some private method). In this case, I think your code here is perfectly fine for the jvm impl. I do not think `iterator` is a function that should be widely used or promoted, or that it should be added to the Clojure API.

---

But given CLJS has `iter`, you think would be that bad to make this parity? The only reason I know about that second way is that I was lucky to find it in other people's code.

Also, that implementation doesn't seem to cover things as good as `iter` from RT, there are special cases for handling maps and strings there (as there are in the `iter` from CLJS).

The code for `iter` in `RT`: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java#L571-L614

This is why I believe Clojure should have the `iter` on core too, to avoid these lower coverage implementations (like the one I sent). What you think?

---

No, I don't think that's a good idea. I don't think it's a good idea that CLJS has `iter`. :)

You're bringing me an answer. Bring me more problems.

---

> they are both about building an iterator function neither of which is ever called in those projects

Alex, I want to clarify that this is incorrect. Meander does, in fact, use the `iter` function via generated macro code to implement some semantics a bit more easily (simpler code generation) in some situations. While I've since realized an opportunity to implement this without `iter`, at the time I didn't have that solution and `iter` solved my problem.

You acknowledge these cases exist and this is the reason some Clojure internals haven't been "locked away". You acknowledge the provided JVM implementation of `iter` given by Wilker is "perfectly fine". Yet you don't think it should be part of the public API or a good idea but you don't give a clear reason why these are your thoughts.

> I do not think `iterator` is a function that should be widely used or promoted

I agree. Many functions in core exist which are not widely used nor promoted but nevertheless are there for their utility. In other words, how promotable or how wide the use case is can't be the reason for excluding `iter`.

> Iterators, in general, are very un-clojurey.

This is an exclusive, essentialist position. Its like, your opinion, man. :^)

Clojure includes numerous opportunities to use "stateful and generally not concurrency friendly" things in the core library and, to your point, makes those things accessible on purpose. So whatever "clojurey" includes, from where I'm sitting, it seems include those "un-clojurey" things.

Now, let me ask, what is the arbitrary criteria a function must meet before "should" be considered?

---

Just stepping way back here, deciding what goes in the core api / language is a matter of collecting problems people are encountering and sifting all those to find patterns and deciding based on frequency, severity, level of workarounds, etc which problems are the most important to address.

The best way to provide input to that process is describe as clearly as possible the problem you have. When you ran into this in Meander, why did you want an iterator? What was the actual problem at hand? What other things did you consider? Why weren't they sufficient?

Wilker's description of needing to implement the iterator method of his own collection type with a backing coll is a useful example. Adding `iter` to core is one possible solution. Another is to write your own function using existing Java apis, as he did. Another might be to encapsulate the general need to build custom collection types into something inside Clojure, or in a library. etc. I do note that the scope of need here is not common (most people do not create custom collection types), he solved his problem without having an `iter` function using available Clojure/Java APIs, and I have no other examples of someone with the same problem (but maybe they exist).

My general bias is against iterators. iterators are dangerous and imperative in ways seqs and reducibles are not and it's no accident that it's not part of the Clojure API. This is not just my opinion, it's Rich's opinion which I am transmitting. He's written/talked about this in lots of places:

* https://clojure.org/about/functional_programming#_extensible_abstractions
* https://clojure.org/reference/sequences
* https://clojure.org/news/2012/05/08/reducers
* https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/ClojureIntroForLispProgrammers.md

Also, we had extensive discussions about this during the design and implementation of transducers, which lean heavily on iterators in the implementation but encapsulate their use for safety.

Even given all that, I don't think it's off the table, but the level of need does not seem high enough to overcome that bias to me so far. If you want to change my mind, bring me problems.

---

Everything below is bracketed in "I have since derived a newer solution which does not rely on `iter` but at the time I came to `iter` to solve my problem it did what I needed it to."

> When you ran into this in Meander, why did you want an iterator?

Being pedantic, I did not *want* an iterator initially but it ended up "solving" my problem which I will detail below.

>  What was the actual problem at hand?

There few intersecting pieces here. Meander has a concept of a pattern substitution which you can think of as sort of the dual to pattern matching. Matching applies a pattern to an object and returns a set of bindings, substitution applies bindings to a pattern and returns an object. The compilation of substitution patterns — in particular repeated patterns containing a type of variable known as a "memory variable" — is where this problem arises.

A memory variable is a variable that is bound to a fifo. Each occurrence of a memory variable in a pattern "disperses" or the first value from the fifo and updates the binding of the variable to the remaining elements in the fifo. This continues until the fifo is empty and is said to be "exhausted". When memory variables appear in a repeated pattern (think Kleene star), then the repeated pattern produces values while at least one of the memory variables is not exhausted. When repeated patterns are compiled to Clojure code, there is a check to see if the pattern contains memory variables. If so, it then compiles iterators for each of the memory variables each of which can be tested for exhaustion via `hasNext` and values can be dispersed with, of course, `next`. While this is not an ideal solution, the semantics I wanted where easy to implement this way.

> What other things did you consider?

Initially, I tried a pure solution but this lead to a lot complexity and headache. This has more to do with the poor quality of the design which relies on `iter` though and less with a pure approach.

I also tried `volatile!` but this produced incorrect results in some cases.

Its been a while but I think considered using an array too but `iter` was there and did what I wanted it to so I didn't bother with it.

> Why weren't they sufficient?

Because either there were complexities that as a maintainer I didn't wish to occupy myself with, or because they resulted in an incorrect implementation of the semantics.

---

>  I don't think it's a good idea that CLJS has `iter`.

This is the heart of the issue. CLJS has always been treated sort of like a dialect of Clojure resulting in many situations like this one. While I understand there is some bias against formally specifying parts of the language, having at least some minimal definition of primitives and what symbols are resolvable by default in an empty namespace could serve to mitigate these issues for future implementations. Personally, I think the `ns` macro is part of the problem because it is pervasive (`in-ns` is more primitive but rarely used; CLJS requires it), and it refers core, the contents of which are arbitrarily based on what the implementation thinks should be there; issues like this one are a natural consequence.

If a future design of the language included a minimal definition of primitives such that clean separation of platform specific symbols could be achieved easily, these issues of disparity between CLJ and CLJS or other implementation would likely vanish.

FWIW I'm also speaking from experience. Over the past 5 years I have *partially* implemented a Clojure interpreter/compiler for Ruby 2.X.X, a small step symbolic interpreter for Clojure, and a partial evaluator for Clojure. In each case I was never able to complete the projects for lack of a clear language definition both in syntax and semantics. For example, there are special forms like `case*` which are not documented and require inspection of the compiler to determine their semantics.

Anyway, I make no demands here. I only wish to share my observations. I think there is a possible future for Clojure where issues like this do not exist. As someone who has programmed in the language professionally and as a hobby for nearly a decade, I think some specification of the language would pave the way to that future.

---

I cannot speak for the core Clojure team that develops and maintains Clojure, but given that others have raised the question of a specification for Clojure several times in the last decade in various public forums, I suspect that (a) the Clojure core team has no interest in writing such a specification, other than the doc strings included in the implementation and the many pages of official documentation on clojure.org today (and the ones still planned to be written, whatever those are), and (b) if that is true, the only specification that would ever be written would be created by someone who is NOT on that team, and the Clojure core team would have no reason to follow it.