clojure.java.data/from-java throws stack overflow

Question

I use the following function to get scrollpoints from a client io.qdrant.client.QdrantClient

(def client ...) => io.qdrant.client.QdrantClient

(defn make-scroll-points [collection-name k v]
  (builder/to-java Points$ScrollPoints
                   Points$ScrollPoints$Builder
                   (Points$ScrollPoints/newBuilder)
                   {:collectionName collection-name
                    :filter (builder/to-java Points$Filter
                                             Points$Filter$Builder
                                             (Points$Filter/newBuilder)
                                             {:addMust (ConditionFactory/matchKeyword (name k) (str v))}
                                             {})
                    :limit 1
                    :withPayload (WithPayloadSelectorFactory/enable true)}
                   {}))

(def scroll-points (qdrant/make-scroll-points  <collection-name> :user "user-1"))

(def scrolled (.get (.scrollAsync client scroll-points)))

(bean scrolled)

works, but when i do

(clojure.java.data/from-java scrolled)

I get a stackoverflow error

Answer 1

Logged as https://clojure.atlassian.net/browse/JDATA-26

Answer 2

Christos,

Can you provide more detail on exactly how I can reproduce this?

Preferably, a small completely self-contained repo on GitHub with a README explaining the steps.

Without more information, I cannot even begin to figure out what might be going on here.

Comments

Done.

https://github.com/gavlooth/problem/blob/master/src/problem/problem.clj

---

Thanks. I can repro with that. I'll dig into it over the next few days. Looks like something related to protobuf...

---

Confirmed related to protobuf...

from-java io.qdrant.client.grpc.Points$ScrollPoints
from-java com.google.protobuf.Descriptors$Descriptor
from-java com.google.protobuf.Descriptors$FileDescriptor
from-java com.google.protobuf.Descriptors$FileDescriptor
from-java com.google.protobuf.Descriptors$FileDescriptor
(repeats forever)

---

Looking at the API docs for protobuf, I don't think you're going to get a sensible Clojure data structure back from any deep conversion, because there's so much metadata involved.

You can see part of the problem if you run this code:

  (-> (jdata/from-java-shallow scroll-points {})
      (update-vals #(jdata/from-java-shallow % {})))

This sort of meta-representation is outside the reasonable scope of an automated general-purpose conversion to Clojure data.

You can get a deep conversion if you do this:

(jdata/from-java-deep scroll-points {:omit #{:allFields :defaultInstanceForType :parserForType :descriptorForType}})

but the result is an enormous nested map and I don't know how useful it will be?

---

Thanks, to be honest i have move past that anyway, but i find nuances like that from time to time.
It would be nice to know what are the failure points and how to go around them, stack overflow error didn't help on that. Next time i might try something with from-java-deep

---

Yeah, it's a tricky one. I had to instrument java.data to figure out the problem was protobuf, and then it was some REPL experimentation with from-java-shallow and from-java-deep to figure out what to omit to avoid the SO. I can add some notes to the README but I expect other Java libraries' objects could be equally problematic.