Accidental breaking change in data.json 2.5.0: previously, read succeeded, now it isn't

Question

Here is a snippet of code that fails on data.json 2.5.0:

clj -Sdeps '{:deps {org.clojure/data.json {:mvn/version "2.5.0"}}}' -M -e '((requiring-resolve (quote clojure.data.json/read)) (clojure.lang.LineNumberingPushbackReader. (java.io.StringReader. "[\n  {\n    \"name\": \"position\",\n    \"type\": \"float32\",\n    \"count\": 3,\n    \"data\": [0,0,0]\n  },\n  {\n    \"name\": \"normal\",\n    \"type\": \"float32\",\n    \"count\": 3,\n    \"data\": [0,0,0]\n  },\n  {\n    \"name\": \"texcoord0\",\n    \"type\": \"float32\",\n    \"count\": 2,\n    \"data\": [0,0]\n  }\n]")))'

Execution error (IOException) at java.io.PushbackReader/unread (PushbackReader.java:166).
Pushback buffer overflow

The same code succeeds on 2.4.0:

clj -Sdeps '{:deps {org.clojure/data.json {:mvn/version "2.4.0"}}}' -M -e '((requiring-resolve (quote clojure.data.json/read)) (clojure.lang.LineNumberingPushbackReader. (java.io.StringReader. "[\n  {\n    \"name\": \"position\",\n    \"type\": \"float32\",\n    \"count\": 3,\n    \"data\": [0,0,0]\n  },\n  {\n    \"name\": \"normal\",\n    \"type\": \"float32\",\n    \"count\": 3,\n    \"data\": [0,0,0]\n  },\n  {\n    \"name\": \"texcoord0\",\n    \"type\": \"float32\",\n    \"count\": 2,\n    \"data\": [0,0]\n  }\n]")))'
[{"name" "position", "type" "float32", "count" 3, "data" [0 0 0]} {"name" "normal", "type" "float32", "count" 3, "data" [0 0 0]} {"name" "texcoord0", "type" "float32", "count" 2, "data" [0 0]}]

The gist of the problem: previously, Clojure's line numbering pushback reader could be used as an input to read. Now it can't: it throws an exception.

For reference, this is a json repro (a valid json!):

[
  {
    "name": "position",
    "type": "float32",
    "count": 3,
    "data": [0,0,0]
  },
  {
    "name": "normal",
    "type": "float32",
    "count": 3,
    "data": [0,0,0]
  },
  {
    "name": "texcoord0",
    "type": "float32",
    "count": 2,
    "data": [0,0]
  }
]

And, the formatted code: that uses the json:

(clojure.data.json/read 
  (clojure.lang.LineNumberingPushbackReader.
    (java.io.StringReader. "...")))

It seems the problem is introduced in https://clojure.atlassian.net/browse/DJSON-50

Answer 1

Interesting,

this smaller json reproduces the issue

clj -Sdeps '{:deps {org.clojure/data.json {:mvn/version "2.5.0"}}}' -M -e '((requiring-resolve (quote clojure.data.json/read)) (clojure.lang.LineNumberingPushbackReader. (java.io.StringReader. "{\"a\":\"b\"}")))'
Execution error (IOException) at java.io.PushbackReader/unread (PushbackReader.java:166).
Pushback buffer overflow

but an empty map or true does not:

clj -Sdeps '{:deps {org.clojure/data.json {:mvn/version "2.5.0"}}}' -M -e '((requiring-resolve (quote clojure.data.json/read)) (clojure.lang.LineNumberingPushbackReader. (java.io.StringReader. "{\n  }")))'
{}

Comments

I see the same error using a java.lang.PushbackReader directly without bothering with the clojure.lang.LineNumberingPushbackReader

---

looks like any json that is a string followed by something else (not necessarily valid json) triggers the pushback buffer overflow, for example

    (json/read (java.io.PushbackReader. (java.io.StringReader. "[\"\",true]")))

---

in the java.io.PushbackReader case, if you know how many characters will be pushed back, you can increase the buffer size and make it work, but not for the clojure.lang.LineNumberingPushbackReader case:

user=> (require '[clojure.data.json :as json])
nil
user=> (json/read (java.io.PushbackReader. (java.io.StringReader. "[\"\",true]")))
Execution error (IOException) at java.io.PushbackReader/unread (PushbackReader.java:166).
Pushback buffer overflow
user=> (json/read (java.io.PushbackReader. (java.io.StringReader. "[\"\",true]") 6))
["" true]
user=> (json/read (clojure.lang.LineNumberingPushbackReader. (java.io.StringReader. "[\"\",true]") 6))
Execution error (IOException) at java.io.PushbackReader/unread (PushbackReader.java:166).
Pushback buffer overflow
user=> (json/read (clojure.lang.LineNumberingPushbackReader. (java.io.StringReader. "[\"\",true]") 100))
Execution error (IOException) at java.io.PushbackReader/unread (PushbackReader.java:166).
Pushback buffer overflow
user=> (json/read (clojure.lang.LineNumberingPushbackReader. (java.io.StringReader. "[\"\",true]") 100000))
Execution error (IOException) at java.io.PushbackReader/unread (PushbackReader.java:166).
Pushback buffer overflow
user=>


When data.json creates the pushbackreader itself it does it with a buffer of size 64. Unclear why LineNumberingPushbackReader seems to ignore the buffer size.

Answer 2

A PBR with buffer size >= 64 must be supplied if you intend to do repeated reads on the same string. The docstring to read has been further updated in 2.5.1 to include this.