Clojure Q&A - Recent activity in tools.reader

Answer selected: tools.reader project file version is outdated

Mon, 24 Aug 2026 14:38:56 +0000

Contrib libraries are built with Maven, not Leiningen. Any project.clj files were added by the individual maintainers, purely for their own convenience, and should not be relied on.

Closed: clojure.tools.reader.edn/read can cause incorrect column number for token at end of input

Tue, 18 Mar 2025 14:31:27 +0000

Context
A user reported that rewrite-clj column number metadata can be incorrect for a token at the end of input.

Rewrite-clj uses clojure.tools/reader to read clojure source.
It uses clojure.tools.reader.reader-types/get-column-number to discover the current column offset.

Affected
clojure.tools/reader v1.3.4 through v1.5.0
Clojure only (not ClojureScript)

Env
On Linux using Clojure CLI 1.12.0.1517 with JDK 23.0.2 (temurin)

Symptom
When the element at end of input is a token, after it is read, the column number is one less than it should be.

Setup
Let's setup a playground in foo.clj

(ns foo
  (:require [clojure.pprint :refer [pprint]]
            [clojure.tools.reader.edn :as edn]
            [clojure.tools.reader.reader-types :as rt]))

(defn reader [s]
  (-> s
      rt/string-push-back-reader
      rt/indexing-push-back-reader))

(defn dump-with-cols [s]
  (println "input string:" s)
  (println "colnums:     " (apply str (take (inc (count s)) (cycle [1 2 3 4 5 6 7 8 9 0])))))

(defn read-by-obj [s]
  (with-open [r (reader s)]
    (loop [acc [{:at :bof :col-number (rt/get-column-number r)}]
           obj (edn/read r false ::eof {})]
      (if (not= ::eof obj)
        (recur (conj acc {:after obj :col-number (rt/get-column-number r)})
               (edn/read r false ::eof {}))
        (conj acc {:at :eof :col-number (rt/get-column-number r)})))))

(let [s (first *command-line-args*)]
  (dump-with-cols s)
  (println "parse result:")
  (pprint (read-by-obj s)))

Reproduction
Using clojure.tools/reader v1.5.0:

$ clojure -Sdeps '{:deps {org.clojure/tools.reader {:mvn/version "1.5.0"}}}' \
        -M foo.clj ':bip :bop'
input string: :bip :bop
colnums:      1234567890
parse result:
[{:at :bof, :col-number 1}
 {:after :bip, :col-number 5}
 {:after :bop, :col-number 9}
 {:at :eof, :col-number 9}]

Notice after :bip we see the correct :col-number 5, but after :bop we see the incorrect :col-number 9; it should be :col-number 10.

An incorrect column number occurs for any unwrapped last token, a number:

$ clojure -Sdeps '{:deps {org.clojure/tools.reader {:mvn/version "1.5.0"}}}' \
        -M foo.clj 123
input string: 123
colnums:      1234
parse result:
[{:at :bof, :col-number 1}
 {:after 123, :col-number 3}
 {:at :eof, :col-number 3}]

a symbol:

$ clojure -Sdeps '{:deps {org.clojure/tools.reader {:mvn/version "1.5.0"}}}' \
        -M foo.clj x
input string: x
colnums:      12
parse result:
[{:at :bof, :col-number 1}
 {:after x, :col-number 1}
 {:at :eof, :col-number 1}]

But we get a correct result for syntax wrapped elements:

$ clojure -Sdeps '{:deps {org.clojure/tools.reader {:mvn/version "1.5.0"}}}' \
        -M foo.clj '(x)'
input string: (x)
colnums:      1234
parse result:
[{:at :bof, :col-number 1}
 {:after (x), :col-number 4}
 {:at :eof, :col-number 4}]

Diagnosis
If I dial clojure.tools/reader back to v1.3.3, I don't see the issue:

$ clojure -Sdeps '{:deps {org.clojure/tools.reader {:mvn/version "1.3.3"}}}' \
        -M foo.clj ':bip :bop'
input string: :bip :bop
colnums:      1234567890
parse result:
[{:at :bof, :col-number 1}
 {:after :bip, :col-number 5}
 {:after :bop, :col-number 10}
 {:at :eof, :col-number 10}]

And then bump one version to v1.3.4, I see the issue:

$ clojure -Sdeps '{:deps {org.clojure/tools.reader {:mvn/version "1.3.4"}}}' \
        -M foo.clj ':bip :bop'
input string: :bip :bop
colnums:      1234567890
parse result:
[{:at :bof, :col-number 1}
 {:after :bip, :col-number 5}
 {:after :bop, :col-number 9}
 {:at :eof, :col-number 9}]

There was only one significant change between v1.3.3 and v1.3.4.

Unreading the character at eof is causing the off-by-one issue.

If I change the existing:

  (loop [sb (StringBuilder.)
         ch initch]
    (if (or (whitespace? ch)
            (macro-terminating? ch)
            (nil? ch))
      (do (unread rdr ch)
          (str sb))
      (if (not-constituent? ch)
        (err/throw-bad-char rdr kind ch)
        (recur (doto sb (.append ch)) (read-char rdr))))))))

To:

  (loop [sb (StringBuilder.)
         ch initch]
    (cond
      (or (whitespace? ch)
          (macro-terminating? ch))
      (do (unread rdr ch)
          (str sb))

      (nil? ch)
      (str sb)

      (not-constituent? ch)
      (err/throw-bad-char rdr kind ch)

      :else
      (recur (doto sb (.append ch)) (read-char rdr)))))))

The problem is resolved for me.

Next Steps
Happy to contribute a patch if the above makes sense to you and you think it is worthwhile to fix.

Commented: tools.reader parse-symbol is too sloppy when reading new n-dimensional class array

Thu, 22 Aug 2024 15:38:53 +0000

Thanks. The perf issue was just a bonus. The main problem was:

Tools.reader accepts this symbol: MyClazz:/3 while it does not accept the symbol MyClass:. This seems inconsistent. That's the main subject of this "ask".

Commented: Support Clojure 1.12-alpha12 Array class notation

Sat, 25 May 2024 12:24:22 +0000

i wrote a patch because i'm eager and had some free time, but that makes sense.

Answer selected: sending cljs file to repl causes reader exception

Thu, 09 Nov 2023 18:38:21 +0000

The issue should be fixed as of shadow-cljs 2.26.0, or rather tools.reader 1.3.7 which was the cause of the problem and is now the declared dependency.

Commented: Clojure Tools Reader intent regarding newline normalization?

Mon, 19 Sep 2022 22:53:00 +0000

Oh right, yes, good point! Thanks for your patience on this.

Rewrite-clj digs deeper into tools.reader than most users might.

It makes use of clojure.tools.reader.reader-types namespace using fns like read-char, unread and peekchar. It takes this lower level approach to return things the higher level reader fns do not return, like whitespace, comments and #_ :skipped-stuff.

So when rewrite-clj gets that whitespace, the newlines are always \n (even if they were originally \r\n).

Commented: No metadata on clojure 1.9 map literals

Tue, 13 Jul 2021 21:25:14 +0000

And Nicola fixed it right away and released a v1.3.6 with the fix. Thanks to both of you!

Answer edited: Looking for a safe way to read EDN with file position metadata

Wed, 23 Jun 2021 08:41:31 +0000

I ended up switching to https://github.com/borkdude/edamame

Answer selected: How to read namespaced keywords with clojure.tools.reader/read?

Mon, 18 Nov 2019 18:07:37 +0000

If you know how to resolve the autoresolved keyword aliases, you can set the *alias-map* dynamic binding and plug something in there: http://clojure.github.io/tools.reader/#clojure.tools.reader/*alias-map*

It's described as a map, but it's invoked as a function, so I think a function would work too.

Answered: Possibility to ignore read-eval?

Wed, 09 Oct 2019 13:40:33 +0000

Doesn't sound like a good idea to me. There are some other Clojure parsers out there that might better suit your needs like https://github.com/borkdude/edamame.

Answered: The error feedback from the read-string function would be improved if it used an indexing reader.

Wed, 26 Jun 2019 12:00:00 +0000

Reference: https://clojure.atlassian.net/browse/TRDR-45 (reported by alex+import)

Answered: Partial parses and complete source info (i.e. info on atoms without metadata)

Wed, 26 Jun 2019 12:00:00 +0000

Reference: https://clojure.atlassian.net/browse/TRDR-49 (reported by bhauman)

Answered: Simple benchmarking bash script to test before & after applying a batch

Wed, 26 Jun 2019 12:00:00 +0000

Reference: https://clojure.atlassian.net/browse/TRDR-29 (reported by dnolen)