Clojure Q&A - Recent questions in Refs, agents, atoms

Why vswap is a macro instead of function like swap?

Sun, 04 May 2025 21:16:23 +0000

Why clojure.core/vswap is a macro with tag hint instead of a function called on clojure.lang.Volatile similar to clojure.core/swap?

Why does `await` fire off an `agent`'s watch?

Sat, 25 Jan 2025 04:40:53 +0000

I'm having trouble understand how await works on agents.
From the doc string reads

Blocks the current thread (indefinitely!) until all actions dispatched
thus far, from this thread or agent, to the agent(s) have occurred.

So sounds like it should wait for any tasks running on a given agent or any work that agent has dispatched itself (say to other agents or threads). So if I send something then I can await for it to finish.

However, it seems like a watch will also triggers for some reason...

(def bond (agent nil))

(add-watch bond
           :test
           (fn [_,_,_,_]
             (println "pew pew")))

(await bond)

;; the value of bond hasnt changed but it prints "pew pew" ..?

;; can be called an arbitrary number of times - keeps firing

(await bond)
(await bond)
(await bond)
(await bond)

add-watch docs read

Whenever the reference's state might have been changed, any registered watches will have their functions called.

However here the agent's state hasn't changed and yet it's fired

What's going on?

Is there a polished version of auto-agents?

Thu, 26 Dec 2024 18:41:21 +0000

I was reading Clojure in Action and thinking about agents. It seems that by hooking up agents with watches, update functions, and some simple locks, you could create a “reactive" state management. So that when some value a is updated then all dependent state values update themselves automatically and in parallel on separate threads. While those values are updating you are free to change other state values that aren't in the same dependency graph

I wrote up a quick draft but then started to look around online (it felt like a bit too obvious of an extension to the agent model). I found some similar ideas back in ~2009 but the trail goes dry. Is there some reason this isn't used more widely? Or is this some modern library I'm missing?

There was a "cell" implementation by Stuart Sierra called auto-agents - though I can't find the code

https://groups.google.com/g/clojure/c/NY834N34QvA

I also found this complete implementation but it's also got no traction at all

https://github.com/apatil/lazy-agent

Has this crystallized into a library somewhere? I'm thinking of writing my own implementation but I wanted a sanity check. I'm concerned maybe this idea was dropped for some reasons I've not considered

(I'm aware there are things like Javelin, Odoyle and Missionary that can accomplish similar things.. but while they decouple code, they're way too complicated and some don't have a threading story)

This is a copy of my question on Reddit:
https://reddit.com/comments/1hklfyh/comment/m3gyyoq

Memory leak in seque via agents

Thu, 10 Oct 2024 23:03:20 +0000

We used seque to manage a large database migration and observed a possible memory leak that required us to restart it several times.

We iterated through hundreds of queries to extract data via reduce, each using its own seque but eventually ran out of memory.

I believe it could have been related to seque and its use of agents, which was predicted to leak memory in CLJ-1125. We have not since tried a migration without seque for comparison, but I instead turned my attention to seque and found possibly-related problems.

seque uses an agent to offer items to its buffer. Agents have a memory leak where the conveyed bindings of a send are held by the executing Thread (the most relevant being *agent*). This means even if the seque is gc'ed, the agent persists if the thread is part of a cached thread pool, usually containing realized items from the producing seq (such as the first item to fail to be offered to the buffer).

I believe this demonstrates seque leaking memory (Clojure 1.12.0):

(let [pool-size 500]
  (defn expand-thread-pool! []
    (let [p (promise)]
      (mapv deref (mapv #(future (if (= (dec pool-size) %) (deliver p true) @p)) (range pool-size))))))

(let [_ (expand-thread-pool!) ;; increases likelihood of observing leak
      ready (promise)
      strong-ref (volatile! (Object.))
      weak-ref (java.lang.ref.WeakReference. @strong-ref)
      the-seque (volatile! (seque 1 (lazy-seq
                                      (let [s (repeat @strong-ref)]
                                        (deliver ready true)
                                        s))))]
  @ready
  (vreset! strong-ref nil)
  (vreset! the-seque nil)
  (System/gc)
  (doseq [i (range 10)
          :while (some? (.get weak-ref))]
    (prn "waiting for gc...")
    (Thread/sleep 1000)
    (System/gc))
  (prn (if (nil? (.get weak-ref))
         "garbage collection successful"
         "seque memory leak!!")))
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"seque memory leak!!"

This can be reproduced with agents. Once an agent has executed an action, a strong reference persists to the agent via *agent* in the cached thread it was executed in. Here we observe the agent is not garbage collected if it has executed an action on a cached thread pool (Clojure 1.12.0):

(let [_ (expand-thread-pool!)
      strong-ref (volatile! (agent nil))
      weak-ref (java.lang.ref.WeakReference. @strong-ref)]
  ;#_#_ ;;uncomment this and the agent is freed
  (send-off @strong-ref vector)
  (doseq [i (range 10)
          :while (not (vector? @@strong-ref))]
    (Thread/sleep 1000))
  (vreset! strong-ref nil)
  (System/gc)
  (doseq [i (range 10)
          :while (some? (.get weak-ref))]
    (prn "waiting for gc...")
    (Thread/sleep 1000)
    (System/gc))
  (prn (if (nil? (.get weak-ref))
         "garbage collection successful"
         "agent memory leak!!")))
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"agent memory leak!!"

Ref transactions causing deadlock at the REPL

Tue, 09 Jul 2024 14:28:33 +0000

Hi,

I've come across some strange behavior while playing around with refs at the REPL.
So I have a ref with a simple watcher function:

(defn watcher [_ r o n]
  (printf "ref: %s old: %s new: %s time: %d\n" @r o n (System/currentTimeMillis))
  (flush))

(def x (ref 2))
(add-watch x nil watcher)

I then evaluate the following expression two times in a row:

(future (dosync (ensure x) (Thread/sleep 3000) (alter x * 2)))

What I would expect to happen is that the watcher function should print the ref's value two times in a row, about 3 seconds apart.
But nothing happens.
When I then try to access the ref with another command such as

(dosync (ensure x) (alter x inc))

The REPL blocks completely. Once I did get the expected results several minutes later, another time I got this error message:

Execution error at reftest.core/eval24449 (form-init12758928819079876959.clj:34).                                               
Transaction failed after reaching retry limit

What I find perplexing is that everything seems to work just fine when I evaluate this let-block:

(let [x (ref 2)]
  (add-watch x nil watcher)
  (println "start:" (deref x))
  (future (dosync (ensure x) (Thread/sleep 3000) (alter x * 2)))
  (future (dosync (ensure x) (Thread/sleep 3000) (alter x * 2)))
  (dosync (ensure x) (alter x inc))
  (println "end:" @x))

Is the behavior at the REPL some kind of bug or am I doing something wrong?
How can I make sure that such STM transactions don't fail in real code?

What's an ideomatic way to manage state in a library?

Thu, 15 Feb 2024 12:46:36 +0000

I’m porting a JS library to clojure (and, potentially cljs), and I’m struggling with reasoning about local state. Imagine a scenario where you do RPC over websockets, and the server reply contextually depend on what the server had already sent you. You'd want to keep a local cache of the replies from the server, then.

In OOP world it's expected you incapsulate both the socket and the cache and provide the library users an interface that sends RPCs and transparently handles the cache population and pulling results from it. What would be the ideomatic way to do the same in clojure[-script]? Would I just return a map and expect my functions to take it as a first argument, returning the update state along with the rpc result?

How to force a timeout?

Tue, 12 Dec 2023 13:56:49 +0000

(defn tid []                                                        
  [(.getName (Thread/currentThread))                                                        
   (.getId (Thread/currentThread))])

   (defn atomize                                                   
  [f data]                                                                                  
  (let [atomix (atom [])                                                                    
        pool (Executors/newFixedThreadPool 12)                                              
        tasks (map                                                                          
               (fn [datum]                                                                  
                 (fn []                                                                     
                   (swap! atomix conj [(f datum) (tid)])))                                  
               data)]                                                                       
    (doseq [future (.invokeAll pool tasks)]                                                 
      (.get future  10 TimeUnit/MILLISECONDS ))                                             
    (.shutdown pool)                                                                        
    @atomix))




I can't figure out where to place a `sleep` to force a timeout.

Somehow I have to loop after all

Tue, 21 Nov 2023 19:50:16 +0000

I know some little Lisp, but Clojure feels a bit different:

(defn mojorate []
  (let [v (vec (range 1 11))
        max (- (count v) 1)
        atomix (atom [])
        chanl (async/chan)]
    (println v)
    (loop [i 0]
      (async/go
        (>! chanl (swap! atomix conj (/ (get v i) 2))))
      (if (= i max)
        (do (<!! (async/go (<! chanl)))
          (async/close! chanl)
          @atomix)
        (recur (inc i))))))

It's just an exercise with channel, atom loop/recur etc.

Can you do it like this?

How can I retrieve all older values of a PDS like atom/ref/agents ?

Sun, 27 Feb 2022 11:04:39 +0000

Based on my impression a PDS like clojure atom/refs/agents stores all new values in a tree like data-structure as they get changed over-time.

Now, my questions are

In clojure, current values of an identity are could be accesses by defer/@ syntax, is it possible to access all the older values of an atom/ref/agent and print them ? If so, how ?
If older values of an identity are NOT accessible, then what happens to them ? They are still in memory right ? If I cannot access them, what is the point of having a tree like structure ?

calling realized? on a delay which is currently executing blocks the caller

Tue, 05 Jan 2021 14:07:29 +0000

If a delay is currently processing, calling realized? causes the caller to block until the delay has completed and then returns true.

The behaviour I would have expected is that it should return false, which is what happens with futures. After all, if you wanted to wait until it was completed, you would deref it.

(also there don't seem to be any tags or categories related to concurrent programming, but since delay is part of clojure.core, I assume this is the correct place to ask)

How to known all the actions of an agent have been executed?

Tue, 05 Jan 2021 03:33:56 +0000

(def a (agent 0))

(dotimes [_ 100]
(send-off a inc))

How to determine the 100 actions have all been finished?

Where is volatile! officially documented?

Sat, 23 May 2020 22:02:28 +0000

I was looking for documentation of volatile! etc., but I came up short.

There is some information in the Jira ticket https://clojure.atlassian.net/browse/CLJ-1512 (opening statement clarified by Fogus), and some additional flavor was contributed in the form of an unusually vivid question about the needs of stateful transducers in https://clojure.atlassian.net/browse/CLJ-2146.

Java also has something called "volatile", which is described (for example) at https://docs.oracle.com/javase/tutorial/essential/concurrency/atomic.html. But Clojure's docstrings do not refer to Java's concept, which they easily could do if it were similar.

Does Clojure have official documentation about volatile!?

Tests for clojure.core/atom

Mon, 18 Apr 2016 09:19:10 +0000

As per discussion with Alex Miller Mars 3rd 2016 on clojure-dev, Alex suggested we should add tests to clojure.core/atom functionality, of which there is none today.

I proposed tests for
- the various ways to instatiate atoms (with and without validator and metadata)
- that validators throws correctly
- adding and removing watchers and that they trig as one would expected.
- various ways of changing values (no aim for finding high-load concurrency issues or patological cases or similar).
- that the arities of the interface in IAtom .swap works as expected - ie no reflection warnings (help/pointers for these type of cases needed!)
- generative, tests trying to find the glitches while using atoms (the things excluded above).

Alex suggested generative testing, but no performance tests.

The patch "0001-atom-unit-tests.clj" (attached) contains unit tests for

creating "bare" atom
creating atom with validator
that validate-fn triggers and that the atom is unchanged
that deref (@) reader macro creates correct '(clojure.core/deref a)
that CAS works for ordinary values (no validator-triggering etc).

There are plenty of combinations not covered with these tests, but this is a start.

To cover all cases (like cas-ing with invalid values and other strange things) generative testing is indeed a must.

atom watchers are not atomic with respect to reset!

Mon, 29 Jun 2015 14:26:56 +0000

It is possible that two threads calling `reset!` on an atom can interleave, causing the corresponding watches to be called with the same old value but different new values. This contradicts the guarantee that atoms update atomically.

(defn reset-test []
  (let [my-atom (atom :start
                      :validator (fn [x] (Thread/sleep 100) true))
        watch-results (atom [])]
    (add-watch my-atom :watcher (fn [k a o n] (swap! watch-results conj [o n])))

    (future (reset! my-atom :next))
    (future (reset! my-atom :next))
    (Thread/sleep 500)
    @watch-results))

(reset-test)

Yields [[:start :next] [:start :next]]. Similar behavior can be observed when mixing reset! and swap!.

h2. Expected behavior

Under atomic circumstances, (reset-test) should yield [[:start :next] [:next :next]]. This would "serialize" the resets and give more accurate information to the watches. This is the same behavior one would achieve by using (swap! my-atom (constantly :next)).

(defn swap-test []
  (let [my-atom (atom :start
                      :validator (fn [x] (Thread/sleep 100) true))
        watch-results (atom [])]
    (add-watch my-atom :watcher (fn [k a o n] (swap! watch-results conj [o n])))

    (future (swap! my-atom (constantly :next)))
    (future (swap! my-atom (constantly :next)))
    (Thread/sleep 500)
    @watch-results))

(swap-test)

Yields [[:start :next] [:next :next]]. The principle of least surprise suggests that these two functions should yield similar output.

h3. Alternative expected behavior

It could be that atoms and reset! do not guarantee serialized updates with respect to calls to watches. In this case, it would be prudent to note this in the docstring for atom.

h2. Analysis

The code for Atom.reset non-atomically reads and sets the internal AtomicReference. This allows for multiple threads to interleave the gets and sets, resulting in holding a stale value when notifying watches. Note that this should not affect the new value, just the old value.

h2. Approach

Inside Atom.reset(), validation should happen first, then a loop calling compareAndSet on the internal state (similar to how it is implemented in swap()) should run until compareAndSet returns true. Note that this is still faster than the swap! constantly pattern shown above, since it only validates once and the tighter loop should have fewer interleavings. But it has the same watch behavior.

public Object reset(Object newval){
    validate(newval);
    for(;;)
        {
            Object oldval = state.get();
            if(state.compareAndSet(oldval, newval))
                {
                    notifyWatches(oldval, newval);
                    return newval;
                }
        }
}