Clojure Q&A - Recent questions and answers in Refs, agents, atoms

Answered: Why vswap is a macro instead of function like swap?

Mon, 05 May 2025 07:41:52 +0000

I don't know for sure, but this comment on the initial ticket seems plausible.

https://clojure.atlassian.net/browse/CLJ-1512?focusedCommentId=19049

the vswap! macro is probably for performance reasons (the main motivation of this code to begin with), to avoid using apply or unrolling tons of arities

Answered: Why does `await` fire off an `agent`'s watch?

Sat, 25 Jan 2025 08:00:40 +0000

If you look at the source of await, you'll see it send a function to each agent provided as arguments, that counts down a latch (and returns the agent's value).

A watch is triggered every time a fn is sent to that agent -- because any fn "might have [..] changed" the state.

Is there a polished version of auto-agents?

Thu, 26 Dec 2024 18:41:21 +0000

I was reading Clojure in Action and thinking about agents. It seems that by hooking up agents with watches, update functions, and some simple locks, you could create a “reactive" state management. So that when some value a is updated then all dependent state values update themselves automatically and in parallel on separate threads. While those values are updating you are free to change other state values that aren't in the same dependency graph

I wrote up a quick draft but then started to look around online (it felt like a bit too obvious of an extension to the agent model). I found some similar ideas back in ~2009 but the trail goes dry. Is there some reason this isn't used more widely? Or is this some modern library I'm missing?

There was a "cell" implementation by Stuart Sierra called auto-agents - though I can't find the code

https://groups.google.com/g/clojure/c/NY834N34QvA

I also found this complete implementation but it's also got no traction at all

https://github.com/apatil/lazy-agent

Has this crystallized into a library somewhere? I'm thinking of writing my own implementation but I wanted a sanity check. I'm concerned maybe this idea was dropped for some reasons I've not considered

(I'm aware there are things like Javelin, Odoyle and Missionary that can accomplish similar things.. but while they decouple code, they're way too complicated and some don't have a threading story)

This is a copy of my question on Reddit:
https://reddit.com/comments/1hklfyh/comment/m3gyyoq

Memory leak in seque via agents

Thu, 10 Oct 2024 23:03:20 +0000

We used seque to manage a large database migration and observed a possible memory leak that required us to restart it several times.

We iterated through hundreds of queries to extract data via reduce, each using its own seque but eventually ran out of memory.

I believe it could have been related to seque and its use of agents, which was predicted to leak memory in CLJ-1125. We have not since tried a migration without seque for comparison, but I instead turned my attention to seque and found possibly-related problems.

seque uses an agent to offer items to its buffer. Agents have a memory leak where the conveyed bindings of a send are held by the executing Thread (the most relevant being *agent*). This means even if the seque is gc'ed, the agent persists if the thread is part of a cached thread pool, usually containing realized items from the producing seq (such as the first item to fail to be offered to the buffer).

I believe this demonstrates seque leaking memory (Clojure 1.12.0):

(let [pool-size 500]
  (defn expand-thread-pool! []
    (let [p (promise)]
      (mapv deref (mapv #(future (if (= (dec pool-size) %) (deliver p true) @p)) (range pool-size))))))

(let [_ (expand-thread-pool!) ;; increases likelihood of observing leak
      ready (promise)
      strong-ref (volatile! (Object.))
      weak-ref (java.lang.ref.WeakReference. @strong-ref)
      the-seque (volatile! (seque 1 (lazy-seq
                                      (let [s (repeat @strong-ref)]
                                        (deliver ready true)
                                        s))))]
  @ready
  (vreset! strong-ref nil)
  (vreset! the-seque nil)
  (System/gc)
  (doseq [i (range 10)
          :while (some? (.get weak-ref))]
    (prn "waiting for gc...")
    (Thread/sleep 1000)
    (System/gc))
  (prn (if (nil? (.get weak-ref))
         "garbage collection successful"
         "seque memory leak!!")))
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"seque memory leak!!"

This can be reproduced with agents. Once an agent has executed an action, a strong reference persists to the agent via *agent* in the cached thread it was executed in. Here we observe the agent is not garbage collected if it has executed an action on a cached thread pool (Clojure 1.12.0):

(let [_ (expand-thread-pool!)
      strong-ref (volatile! (agent nil))
      weak-ref (java.lang.ref.WeakReference. @strong-ref)]
  ;#_#_ ;;uncomment this and the agent is freed
  (send-off @strong-ref vector)
  (doseq [i (range 10)
          :while (not (vector? @@strong-ref))]
    (Thread/sleep 1000))
  (vreset! strong-ref nil)
  (System/gc)
  (doseq [i (range 10)
          :while (some? (.get weak-ref))]
    (prn "waiting for gc...")
    (Thread/sleep 1000)
    (System/gc))
  (prn (if (nil? (.get weak-ref))
         "garbage collection successful"
         "agent memory leak!!")))
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"waiting for gc..."
;"agent memory leak!!"

Answered: Ref transactions causing deadlock at the REPL

Tue, 09 Jul 2024 15:42:03 +0000

I assume both transactions are failing and retrying multiple times, and each retry takes a very long time until one of them happens to get the timing right to work or maxes out the retries.

You should never do blocking operations in your transactions.

Answered: What's an ideomatic way to manage state in a library?

Thu, 15 Feb 2024 13:56:21 +0000

Yes, make state explicit, passed as arguments. Don't use global state. Don't use dynamic vars either. Don't assume a specific state framework such as Integrant or Component.

This way consumers will have the most flexibility and simplicity.

Answered: How to force a timeout?

Sat, 16 Dec 2023 01:06:12 +0000

Hard to tell exactly what your question is, but my guess is you want the execution of the future to stop if it doesn't complete in a certain amount of time.

.get of course does not do this, the timeout for .get is a timeout on the operation of waiting for the result of the future, not a timeout on the future itself.

you could pair .get with a timeout and future-cancel to maybe get something like that, but even future-cancel is not guaranteed to stop a running thread. And the jvm authors keep removing ways to uncooperatively stop a running a thread.

The best way is to write the process that is running in such a way as to cooperate with being shutdown, either waiting for some signal to shutdown or stopping execution after some time limit has been reached. This requires a fair bit of communication between processes and sometimes requires processes to be able to wait on multiple threads at once (wait for a message from another process and also wait for the signal to stop). core.async provides some nice primitives for that.

Answered: Somehow I have to loop after all

Wed, 22 Nov 2023 07:00:36 +0000

You have way too many go blocks, not only in terms of their amount in the code but also in terms of how many are executed. That first go block in the loop doesn't block - it lets the execution continue immediately. There's no guarantee that by the time that same block is executed again the previous block has had enough time to put the value into the channel. It's a race condition.

To confirm, simply run this code: (set (repeatedly 10000 mojorate)). Increase the number if all you get is a set of one element.

The snippet sends 10 numbers to a channel and gets them back - divided by two.

The first half of the sentence is an implementation detail that's beside the point from the perspective of the user of the function. If the only thing you need to do is to get a vector of 10 numbers starting from 1 divided by 2, then it's (mapv #(/ % 2) (range 1 11)) of course.

If you need something else, then please describe in detail so we're not stuck in the XY problem. There are multiple ways in which the original function can be rewritten - none of which could potentially be correct if your real problem demands something specific.

Answered: How can I retrieve all older values of a PDS like atom/ref/agents ?

Mon, 28 Feb 2022 13:23:12 +0000

I will add to the earlier answer.

Clojure's reference types (atoms, ref, agents) are orthogonal to persistent data structures. They are used in conjunction with persistent data structures (e.g. the expectation is that immutable values will be contained in said reference types to enforce software transactional memory semantics and the like), but they aren't directly related. Reference types model an identity, e.g. some information that may vary over time where the successive values are related by some named thing.

Persistent data structures let you maintain references to an original value, and create new values that represent semantic "modifications" to the original (e.g. associate a new key in map, change the value at index 3 in a vector, conj an item to a set, etc.). Since this is accomplished via typically hash array mapped trie variants (and in the case of sorted collections, red/black balanced binary trees, or cons cells in the case of seqs), the efficiency gains come from sharing as much structure as possible between the derived "new" value and the old. So some form of path copying scheme exists where only a minimal subset of the old tree is actually necessary to copy and then safely modify the copied subtree when creating the new structure; everything else is referenced.

Aside from performance (minimal path copying in tries makes for surprisingly efficient vectors and maps), this enables writing in the pure functional style that clojure defaults to. We can write complicated, useful programs without basing them on mutation and side effects. Since our fundamental collections are persistent, then reasoning about the basic operations is trivial (pervasively so!): I always know that assoc, dissoc, conj ,disj, etc. will not modify the input and will return either an equivalent value or a different value (as opposed to a mutable container where value/equality semantics are ambiguous). Information flow through a program written in this style is distilled to functions, input, and output where information can only really flow one direction (since inputs are never modified).

The other upside of this is that you have the ability to retain cheap (memory efficient) copies of older versions. This is not intrinsic and has nothing to do with reference types directly, but we can use a reference type (like an atom) to implement some naive form of versioning:

(def versions (atom [[]]))
(defn current
  ([v] (peek @v))
  ([] (current versions)))
(defn update! [f & args]
  (let [newv (apply f (current) args)]
    (swap! versions conj newv)
    newv))

(defn roll-back!
  ([v] (-> v (swap! #(if (seq (peek %)) (pop %) %)) peek))
  ([] (roll-back! versions)))

;; user=> (update! conj 2)
;; [2]
;; user=> (update! conj 3)
;; [2 3]
;; user=> (roll-back!)
;; [2]
;; user=> (roll-back!)
;; []

(let [original (current)
      v1 (update! conj 2)
      v2 (update! conj 3)
      v3 (update! conj 4)
      v4 (do (roll-back!) (roll-back!))]
  {:v1 v1
   :v2 v2
   :v3 v3
   :v4 v4})

;;{:v1 [2], :v2 [2 3], :v3 [2 3 4], :v4 [2]}

In the let demo, the references associated in the map for v1 v2 v3 and v4 are all different values, although there is structural sharing between them. They are all thread safe and can persist without affecting anything else (since they can't change). Assuming space isn't a concern, using this kind of scheme allows trivial "undo" at potentially complex application data levels. It is trivial to maintain everything in an app state, and retain a log of previous versions. Rewinding the application state is as simple as popping items off of the history. We don't need to delta diff or anything, or generate patches, since the shared structure is already taken care of for us.

If you stick with the persistent data structures to model your application data, then you effectively have the ability to snapshot the entire "world" at whatever level of granularity you would like (if it makes senses).

Answered: calling realized? on a delay which is currently executing blocks the caller

Tue, 05 Jan 2021 22:46:24 +0000

Logged as https://clojure.atlassian.net/browse/CLJ-2600, thanks.

Answered: How to known all the actions of an agent have been executed?

Tue, 05 Jan 2021 03:51:01 +0000

await does exactly this:
(clojure.repl/doc await)
clojure.core/await ([& agents]) Blocks the current thread (indefinitely!) until all actions dispatched thus far, from this thread or agent, to the agent(s) have occurred. Will block on failed agents. Will never return if a failed agent is restarted with :clear-actions true or shutdown-agents was called.

To try it out:
(def a (agent 0)) (dotimes [_ 100] (send-off a (fn [x] (Thread/sleep 30) (inc x)))) (await a)
(Will block for a few seconds)

Answered: Where is volatile! officially documented?

Sat, 23 May 2020 23:36:32 +0000

Clojure’s use does build on Java’s volatiles so the name is intentional.

The only other place I can think of where it’s documented is in the release notes https://github.com/clojure/clojure/blob/master/changes.md#21-transducers

It’s considered kind of an advanced tool so it’s intentionally not front and center. Could probably be mentioned in the transducer docs as an impl tool.

Answered: atom watchers are not atomic with respect to reset!

Wed, 26 Jun 2019 12:00:00 +0000

Reference: https://clojure.atlassian.net/browse/CLJ-1770 (reported by ericnormand)

Answered: Tests for clojure.core/atom

Wed, 26 Jun 2019 12:00:00 +0000

Reference: https://clojure.atlassian.net/browse/CLJ-1915 (reported by claj)