Pub/sub leaks memory

Question

The following code will cause OOME:


(require '[clojure.core.async :refer [chan close! <! pub sub unsub go timeout]])

(def p* (chan))

(def p (pub p* :topic))

(go
  (loop []
    (let [s (chan)
          t (rand-int Integer/MAX_VALUE)]
      (sub p t s)
      (<! (timeout 10))
      (unsub p t s)
      (close! s)
      (recur))))


(It grows slowly: to see the OOME in a reasonable amount of time, either give JVM very small
memory like 64m, or remove the timeout step.)

I tried to profile the code, and the reason seems to be that even though I
unsubed the channel from the port, something is still retained which causes
the heap to be used up.

Answer 1

Comment made by: amadeoh

Here is the problem:

https://github.com/clojure/core.async/blob/96de9a47ac511d9bb4309645a3bc594a2fc0c33a/src/main/clojure/clojure/core/async.clj#L826-L828

When unsub* is called, it just untaps the channel from the mult specified by the topic. The mult still remains in the atom called mults even if the mult has no taps left.

I can't think of a clean fix for this problem, since currently the channels which are tapping a mult aren't exposed, i.e., we currently have no way of knowing if a mult has any taps on it.

Answer 2

Comment made by: gshayban

I also cannot think of a clean fix, as mults do not expose their registrants. (The notion of "current" is a concurrent system is subtle)
Besides saying (and perhaps amending the docstring) that pubs are indeed resources, their footprint grows by the 1. of seen topics, and that resources should almost always be bounded.

Answer 3

Comment made by: desk@danielcompton.net

This bit us too in https://github.com/apa512/clj-rethinkdb/issues/97. My suggestion for a docstring addition is:

pubs are resources, and their footprint grows by the number of seen topics. unpub does not reclaim this resource. You should not use pub to subscribe to an unbounded number of topics.

Are there any changes/suggestions? Would you like me to create a patch for this?

Answer 4

Comment made by: desk@danielcompton.net

Also, is a pub able to be garbage collected when it's no longer used? When I was doing some limited testing, it didn't seem like it was, but I may be wrong.

Answer 5

Comment made by: desk@danielcompton.net

A partial workaround (I think?) is to use
`(unsub-all pub topic)`
if you know you don't need a topic anymore.

Answer 6

Reference: https://clojure.atlassian.net/browse/ASYNC-90 (reported by alex+import)