Clojure Q&A - Recent questions tagged core.async

How to use core.async from cljr command line tool?

Mon, 24 Nov 2025 06:05:40 +0000

At some point it would be good to facilitate use of core.async from cljr command line tool (clr.core.cli).

I think what's required is:

add a deps-clr.edn to clojure/tools.analyzer
add a deps-clr.edn to clojure/tools.analyzer.clr (depends on tools.analyzer, clr.tools.reader, clr.core.memoize -> clr.core.cache -> clr.data.priority-map)
add a deps-clr.edn to clojure/clr.core.async
also, archive clojure/clr.tools.analyzer

This is not urgent because it's possible to procure core.async from NuGet but it would be convenient if it could also be procured via git.

`pipeline-blocking` uses `thread` internally instead of `io-thread`

Wed, 29 Oct 2025 15:54:00 +0000

pipeline-blocking is explicitly meant for blocking operations (implying I/O). However, internally it uses thread rather than io-thread for processing: https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async.clj#L637-L640.

Since 1.9.829-alpha2 this has practical implications, as only io-thread will utilize virtual threads, when available.

Can this be fixed / changed?
If not, can an explicit pipeline-io (or similar) function be added to support this use case?

Thanks

Choosing between core.async pipelines

Sun, 26 Oct 2025 22:19:12 +0000

It's no longer clear what kind of pipeline is suitable for what kind of task in core.async.

Previously, my understanding was that anything I/O-bound should go into pipeline-blocking, while CPU-bound tasks went into regular (i.e. :compute) pipeline. But a while back :compute pipelines were changed to use threads instead of go-blocks. And now, with the introduction of virtual threads, pipeline-async seems like the right choice for I/O-bound tasks.

It seems that workloads are being classified into one of three types: :compute, :io, and :mixed. Is :async meant to be the equivalent of :io, and :blocking the equivalent of :mixed? I don't think that's original semantics, but it seems to be the mapping now.

Replace core.async’s internal LinkedList queue with ArrayDeque?

Tue, 02 Sep 2025 08:02:18 +0000

Hi! While reading the channel implementation I noticed the internal queue is a java.util.LinkedList:
https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async/impl/channels.clj

The common JVM guidance these days is to prefer java.util.ArrayDeque for FIFO/LIFO queues due to better locality and lower GC overhead. For example, the JDK docs state: “This class is likely to be faster than Stack when used as a stack, and faster than LinkedList when used as a queue.”
https://docs.oracle.com/javase/8/docs/api/java/util/ArrayDeque.html
Related SO discussion: https://stackoverflow.com/questions/6163166/why-is-arraydeque-better-than-linkedlist

A few questions for the maintainers:

Was LinkedList originally chosen for a specific reason (e.g., very old JDK compatibility)?
Are there behaviors in channels that specifically rely on LinkedList (e.g., allowing null, I believe channels disallow nil anyway, or particular iterator characteristics), or would ArrayDeque be a drop-in replacement for the add/remove-from-ends operations used?
If we provide a small PR and benchmark showing an improvement (lower allocations / better throughput under contention), would such a change be considered?
Since ArrayDeque is available on Java 8+, and current Clojure/core.async baselines target that or newer, is there any remaining compatibility concern?

Context: there was an older conversation in Clojurians (#clojure) that suggested ArrayDeque could reduce GC pressure under load and that the original choice may have been influenced by older JDKs:
https://clojurians.slack.com/archives/C03S1KBA2/p1526551888000376

If this sounds reasonable, I’m happy to run the core.async test suite, add JMH-style benchmarks focused on channel put/take hot paths, and submit a PR.

Thanks!

Typos in Flow guide

Mon, 18 Aug 2025 08:19:35 +0000

As reported by a Hacker News user, there are typos in the Flow Guide :

"excepiton" should be "exception"
"provded" should be "provided"

and also, it seems that in the transition section, "The description arity takes the current state ..." should be "The transition arity takes the current state ..."

Is core.async.flow available in ClojureScript?

Thu, 17 Jul 2025 08:31:10 +0000

Hi,

I know it's early days for core.async.flow but I thought I would give it a try in one of my ClojureScript projects.

I've got {:mvn/version "1.9.808-alpha1"} in my deps.edn but I'm getting the following error when I try to access the namespace:

No such namespace: clojure.core.async.flow, could not locate clojure/core/async/flow.cljs

Thanks for your help.

How best to call long-running tasks from a core.async.flow process?

Wed, 09 Jul 2025 18:31:41 +0000

I'd like to write to a database as the final process in a core.async.flow flow. The db load can take several minutes. If I do this work directly in the :transform function's thread, flow-monitor can't ping that process for the duration of the load. But if I offload the work to a different thread and return from the :transform function, I can't as easily control back pressure from that process.

The relevant process currently looks something like:

(def db-loader
  (flow/map->step
   {:describe (fn [] {:workload :io
                      :ins {:in "Batched items"}})
    :init     (fn [state] state)
    :transition (fn [state status] state)
    :transform (fn [state _ batch]
                 (t/log! "DB Loader is loading a batch.")
                 (Thread/sleep 10000) ;; imagine this code writes to the db.
                 (t/log! "DB Loader loaded a batch.")
                 [state])}))

A more complete code example is at https://gist.github.com/tomconnors/245fb69ed757b34502c8d57637db8de2.

Is there a better option for this process?

what kind of systems could be made with clojure.async.flow?

Tue, 29 Apr 2025 15:48:35 +0000

Trying to capture mechanical essence of clojure.async.flow with basic terms instead of flow own terms i guess that clojure.async.flow allows to structure a system as a set of stateful functions running concurrently in the same jvm process and interacting with each other explicitly via channels.

what kind of systems could be made with clojure.async.flow?

it's definitely not naive service which handles API http requests, transforms it somehow and puts it into database.

`doseq` and `for` expands body twice

Wed, 26 Feb 2025 23:15:06 +0000

The bodies of doseq and for seem to be duplicated in their expansions.

$ clojure
user=> (defmacro a [] (prn :expand))
#'user/a
user=> (doseq [_ nil] (a))
:expand
:expand
nil
user=> (for [_ nil] (a))
:expand
:expand
()

I found the same problem in ClojureScript. I could not find an existing discussion about this so I'm not sure if this is by design, apologies if so. I'm aware that these macros have a large code footprint and that doseq cannot use closures (the usual way to prevent exponential expansion), but I missed this detail.

This leads to exponential code blowup if these forms are nested. Artificial demonstration:

$ clojure
Clojure 1.12.0
user=> (def counter (atom -1))
#'user/counter
user=> (defmacro a [] (prn :expand (swap! counter inc)))
#'user/a
user=> #(doseq [_ 1] (doseq [_ 2] (doseq [_ 3] (doseq [_ 4] (a)))))
:expand 0
:expand 1
:expand 2
:expand 3
:expand 4
:expand 5
:expand 6
:expand 7
:expand 8
:expand 9
:expand 10
:expand 11
:expand 12
:expand 13
:expand 14
:expand 15
#object[user$eval926$fn__927 0x76563d26 "user$eval926$fn__927@76563d26"]

This might be especially relevant to core.async, where placing go under doseq is idiomatic, since go is expensive in both expansion time and code size.

user=> (time (eval '(doseq [_ nil] (a/go (doseq [_ nil] (a/go (doseq [_ nil] (a/go))))))))
"Elapsed time: 837.230917 msecs"
nil

It also leads to duplicated reflection warnings from the fully expanded forms, which is how I found this problem.

user=> (doseq [_ nil] (Thread/sleep (identity 1)))
Reflection warning, NO_SOURCE_PATH:1:16 - call to static method sleep on java.lang.Thread can't be resolved (argument types: unknown).
Reflection warning, NO_SOURCE_PATH:1:16 - call to static method sleep on java.lang.Thread can't be resolved (argument types: unknown).
nil

I believe the exponential expansion for doseq was introduced in Clojure 1.1.0 with support for chunked seqs with this commit.

$ clojure -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.0.0"}}}' 
Downloading: org/clojure/clojure/1.0.0/clojure-1.0.0.pom from central
Downloading: org/clojure/clojure/1.0.0/clojure-1.0.0.jar from central
Clojure 1.0.0-
user=> (defmacro a [] (prn :expand))
#'user/a
user=> (doseq [_ nil] (a))
:expand
nil
user=> ^D
$ clojure -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.1.0"}}}'
Downloading: org/clojure/clojure/1.1.0/clojure-1.1.0.pom from central
Downloading: org/clojure/clojure/1.1.0/clojure-1.1.0.jar from central
Clojure 1.1.0
user=> (defmacro a [] (prn :expand))
#'user/a
user=> (doseq [_ nil] (a))
:expand
:expand
nil
user=>

At first I wasn't sure if this was hopeless, but I think doseq is fixable fusing the chunked and non-chunked cases like so:

(defmacro doseq
  "Repeatedly executes body (presumably for side-effects) with
  bindings and filtering as provided by \"for\".  Does not retain
  the head of the sequence. Returns nil.
  
  Unlike clojure.core/doseq, does not cause exponential macro expansion
  of expressions in bindings or body."
  [seq-exprs & body]
  (#'clojure.core/assert-args
     (vector? seq-exprs) "a vector for its binding"
     (even? (count seq-exprs)) "an even number of forms in binding vector")
  (let [step (fn step [recform exprs]
               (if-not exprs
                 [true `(do ~@body)]
                 (let [k (first exprs)
                       v (second exprs)]
                   (if (keyword? k)
                     (let [steppair (step recform (nnext exprs))
                           needrec (steppair 0)
                           subform (steppair 1)]
                       (cond
                         (= k :let) [needrec `(let ~v ~subform)]
                         (= k :while) [false `(when ~v
                                                ~subform
                                                ~@(when needrec [recform]))]
                         (= k :when) [false `(if ~v
                                               (do
                                                 ~subform
                                                 ~@(when needrec [recform]))
                                               ~recform)]))
                     (let [seq- (gensym "seq_")
                           chunk- (with-meta (gensym "chunk_")
                                             {:tag 'clojure.lang.IChunk})
                           count- (gensym "count_")
                           i- (gensym "i_")
                           in-chunk- (gensym "in-chunk_")
                           recform `(if ~in-chunk-
                                      (recur ~seq- ~chunk- ~count- (unchecked-inc ~i-))
                                      (recur (next ~seq-) nil 0 0))
                           steppair (step recform (nnext exprs))
                           needrec (steppair 0)
                           subform (steppair 1)]
                       [true
                        `(loop [~seq- (seq ~v), ~chunk- nil,
                                ~count- 0, ~i- 0]
                           (let [~in-chunk- (< ~i- ~count-)
                                 ~seq- (if ~in-chunk- ~seq- (seq ~seq-))]
                             (when (if ~in-chunk- true ~seq-)
                               (let [chunked?# (if ~in-chunk- false (chunked-seq? ~seq-))
                                     ~k (if ~in-chunk-
                                          (.nth ~chunk- ~i-)
                                          (if chunked?# nil (first ~seq-)))]
                                 (if (if ~in-chunk- false chunked?#)
                                   (let [c# (chunk-first ~seq-)]
                                     (recur (chunk-rest ~seq-) c#
                                            (int (count c#)) (int 0)))
                                   (do ~subform
                                       ~@(when needrec [recform])))))))])))))]
    (nth (step nil (seq seq-exprs)) 1)))

core.async 1.8 beta1 cached thread pools can grow to hundreds of threads when doing io work

Wed, 26 Feb 2025 18:54:58 +0000

Hi, I've been testing the new clojure.core.async 1.8.711-beta1, and suspected that since the default fixed thread pool was replaced with a cached thread pool, it could be possible for the system to create a very large number of OS threads, in the order of hundreds or thousands depending on the size of the workload.

I am wondering if this was intended or if there might be a better middle-ground such as still using a CachedThreadPool but potentially accepting maximum sizes for each pool type via sys props, with reasonable defaults.

Example code below:

(ns user)

(require '[clojure.core.async :as a]
         '[clojure.core.async.impl.dispatch :as d])

(defn do-work
  "Simulate some async work which "
  [task-count task-timeout]
  (a/go-loop [c (a/merge
                 (doall
                  (for [i (range task-count)]
                    (a/go
                      ;;; this simulates non-blocking park insdie the go-block
                      (a/<! (a/timeout
                             (rand-int task-timeout)))
                      ;;; this simulates blocking IO inside an io-thread
                      (a/<! (a/io-thread
                             (Thread/sleep
                              ^int (rand-int task-timeout))))
                      i))))]
    (let [v (a/<! c)]
      (if (nil? v)
        {:pool-size (.getPoolSize ^java.util.concurrent.ThreadPoolExecutor (d/executor-for :core-async-dispatch))}
        (recur c)))))

(def active
  (atom false))

(defn run-sim
  []
  (reset! active true)
  (a/go-loop [iter 1]
    (let [start (System/currentTimeMillis)
          {:keys [pool-size]} (a/<! (do-work
                                     2000  ;;; simulated number of tasks
                                     100)) ;;; simulated max delay in ms
          stop (System/currentTimeMillis)
          elapsed (- stop start)]
      (println "Iteration:" iter "Pool Size:" pool-size "Elapsed:" elapsed)
      (when (and @active
                 (< iter 10))
        (recur (inc iter))))))

(defn stop-sim
  []
  (reset! active false))

(comment
  (run-sim)   ;;; run the sim
  (stop-sim)) ;;; stop the sim

core.async go block error with clojure 1.12 method values

Tue, 25 Feb 2025 19:17:23 +0000

I haven't seen this reported.

(Long/.toString 123) ;;works
(a/<!! (a/thread (Long/.toString 123))) ;;works
(a/go (Long/.toString 123)) ;;error

Note: The following stack trace applies to the reader or compiler, your code was not executed.

CompilerException Syntax error macroexpanding a/go at (1:1). {:clojure.error/phase :macro-syntax-check, :clojure.error/line 1, :clojure.error/column 1, :clojure.error/source "NO_SOURCE_PATH", :clojure.error/symbol a/go}
ExceptionInfo No matching method: .toString for class: class java.lang.Long and arity: 1 {:method .toString, :class java.lang.Long, :argc 1, :file "NO_SOURCE_PATH", :column 7, :line 1}
	clojure.tools.analyzer.passes.jvm.validate/validate-call (validate.clj:127)
	clojure.tools.analyzer.passes.jvm.validate/validate-call (validate.clj:87)
	clojure.tools.analyzer.passes.jvm.validate/eval4802/fn--4803 (validate.clj:137)
	clojure.lang.MultiFn.invoke (MultiFn.java:229)
	clojure.tools.analyzer.passes.jvm.validate/validate (validate.clj:265)
	clojure.tools.analyzer.passes.jvm.validate/validate (validate.clj:240)
	clojure.lang.Var.invoke (Var.java:386)
	clojure.tools.analyzer.passes/compile-passes/fn--3424/fn--3429 (passes.clj:167)
	clojure.tools.analyzer.passes/compile-passes/fn--3424/fn--3431 (passes.clj:169)
	clojure.tools.analyzer.passes/compile-passes/fn--3424/fn--3431 (passes.clj:169)
	clojure.tools.analyzer.passes/compile-passes/fn--3424/fn--3431 (passes.clj:169)
	clojure.core/partial/fn--5929 (core.clj:2648)

Somehow I have to loop after all

Tue, 21 Nov 2023 19:50:16 +0000

I know some little Lisp, but Clojure feels a bit different:

(defn mojorate []
  (let [v (vec (range 1 11))
        max (- (count v) 1)
        atomix (atom [])
        chanl (async/chan)]
    (println v)
    (loop [i 0]
      (async/go
        (>! chanl (swap! atomix conj (/ (get v i) 2))))
      (if (= i max)
        (do (<!! (async/go (<! chanl)))
          (async/close! chanl)
          @atomix)
        (recur (inc i))))))

It's just an exercise with channel, atom loop/recur etc.

Can you do it like this?

Why do core.async timeouts rely on non-monotonic system clock?

Wed, 01 Nov 2023 16:26:33 +0000

clojure.core.async/timeout uses System/currentTimeMillis to calculate its deadline (see here and here). While the docstring for System/currentTimeMillis doesn't mention it explicitly, it's quite clearly unsuitable for this purpose because it returns the current time as reported by the OS. Since that time can change arbitrarily (e.g. by correction via NTP), the deadline calculations may end up inconsistent. This behavior is very much intended, see e.g. this ticket where it was suggested to switch System/currentTimeMillis over to a monotonic clock where possible. It was refused with the following comment:

System.currentTimeMillis() is required to return milliseconds since the epoch and should reflect external changes to the system/wall-clock time. CLOCK_MONOTONIC_COARSE does not provide that and so can not be used for this purpose.

Also, the docstring for java.util.concurrent.ScheduledExecutorService states this quite plainly:

All schedule methods accept relative delays and periods as arguments, not absolute times or dates. It is a simple matter to transform an absolute time represented as a Date to the required form. For example, to schedule at a certain future date, you can use: schedule(task, date.getTime() - System.currentTimeMillis(), TimeUnit.MILLISECONDS). Beware however that expiration of a relative delay need not coincide with the current Date at which the task is enabled due to network time synchronization protocols, clock drift, or other factors.

Since all we care about here is deltas, it seems advisable to switch to System/nanoTime instead. Unfortunately, its docstring also doesn't make its properties very obvious. It doesn't mention that it's a monotonic clock but only hints at it by stating:

Returns the current value of the running Java Virtual Machine's high-resolution time source, in nanoseconds. This method can only be used to measure elapsed time and is not related to any other notion of system or wall-clock time. [...] The values returned by this method become meaningful only when the difference between two such values, obtained within the same instance of a Java virtual machine, is computed.

A case in point for using it is OpenJDK's implementation of java.util.concurrent.ScheduledThreadPoolExecutor whose deadline calculation method does so, as well as its getDelay implementation.

Furthermore--if it's anything to go by--various StackOverflow posts also discuss this and arrive at the same conclusion, e.g. this one:

Nonetheless, nanoTime() should still be preferred for implementing timed blocking, interval waiting, timeouts, etc. to currentTimeMillis() because the latter is a subject to the "time going backward" phenomenon (e. g. due to server time correction), i. e. currentTimeMillis() is not suitable for measuring time intervals at all. See this answer for more information.

Virtual thread executor for core.async

Sun, 08 Oct 2023 23:30:28 +0000

There have been a few people on Slack asking about virtual threads and core.async before. Here is an example of one such thread https://clojurians.slack.com/archives/C05423W6H/p1677676131782299

Of course, consideration is also needed for ClojureScript and non-Loom code. I would assume a virtual thread aware implementation of core.async would probably exist in a dedicated namespace to prevent breaking old code.

I am posting this question both as a feature request and a central place for people to vote.

core.async: Add a function to set thread executor

Sat, 25 Feb 2023 11:21:02 +0000

See slack #babashka discussion https://clojurians.slack.com/archives/CLX41ASCS/p1677319666536479

The use case is setting a virtual thread pool executor with SCI.
In Clojure I could set clojure.core.async.impl.exec.threadpool/thread-pool-executor.

Doing the intern in SCI will not have the desired effect since the pre-compiled functions don't see the change.

Clojure.core agents also allow this via set-agent-send-executor! etc.

Fix for the ASYNC-163 - Off by two error in pipeline-async

Tue, 22 Nov 2022 17:48:51 +0000

Hello!

I've been working on a fix for the ASYNC-163 bug, and I think I've got a robust solution that doesn't require many changes to the pipeline* function.

I'm new to contributing to Clojure, so I'm a bit unsure about the process, though I've read about it on the clojure.org website. There's already an existing ticket for the problem, which I've linked above, however, it was closed as "won't fix", and I don't have a Jira account to comment there and send a patch, so I've decided to post here.

After studying the source code of the pipeline* function, I've understood the whole process of how jobs are passed between go-threads and channels and found a simple fix, that requires adding one more channel to synchronize job-producer and result-consumer threads.
I have a more in-depth explanation of the bug in my blog. It is rather long and convoluted, so I think it would be best if I will not repeat myself here, making this question unnecessarily long and much harder to understand. The explanation is available here.

I'm attaching a patch here, feel free to point me in the right direction if this place is not for this type of question. I've tested the in the REPL with various scenarios and in one of my projects, that uses regular pipeline-async, it works correctly. In addition, it passes tests in the core.async library, which I ran with the lein test command.

Patch: https://andreyorst.gitlab.io/0001-fix-ASYNC-163.patch

At the moment I have some problems with running CLJS tests with or without my patch for the async.cljs file, but once I figure out the problem I'll post an updated patch, for both JVM and CLJS runtimes

Edit: Nov 25 2022

Patch v2: https://andreyorst.gitlab.io/0001-fix-ASYNC-163-2.patch

This patch includes the same fix for async.cljs, though I wasn't able to run tests for some reason - I get the Uncaught TypeError: process.on is not a function error in the browser. Not sure what I'm missing, the project's readme doesn't mention any requirements besides building with lein and opening the HTML file. But the fix is essentially the same, so it should work fine.

Edit Dec 08 2022

I've put the fixed version of pipeline* into a library pipeline-extras. It also has unordered versions of all pipelines, which should have higher throughput because the conveyer doesn't stop if any of the tasks take longer to complete than the other ones.

core.async readme oversight

Wed, 09 Nov 2022 10:55:02 +0000

https://github.com/clojure/core.async/blob/96cb49e15b046e17e5b73eee5f9ce61ab44883ba/README.md?plain=true#L17 refers to 1.5, while the latest at the moment is 1.6

core.async mult test is passing accidentally

Wed, 03 Aug 2022 17:37:17 +0000

Think I’ve found a bug in core.async tests, the following test for mult working seems purely coincidental:

;; ASYNC-127 
(let [ch (to-chan! [1 2 3])
      m (mult ch)
      t-1 (chan)
      t-2 (chan)
      t-3 (chan)]
  (tap m t-1)
  (tap m t-2)
  (tap m t-3)
  (close! t-3)
  (is (= 1 (<!! t-1)))
  (is (= nil (a/poll! t-1))) ;; t-2 hasn't taken yet
  (is (= 1 (<!! t-2)))
  (is (= 2 (<!! t-1))) ;; now available
  (is (= nil (a/poll! t-1)))))

The docstring of core.async/mult states that: Items received when there are no taps get dropped. Following this statement, the 3 items put by to-chan! should be dropped, as when the mult is created, it would start consuming them without taps yet on the mult.My suspicion is that it passes because the go’s created inside mult and to-chan! might be slow enough to create so that the taps get executed before the mult starts consuming.

Does this make sense? Am I missing something?

EDIT: Found this issue https://clojure.atlassian.net/browse/ASYNC-246 which seems to be related, in that case probably hangs because the items might be properly dropped and then (<!! t-1) hangs

Literal `()` erroneously produces `nil` in cljs.core.async `go` blocks

Sat, 29 Jan 2022 19:26:41 +0000

(go (println () [] (range 0) (list) {} #{}))

This prints

nil [] () () {} #{}

But it should print (which it properly does outside of go blocks)

() [] () () {} #{}

This appears to only be an issue in ClojureScript.

Clojurescript core.async go block incorrectly handles records returned from tagged literals

Fri, 27 Aug 2021 18:53:31 +0000

In clojurescript, I have a data reader that returns a record from a tagged literal:

(ns x.x)

(defrecord X [y])

(defn make-x [y] (->X y))

data_readers.cljc:

{x x.x/make-x}

This doesn't work correctly inside of go blocks:

(println "outside of go:" #x"a")
(go
  (println "inside of go:" (<! (go #x"b"))))

Results in:

outside of go: #x.x.X{:y a}
inside of go: {:y b}

@hiredman on the clojurians slack pointed out the issue:

what is happening is here
https://github.com/clojure/core.async/blob/master/src/main/clojure/cljs/core/async/impl/ioc_macros.clj#L403

map? is returning true so it is compiling your record as a map

How to work with AsyncIterable Interface in CLJS

Wed, 04 Aug 2021 03:10:13 +0000

Does clojurescript has any way to work with AsyncIterable inteface in JS. The JS way is to use for await of .
For normal async/await or promise, I found - https://clojurescript.org/guides/promise-interop

I'm trying to build a IPFS file browser and its JS api returns AsyncIterable.

Strategies for diagnosing core.async code that is unexpectedly blocking the main thread?

Sun, 14 Feb 2021 21:07:06 +0000

I'm new to core.async (and CSP more generally) and struggling to find good debugging strategies. In particular, I frequently fail to diagnosis which part of the code is infinitely blocking the main thread. I'll give a particular example:

(deftest passing-test
  (let [c (async/to-chan! [2 1 0 5 4])]
    (is (= [2 1 0 5 4]
           (async/<!! (async/into [] c))))))


(deftest hanging-test
  (let [[e o] (->> (async/to-chan! [2 1 0 5 4])
                   (async/split even?))]
    (is (= [2 0 4]
           (async/<!! (async/into [] e))))
    (is (= [1 5]
           (async/<!! (async/into [] o))))))

The first test passes and does not block the main thread. The second test hangs forever. The only difference is the addition of async/split to route even and odd integers to their respective channels.

My specific questions are as follows:

What explains the difference between the first and second tests with respect to blocking the main thread?
What are some strategies to debugging core.async code that is unexpectedly blocking the main thread?
What is the best way to write unit tests for code that creates and manipulates channels? Is (async/<!! (async/into [] o)) a reasonable way to collect the values flowing through a channel into a collection?

cljs core.async bound symbol `new` conflicts with fn `new`causing errors

Mon, 01 Feb 2021 07:39:32 +0000

Hi there,

it seems to me I found a bug in cljs core.async 1.3.610 (clj works well). Do you guys agree with me?

The issue:

(cljs.core.async/go
  (println "XXX before")     
  (let [old 1
      new „my-new"]
   (println "XXX after")
   (println [old new])))
=> „XXX vorher"
=> „XXX nacher"
=> TypeError TypeError: 1.call is not a function

Changing the new symbol to xnew works:

(cljs.core.async/go
  (println "XXX before")     
  (let [old 1
      xnew „my-new"]
   (println "XXX after")
   (println [old xnew])))
=> „XXX vorher"
=> „XXX nacher"
=> [1 „my-new"]

The following snippet of macroexpanding shows the error I mean. The new for instantiating cljs.core/PersistentVector got resolved with the value I bound to new

...
(clojure.core/let
 [inst_35971
 (println "XXX before")
 inst_35972
 (. cljs.core/PersistentVector -EMPTY-NODE)
 inst_35973
 (js* "[{},{}]" 1 "my-new")
 inst_35974
 ("my-new"
  cljs.core/PersistentVector
  nil
  2
  5
  inst_35972
  inst_35973
  nil)
 inst_35975
 (println inst_35974)
 state_35977
 …]
…))

So do not use new as symbol! :-$

Is "port" inside core.asyn documentation same as a "chan"?

Tue, 29 Dec 2020 23:51:09 +0000

Inside the documentation for core.async, the identifier port is used as an input to some functions/forms. Is port the same as chan? Or is it an interface or a protocol implemented by chan?

How to do a blocking put on a channel ?

Tue, 08 Dec 2020 19:43:21 +0000

I am getting the following error

No more than 1024 pending takes are allowed on a single channel

However, I though that if i defined a channel with a specifed size , the puts would block.

(def event-chan (chan 10))

I am calling this in a loop

(>!! event-chan evt)

My expectation is that it would block is there were pending 10 items in the channel .. however does not seem to be happening. My code is reading a fifo queue in a database and proceesiing it. it seems to not block even if the items are pending in the channel

edit: updated to copy paste the full actual code.

(defn process-edn [{:keys [data code]}]
; lots db processing code here
)
  
(def event-chan (chan 10))
  
(defn eat-now-or-later[evt]
    (if ( = (evt :code) :loadevt)
        (do (println "sync processing load event ")
            (process-edn evt))
        (>!! event-chan evt)
      ))
  
  (defn async-deque [shutdown?]
    (time! dq-call-processing-time
           (let [k-t (ftup/from commons/*JOB_FIFO_EVENT_QUEUE*)
                 fdb (cfdb/select-api-version cfdb/clj-fdb-api-version)]
             (go (while true (process-edn (<! event-chan))))

             (loop[evt (with-open [^Database db (cfdb/open fdb)] (fifo/dequeue db k-t))]
              
              (if (or (shutdown?)  (nil? evt)) 0
                  
                  (do                   
                   (eat-now-or-later evt)  
                   (recur (with-open [^Database db (cfdb/open fdb)] (fifo/dequeue db k-t))))
                  )
              )))
    (ptasks/print-stats))

The above is the function definition. I am calling this function from the daemon code below

(ns dqdaemon.core
  (:require [shutdown.core :as shutdown])
  (:require [com.videocloudmanager.petikaa.dequeue-tasks :as dqt])
 )

;; A crude approximation of your application's state.
(def state (atom {}))

(defn shutdown?[] (false? (:running @state)))

(defn init [args]
  (swap! state assoc :running true)
  (shutdown/add-hook! ::descriptive-name #(do
                                            (swap! state assoc :running false)         
                                            (println "It's about to go down!")))
  )

(defn start []
  (while (:running @state)
    (println "tick")
     (dqt/async-deque shutdown?)
    (Thread/sleep 2000)))


;; Enable command-line invocation
(defn -main [& args]
  (init args)
  (start))

Update:
I understand that this is not an issue with blocking puts. It is because my code is creating unlimited goloops.

This code is being called in a loop and the parent method of this loop , async-dequ, is being called in a forever loop.

(go (while true (process-edn (<! event-chan)))))

I am moving this code out of the loop-recur and into the main method to initialize when the daemon is started

(CLJS) finally-block prevents exception throw in go-block

Mon, 07 Sep 2020 12:47:54 +0000

ClojureScript 1.10.520, could not reproduce in Clojure. Reproduced in core.async 1.3.610 and 1.1.582, not reproducible in 1.0.567.

Looks the same as https://clojure.atlassian.net/browse/ASYNC-198?focusedCommentId=12026.

Code:

(ns app.main
  (:require [clojure.core.async :as a]))

(a/go
  (try
    (prn :ret
         (try
           (throw (ex-info "expected" {}))

           (catch js/Error e
             (prn :catch1)
             (throw e))

           (finally
             (prn :finally))))

    (catch js/Error _e
      (prn :catch2))))

Expected:

:catch1
:finally
:catch2

Actual:

:catch1
:finally
:ret #error {:message "expected", :data {}}

Infinite loop with stateful transducer on core.async channel

Sat, 08 Aug 2020 13:03:55 +0000

I am trying to implement a simple stateful transducer that counts the number of items (with ClojureScript):

(defn stateful-counter []
  (fn [xf]
    (let [counter (atom 0)]

      (fn
        ([] (xf))

        ([result]
         (xf (xf result @counter)))

        ([result _]
          (swap! counter inc)
          result)))))

When running this on a sequence I get the output:

(into [] (stateful-counter) (range 5))

[5]

This is exactly what I expect.

When running this on a core.async channel I get an infinite sequence of 5s:

(go (println
     (<! (let [c (async/chan 1 (stateful-counter))]

           (async/onto-chan! c (range 5))

           (async/into []
                       (async/take 10 c))))))

[5 5 5 5 5 5 5 5 5 5]

If I don't use (async/take 10 _) there seems to be an infinite loop. The expected outcome would be [5].

I have also tried using net.cgrand.xforms/count from the xforms library and get the same unexpected result:

(go (println
     (<! (let [c (async/chan 1 xforms/count)]

           (async/onto-chan! c (range 5))

           (async/into []
                       (async/take 10 c))))))

I was assuming that I made an implementation error in stateful-counter that caused the loop. But am confused that xforms/count produces the same (unexpected) result.

This is a ClojureScript snippet, but I have been able to produce the same unexpected result with Clojure.

Can somebody help me understand why applying the stateful transducer on a core.async channel results in an infinite sequence?

EDIT: Same unexpected behavior seen with Clojure.

how to consume the chan return by go block in cljs

Tue, 21 Jul 2020 00:19:03 +0000

I follow this article to play with cljs, but wonder how to wait go block in the top level, the problem arise when eval go block in REPL, which return #object[cljs.core.async.impl.channels.ManyToManyChannel].

After compiled to file, and exec via node target/main.js, it seems top level go block get blocked, waiting for its execution.

I suspect the difference between those two execution ways is related to how go is implement, or how event loop. works, but can't be sure.

(go ;; how to wait this ??
  (let [browser (<! (u/new-browser))
        page (<p! (.newPage browser))]
    (try
      (<p! (.goto page "https://clojure.org"))
      (<p! (.screenshot page #js{:path "./target/screenshot.png"}))
      (catch js/Error err
        (println (ex-cause err)))
      (finally (<p! (.close browser))))))  ;; eval here return ManyToManyChannel

(defn new-browser [& {:as opts}]
  (let [pp-opts (-> opts
        (update :user-data-dir #(or % (str home-dir "/pp")))
        (clj->js :keyword-fn csk/->camelCase))]
    (go (<p! (.launch puppeteer pp-opts)))))

A lazy sequence backed by a channel

Wed, 03 Jun 2020 13:22:25 +0000

I'm trying to convert a channel into a seq. This is what I'm using currently:

(defn chan->seq [ch]
  (when [v (<!! ch)]
    (lazy-seq (cons v (chan->seq ch)))))

It works but it uses a blocking read, so the current thread will be blocked until the channel is closed, and you can't have many calls of this function active at the same time. Is it possible to have a non-blocking version of this function?

This example demonstrates this issue:

(doseq [_ (range 20)]
  (go (chan->seq (chan 10))))

(go (println :OK))

The last statement won't print anything since chan->seq calls are squatting on all the threads in the threadpool. It works if 20 is changed to 5 (in a new repl session).

Add additional arity to the ex-handler [core.async]

Mon, 20 Apr 2020 22:53:25 +0000

I find it limiting that ex-handler for async/pipeline(-blocking) accepts an exception only, but not the value for which that exception occurred. Another arity (fn [ex v]) would be useful for use cases such as retrying, better error reporting etc. I guess it could be supported in a backward-compatible way via reflection arity checking (e.g. https://stackoverflow.com/a/1813967).

Is this the correct way to use go-block?

Thu, 02 Apr 2020 09:52:47 +0000

In general, we want to avoiding blocking IO and make it parking. Let us say we make saving to DB nonblocking by a go block.

(defn save-user-info [user]
   (go (...)))

Now if I want to use this function, I want to call it in a go block since I do not want blocking.

(defn register-user [user other-info]
  (go (validate-user ...)
      ...
      (<! (save-user-info user))))

The function save-user and register-user can be used in many places and then we need another go block.

(defn send-welcome-email [user params]
    (go ...
        (<! (register-user user))
        ...))

And again send-welcome-email can be used in other functions (say last-fn) and we then need another go block. So we have created a chain of functions using go since the "origin" one does.

Then if we call last-fn, we have to create several go blocks one-by-one.

My question is: is this the acceptable/correct way to use go? Or my question is wrong.

I am worried because it seems in order to execute one function, we need to create several go (and then channels) and GC them. it is not elegant some how. I do not understand core.async quite in deep but believe go is to make IO non-blocking. On the other side, if we think go as a state machine, then it seems we shall avoid IO in go and make things inside side-effect free.

core.async thread pool vs willy-nilly Java threads?

Sun, 29 Mar 2020 16:51:04 +0000

core.async has a thread pool, limited by Java system property "clojure.core.async.pool-size". But the following little REPL interaction (briefly) creates thousands of Java threads.

(require '[clojure.core.async :as a])
(def p (a/promise-chan))
(def a (atom 0))
(def b (atom 0))
(def cc (into [] (repeatedly 10000 (fn []
                                     (a/go
                                       (swap! a inc)
                                       (a/<! p)
                                       (swap! b inc))))))

Immediately upon the (def cc...), I noticed, among other things, that jstack (in another terminal) showed a brief increase of thousands of Java threads.

Question: When is the fixed-size thread pool used, vs creating a Java thread for every "go"?