I've been working on a fix for the ASYNC-163 bug, and I think I've got a robust solution that doesn't require many changes to the
I'm new to contributing to Clojure, so I'm a bit unsure about the process, though I've read about it on the clojure.org website. There's already an existing ticket for the problem, which I've linked above, however, it was closed as "won't fix", and I don't have a Jira account to comment there and send a patch, so I've decided to post here.
After studying the source code of the
pipeline* function, I've understood the whole process of how jobs are passed between go-threads and channels and found a simple fix, that requires adding one more channel to synchronize job-producer and result-consumer threads.
I have a more in-depth explanation of the bug in my blog. It is rather long and convoluted, so I think it would be best if I will not repeat myself here, making this question unnecessarily long and much harder to understand. The explanation is available here.
I'm attaching a patch here, feel free to point me in the right direction if this place is not for this type of question. I've tested the in the REPL with various scenarios and in one of my projects, that uses regular
pipeline-async, it works correctly. In addition, it passes tests in the
core.async library, which I ran with the
lein test command.
At the moment I have some problems with running CLJS tests with or without my patch for the async.cljs file, but once I figure out the problem I'll post an updated patch, for both JVM and CLJS runtimes
Edit: Nov 25 2022
Patch v2: https://andreyorst.gitlab.io/0001-fix-ASYNC-163-2.patch
This patch includes the same fix for async.cljs, though I wasn't able to run tests for some reason - I get the
Uncaught TypeError: process.on is not a function error in the browser. Not sure what I'm missing, the project's readme doesn't mention any requirements besides building with lein and opening the HTML file. But the fix is essentially the same, so it should work fine.
Edit Dec 08 2022
I've put the fixed version of
pipeline* into a library pipeline-extras. It also has unordered versions of all pipelines, which should have higher throughput because the conveyer doesn't stop if any of the tasks take longer to complete than the other ones.