Background jobs in Clojure?

Question

In Ruby on Rails they have ActiveJob for implementing background jobs. Examples include delayed_job, resque, etc.

Is there something similar in Clojure?

Basically, I want to have a REST end-point invocation which queues a long-running task to be run. Though it returns something like a job id immediately, not waiting for the task to finish. The task should continue to run in the background until it completes.

Answer 1

There's Proletarian, which I made for exactly the use cases that delayed_job and Resque support. There's no REST interface, though - you enqueue jobs by invoking a function that writes to the job table. It's backed by Postgres, and can partake in Postgres transaction that you already use. This gives you guarantees that HTTP-based or Redis-based solutions can't, namely that the job is atomically persisted along with whatever other business data that was persisted in the transaction.

Comments

Thanks, this looks promising. I am using Postgres in my scenario.

Answer 2

Hi Edward,
You could look into nilenso/goose, a reliable and scalable background job processing library for Clojure; which handles Job failures and supports scheduled, cron and batch Jobs as well.

Goose supports Redis and RabbitMQ message brokers out of the box. If users need a different broker like AWS SQS or Postgres, Goose has provisions to plug-in a custom broker as well.

Please note that Goose Jobs cannot be created via an API endpoint, but using a function invocation akin to ActiveJobs in RoR.
I like your idea though. API invocation is a doable feature, with some limitations on data-types of arguments.

I have created an issue #144 in Goose to enable Job creation from HTTP endpoints. Thanks for the idea, Edward!