Task Queues

A list of message brokers and task queue libraries spanning many programming languages and implementations.

Brokers

Message brokers distribute messages from producers to consumers.

Amazon SQS

https://aws.amazon.com/sqs/

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that makes it easy to decouple and scale microservices, distributed systems, and serverless applications. Building applications from individual components that each perform a discrete function improves scalability and reliability, and is best practice design for modern applications.

Apache ActiveMQ

http://activemq.apache.org/

Apache ActiveMQ™ is the most popular open source, multi-protocol, Java-based messaging server.

Apache Kafka

http://kafka.apache.org/

Kafka® is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.

Apache Pulsar

http://pulsar.incubator.apache.org/

Apache Pulsar is an open-source distributed pub-sub messaging system originally created at Yahoo and now part of the Apache Software Foundation.

Apache Qpid

http://qpid.apache.org/

Apache Qpid™ makes messaging tools that speak AMQP and support many languages and platforms.

Apache RocketMQ

https://rocketmq.apache.org/

Apache RocketMQ™ is an open source distributed messaging and streaming data platform.

Azure Service Bus

https://azure.microsoft.com/en-us/services/service-bus/

Depend on Azure Service Bus when you need highly-reliable cloud messaging service between applications and services, even when one or more is offline.

Beanstalk

http://kr.github.io/beanstalkd/

Beanstalk is a simple, fast work queue. Its interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously.

Bedrock::Jobs

http://bedrockdb.com/jobs.html

Bedrock::Jobs is a plugin to the Bedrock data foundation that manages a scheduled job queue.

Faktory

http://contribsys.com/faktory/

At a high level, Faktory is a work server. It is the repository for background jobs within your application. Jobs have a type and a set of arguments and are placed into queues for workers to fetch and execute.

Gearman

http://gearman.org/

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events. In other words, it is the nervous system for how distributed processing communicates.

Google Cloud PubSub

https://cloud.google.com/pubsub/docs/

Google Cloud Pub/Sub is a fully-managed real-time messaging service that allows you to send and receive messages between independent applications.

Inngest

https://www.inngest.com

Inngest is an open source event-driven queue in which serverless functions are triggered by HTTP events. Inngest does not require one to configure queues up front and instead, systems can just send events to start using in minutes. Inngest aims to be SDK-less and use standard libraries and interfaces to provide a better developer experience. Inngest can be self-hosted or one can use Inngest's cloud platform offering.

IronMQ

http://www.iron.io/mq

MQ provides a reliable way to communicate between services and components. Highly available, persistent by design, with best-effort one-time delivery, MQ is the most industrial strength, cloud-native solution for modern application architecture.

Mosquitto

https://mosquitto.org/

Eclipse Mosquitto™ is an open source (EPL/EDL licensed) message broker that implements the MQTT protocol versions 3.1 and 3.1.1. MQTT provides a lightweight method of carrying out messaging using a publish/subscribe model. This makes it suitable for Internet of Things messaging such as with low power sensors or mobile devices such as phones, embedded computers or microcontrollers like the Arduino.

NATS

https://nats.io/

NATS is an open source, lightweight, high-performance cloud native infrastructure messaging system. It implements a highly scalable and elegant publish-subscribe (pub/sub) distribution model. The performant nature of NATS make it an ideal base for building modern, reliable, scalable cloud native distributed systems.

NSQ

http://nsq.io/

NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee.

PostgreSQL

https://www.postgresql.org/

While PostgreSQL is not a message queue per se, it does support facilities for implementing queues via the "SELECT ... FOR UPDATE SKIP LOCKED" clause and the notification subsystem. Postgres-based message queues can be very convenient if you're already using Postgres as your database and you don't need too much throughput. Another advantage to Postgres is you can enqueue jobs transactionally with other database operations, causing related operations to be rolled back if a job can't be enqueued.

RabbitMQ

http://rabbitmq.com/

RabbitMQ is lightweight and easy to deploy on premises and in the cloud. It supports multiple messaging protocols. RabbitMQ can be deployed in distributed and federated configurations to meet high-scale, high-availability requirements.

Red Panda

https://redpanda.com

Redpanda is a Kafka®-compatible streaming data platform that is up to 10x faster and 6x more hardware-efficient. It is also JVM-free, ZooKeeper®-free, Jepsen-tested and source available.

Redis

https://redis.io

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.

ServerlessQ

https://serverlessq.com

ServerlessQ is a hosted message queue service for the serverless era of computing. Without any setup you can re-use your existing APIs and let ServerlessQ handle queueing and retrying your messages. You also get an overview of all your messages and error codes.

Zeplo

https://zeplo.io

Zeplo is a next generation message queue powered by HTTPS, no setup required. Supports delayed, scheduled and retry of messages. In-built console to debug and reprocess failed records. Infinitely scaleable.

Libraries and Frameworks

Task queue libraries generally provide higher-level and language specific abstractions over message brokers.

Bee Queue

https://github.com/bee-queue/bee-queue

Redis Node.js

A simple, fast, robust job/task queue for Node.js, backed by Redis.

npm install --save bee-queue

Celery

http://celeryproject.org/

Amazon SQS RabbitMQ Redis Python PyPy

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

pip install celery

db-queue

https://github.com/yoomoney-tech/db-queue

Db-queue provides a rich interface to database-backed message queue (PostgreSQL, Oracle, MSSQL). Db-queue offers a rich set of features and is designed to handle high throughput. Db-queue assumes that you are already using relational databases in your production environment and that adding another dependency is undesirable.

Delayed::Job

https://github.com/collectiveidea/delayed_job

Redis Ruby

Delayed::Job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background.

echo "gem 'delayed_job_active_record'" >> Gemfile
bundle install

Dramatiq

https://dramatiq.io

Memory RabbitMQ Redis Python PyPy

Dramatiq is an alternative to Celery with a focus on simplicity, reliability and performance.

pip install dramatiq

exq

https://github.com/akira/exq

Redis Elixir

Exq is a job processing library compatible with Resque / Sidekiq for the Elixir language.

echo >mix.exs <<EOF
defp deps do
  [
    {:exq, "~> 0.9.1"}
  ]
end
EOF
mix deps.get

faust

https://github.com/faust-streaming/faust

Kafka python

A library for building streaming applications in Python.

pip install faust-streaming

huey

https://huey.readthedocs.io/en/latest/

Redis Python

A lightweight alternative.

pip install huey

koyo

https://koyoweb.org/job/index.html

PostgreSQL Racket

Koyo is a web-development toolkit for Racket that comes with a built-in task queue based on PostgreSQL.

raco pkg install koyo

Kue

http://automattic.github.io/kue/

Redis Node.js

Kue is a feature rich priority job queue for node.js backed by redis. A key feature of Kue is its clean user-interface for viewing and managing queued, active, failed, and completed jobs.

npm install --save kue

Laravel

https://laravel.com/docs/5.5/queues

Laravel queues provide a unified API across a variety of different queue backends, such as Beanstalk, Amazon SQS, Redis, or even a relational database. Queues allow you to defer the processing of a time consuming task, such as sending an email, until a later time. Deferring these time consuming tasks drastically speeds up web requests to your application.

composer global require "laravel/installer"

Machinery

https://github.com/RichardKnop/machinery

Machinery is an asynchronous task queue/job queue based on distributed message passing.

go get github.com/RichardKnop/machinery/v1

MassTransit

http://masstransit-project.com/

MassTransit is a free, open source, lightweight message bus for creating distributed applications using the .NET framework. MassTransit provides an extensive set of features on top existing message transports, resulting in a developer friendly way to asynchronously connect services using message-based conversation patterns. Message-based communication is a reliable and scalable way to implement a service oriented architecture.

nuget Install-Package MassTransit

node-rethinkdb-job-queue

https://github.com/grantcarthew/node-rethinkdb-job-queue

RethinkDB Node.js

rethinkdb-job-queue is a persistent job or task queue backed by RethinkDB. It has been built as an alternative to the many queues available on NPM.

npm install --save rethinkdb-job-queue

queue_classic

https://github.com/QueueClassic/queue_classic

PostgreSQL Ruby

queue_classic provides a simple interface to a PostgreSQL-backed message queue. queue_classic specializes in concurrent locking and minimizing database load while providing a simple, intuitive developer experience. queue_classic assumes that you are already using PostgreSQL in your production environment and that adding another dependency (e.g. redis, beanstalkd, 0mq) is undesirable.

echo "gem 'queue_classic'" >> Gemfile
bundle install
export QC_DATABASE_URL="postgres://username:password@localhost/your_database"
ruby -r queue_classic -e "QC::Setup.create"

Resque

https://github.com/resque/resque

Redis Ruby

Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.

echo "gem 'resque'" >> Gemfile
bundle install

RQ

http://python-rq.org/

Redis Python

RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. It is backed by Redis and it is designed to have a low barrier to entry. It can be integrated in your web stack easily.

pip install rq

Sidekiq

http://sidekiq.org/

Redis Ruby

Sidekiq is a full-featured background processing framework for Ruby. It aims to be simple to integrate with any modern Rails application and much higher performance than other existing solutions.

echo "gem 'sidekiq'" >> Gemfile
bundle install

Sidekiq.cr

https://github.com/mperham/sidekiq.cr

Redis Crystal

A version of Sidekiq for the Crystal language.

echo >shards.yml <<EOF
dependencies:
  sidekiq:
    github: mperham/sidekiq.cr
    branch: master
EOF
shards update

Temporal

https://github.com/temporalio/temporal

Golang Java Ruby

Temporal is a microservice orchestration platform which enables developers to build scalable applications without sacrificing productivity or reliability. Temporal server executes units of application logic, Workflows, in a resilient manner that automatically handles intermittent failures, and retries failed operations.

curl -L https://github.com/temporalio/temporal/releases/latest/download/docker.tar.gz | tar -xz
cd docker
docker-compose up

Toniq

https://github.com/joakimk/toniq

Redis Elixir

Simple and reliable background job processing library for Elixir.

echo >mix.exs <<EOF
defp deps do
  [
    {:exredis, ">= 0.1.1"},
    {:toniq, "~> 1.0"}
  ]
end
EOF
mix deps.get

worker

https://github.com/graphile/worker

PostgreSQL Node.js SQL

High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)

ZeroMQ

https://zeromq.org/

C C++ C# Dart Erlang F# Go Haskell Java Node.js Perl Python Ruby Rust

ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems.