Task Queues

A list of message brokers and task queue libraries spanning many programming languages and implementations.

Newsletter

Subscribe to our newsletter to get periodic updates when we make changes.


Brokers

Message brokers distribute messages from producers to consumers.

Amazon SQS

https://aws.amazon.com/sqs/

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that makes it easy to decouple and scale microservices, distributed systems, and serverless applications. Building applications from individual components that each perform a discrete function improves scalability and reliability, and is best practice design for modern applications.

Apache Kafka

http://kafka.apache.org/

Kafka® is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.

Apache Pulsar

http://pulsar.incubator.apache.org/

Apache Pulsar is an open-source distributed pub-sub messaging system originally created at Yahoo and now part of the Apache Software Foundation.

Apache Qpid

http://qpid.apache.org/

Apache Qpid™ makes messaging tools that speak AMQP and support many languages and platforms.

Apache RocketMQ

https://rocketmq.apache.org/

Apache RocketMQ™ is an open source distributed messaging and streaming data platform.

Azure Service Bus

https://azure.microsoft.com/en-us/services/service-bus/

Depend on Azure Service Bus when you need highly-reliable cloud messaging service between applications and services, even when one or more is offline.

Beanstalk

http://kr.github.io/beanstalkd/

Beanstalk is a simple, fast work queue. Its interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously.

Bedrock::Jobs

http://bedrockdb.com/jobs.html

Bedrock::Jobs is a plugin to the Bedrock data foundation that manages a scheduled job queue.

Faktory

http://contribsys.com/faktory/

At a high level, Faktory is a work server. It is the repository for background jobs within your application. Jobs have a type and a set of arguments and are placed into queues for workers to fetch and execute.

Gearman

http://gearman.org/

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events. In other words, it is the nervous system for how distributed processing communicates.

Google Cloud PubSub

https://cloud.google.com/pubsub/docs/

Google Cloud Pub/Sub is a fully-managed real-time messaging service that allows you to send and receive messages between independent applications.

IronMQ

http://www.iron.io/mq

MQ provides a reliable way to communicate between services and components. Highly available, persistent by design, with best-effort one-time delivery, MQ is the most industrial strength, cloud-native solution for modern application architecture.

Mosquitto

https://mosquitto.org/

Eclipse Mosquitto™ is an open source (EPL/EDL licensed) message broker that implements the MQTT protocol versions 3.1 and 3.1.1. MQTT provides a lightweight method of carrying out messaging using a publish/subscribe model. This makes it suitable for Internet of Things messaging such as with low power sensors or mobile devices such as phones, embedded computers or microcontrollers like the Arduino.

NATS

https://nats.io/

NATS is an open source, lightweight, high-performance cloud native infrastructure messaging system. It implements a highly scalable and elegant publish-subscribe (pub/sub) distribution model. The performant nature of NATS make it an ideal base for building modern, reliable, scalable cloud native distributed systems.

NSQ

http://nsq.io/

NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee.

RabbitMQ

http://rabbitmq.com/

RabbitMQ is lightweight and easy to deploy on premises and in the cloud. It supports multiple messaging protocols. RabbitMQ can be deployed in distributed and federated configurations to meet high-scale, high-availability requirements.

Redis

https://redis.io

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.

Libraries and Frameworks

Task queue libraries generally provide higher-level and language specific abstractions over message brokers.

Bee Queue

https://github.com/bee-queue/bee-queue

Redis Node.js

A simple, fast, robust job/task queue for Node.js, backed by Redis.

npm install --save bee-queue

Celery

http://celeryproject.org/

Amazon SQS RabbitMQ Redis Python PyPy

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

pipenv install celery

Delayed::Job

https://github.com/collectiveidea/delayed_job

Redis Ruby

Delayed::Job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background.

echo "gem 'delayed_job_active_record'" >> Gemfile
bundle install

Dramatiq

https://dramatiq.io

Memory RabbitMQ Redis Python PyPy

Dramatiq is an alternative to Celery with a focus on simplicity, reliability and performance.

pipenv install dramatiq

exq

https://github.com/akira/exq

Redis Elixir

Exq is a job processing library compatible with Resque / Sidekiq for the Elixir language.

echo >mix.exs <<EOF
defp deps do
  [
    {:exq, "~> 0.9.1"}
  ]
end
EOF
mix deps.get

huey

https://huey.readthedocs.io/en/latest/

Redis Python

A lightweight alternative.

pipenv install huey

Kue

http://automattic.github.io/kue/

Redis Node.js

Kue is a feature rich priority job queue for node.js backed by redis. A key feature of Kue is its clean user-interface for viewing and managing queued, active, failed, and completed jobs.

npm install --save kue

Laravel

https://laravel.com/docs/5.5/queues

Laravel queues provide a unified API across a variety of different queue backends, such as Beanstalk, Amazon SQS, Redis, or even a relational database. Queues allow you to defer the processing of a time consuming task, such as sending an email, until a later time. Deferring these time consuming tasks drastically speeds up web requests to your application.

composer global require "laravel/installer"

Machinery

https://github.com/RichardKnop/machinery

Machinery is an asynchronous task queue/job queue based on distributed message passing.

go get github.com/RichardKnop/machinery/v1

MassTransit

http://masstransit-project.com/

MassTransit is a free, open source, lightweight message bus for creating distributed applications using the .NET framework. MassTransit provides an extensive set of features on top existing message transports, resulting in a developer friendly way to asynchronously connect services using message-based conversation patterns. Message-based communication is a reliable and scalable way to implement a service oriented architecture.

nuget Install-Package MassTransit

node-rethinkdb-job-queue

https://github.com/grantcarthew/node-rethinkdb-job-queue

RethinkDB Node.js

rethinkdb-job-queue is a persistent job or task queue backed by RethinkDB. It has been built as an alternative to the many queues available on NPM.

npm install --save rethinkdb-job-queue

queue_classic

https://github.com/QueueClassic/queue_classic

PostgreSQL Ruby

queue_classic provides a simple interface to a PostgreSQL-backed message queue. queue_classic specializes in concurrent locking and minimizing database load while providing a simple, intuitive developer experience. queue_classic assumes that you are already using PostgreSQL in your production environment and that adding another dependency (e.g. redis, beanstalkd, 0mq) is undesirable.

echo "gem 'queue_classic'" >> Gemfile
bundle install
export QC_DATABASE_URL="postgres://username:password@localhost/your_database"
ruby -r queue_classic -e "QC::Setup.create"

Resque

https://github.com/resque/resque

Redis Ruby

Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.

echo "gem 'resque'" >> Gemfile
bundle install

RQ

http://python-rq.org/

Redis Python

RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. It is backed by Redis and it is designed to have a low barrier to entry. It can be integrated in your web stack easily.

pipenv install rq

Sidekiq

http://sidekiq.org/

Redis Ruby

Sidekiq is a full-featured background processing framework for Ruby. It aims to be simple to integrate with any modern Rails application and much higher performance than other existing solutions.

echo "gem 'sidekiq'" >> Gemfile
bundle install

Sidekiq.cr

https://github.com/mperham/sidekiq.cr

Redis Crystal

A version of Sidekiq for the Crystal language.

echo >shards.yml <<EOF
dependencies:
  sidekiq:
    github: mperham/sidekiq.cr
    branch: master
EOF
shards update

Toniq

https://github.com/joakimk/toniq

Redis Elixir

Simple and reliable background job processing library for Elixir.

echo >mix.exs <<EOF
defp deps do
  [
    {:exredis, ">= 0.1.1"},
    {:toniq, "~> 1.0"}
  ]
end
EOF
mix deps.get