From 5e5bf2cc8daf7726867322ef3fd8b330f7f4ffbb Mon Sep 17 00:00:00 2001 From: Ilan Steemers Date: Sun, 6 Sep 2015 14:48:33 +0200 Subject: [PATCH] docs: adjusted for multiple brokers --- README.rst | 2 +- docs/brokers.rst | 95 ++++++++++++++++++++++++++++++++++++++++++++++ docs/cluster.rst | 12 +++--- docs/configure.rst | 69 ++++++++++++++++++++++++++++----- docs/index.rst | 3 +- docs/install.rst | 12 +++--- 6 files changed, 171 insertions(+), 22 deletions(-) create mode 100644 docs/brokers.rst diff --git a/README.rst b/README.rst index dde7c86..b5e64ae 100644 --- a/README.rst +++ b/README.rst @@ -55,7 +55,7 @@ Installation $ python manage.py migrate - Make sure you have a `Redis `__ server running - somewhere + somewhere or configure one of the other `brokers `__. Read the full documentation at `https://django-q.readthedocs.org `__ diff --git a/docs/brokers.rst b/docs/brokers.rst new file mode 100644 index 0000000..e8b487f --- /dev/null +++ b/docs/brokers.rst @@ -0,0 +1,95 @@ +Brokers +======= + +The broker sits between your Django instances and your Django Q cluster instances, accepting and delivering task packages. +Currently we only support Redis and Disque, but support for other brokers is being worked on. +Even though `Disque `__ is still considered Alpha software, it's been gathering a lot of support and test results are positive. + +Clients for `Amazon SQS `__ and `IronMQ `__ are being tested. + + +Redis +----- +The default broker for Django Q clusters is `Redis `__. + +* Atomic +* Does not need separate cache framework for monitoring +* Can use existing `Django-Redis `__ connections through the :ref:`django_redis` setting +* Requires `Redis-py `__ as a client +* Does not support receipts + +Can be configured with :ref:`redis_configuration` configuration settings or :ref:`django_redis`. + + +Disque +------ +Unlike Redis, Disque supports message receipts which make delivery to the cluster workers guaranteed. If a task never produces a failed or successful result, it will automatically be sent to the cluster again for a retry. +You can control the amount of time Disque should wait for completion of a task by configuring the :ref:`retry` setting. + +* Supports receipts +* Atomic +* Needs Django's `Cache framework `__ configured for monitoring +* Compatible with `Tynd `__ Disque addon on `Heroku `__ +* Still considered Alpha software +* Requires `Redis-py `__ as a client + +See the :ref:`disque_configuration` configuration section for more info. + +Amazon SQS +---------- +*TBA* + + +Iron MQ +------- +*TBA* + +Reference +--------- +The :class:`Broker` class is used internally to communicate with the different types of brokers. +You can override this class if you want to contribute and support your own broker. + +.. py:class:: Broker + + .. py:method:: enqueue(task) + + Sends a task package to the broker queue and returns a tracking id. + + .. py:method:: dequeue() + + Gets a task package from the broker. + + .. py:method:: acknowledge(id) + + Notifies the broker that the task has been processed. + Only works with brokers that support delivery receipts. + + .. py:method:: fail(id) + + Tells the broker that the message failed to be processed by the cluster. + Only available on brokers that support this. + Currently only occurs when a cluster fails to unpack a task package. + + .. py:method:: delete(id) + + Instructs the broker to delete this message from the queue. + + .. py:method:: purge_queue() + + Empties the current queue of all messages. + + .. py:method:: delete_queue() + + Deletes the current queue from the broker. + + .. py:method:: queue_size() + + Returns the amount of messages in the brokers queue. + + .. py:method:: ping() + + Returns True if the broker can be reached. + + .. py:method:: info() + + Shows the name and version of the currently configured broker. diff --git a/docs/cluster.rst b/docs/cluster.rst index 4911889..278521a 100644 --- a/docs/cluster.rst +++ b/docs/cluster.rst @@ -48,7 +48,7 @@ Multiple Clusters ----------------- You can have multiple clusters on multiple machines, working on the same queue as long as: -- They connect to the same Redis server or Redis cluster. +- They connect to the same :doc:`broker`. - They use the same cluster name. See :doc:`configure` - They share the same ``SECRET_KEY`` for Django. @@ -92,15 +92,16 @@ Architecture Signed Tasks """""""""""" -Tasks are first pickled and then signed using Django's own :mod:`django.core.signing` module using the ``SECRET_KEY`` and cluster name as salt, before being sent to a Redis list. This ensures that task -packages on the Redis server can only be executed and read by clusters +Tasks are first pickled and then signed using Django's own :mod:`django.core.signing` module using the ``SECRET_KEY`` and cluster name as salt, before being sent to a message broker. This ensures that task +packages on the broker can only be executed and read by clusters and django servers who share the same secret key and cluster name. -Optionally the packages can be compressed before transport +If a package fails to unpack, it will be marked failed with the broker and discarded. +Optionally the packages can be compressed before transport. Pusher """""" -The pusher process continuously checks the Redis list for new task +The pusher process continuously checks the broker for new task packages. It checks the signing and unpacks the task to the Task Queue. Worker @@ -115,6 +116,7 @@ Monitor The result monitor checks the Result Queue for processed packages and saves both failed and successful packages to the Django database. +If the broker supports it, a delivery receipt is sent. .. _sentinel: diff --git a/docs/configure.rst b/docs/configure.rst index fe76ffe..a2bdeac 100644 --- a/docs/configure.rst +++ b/docs/configure.rst @@ -52,6 +52,14 @@ timeout The number of seconds a worker is allowed to spend on a task before it's terminated. Defaults to ``None``, meaning it will never time out. Set this to something that makes sense for your project. Can be overridden for individual tasks. +.. _retry: + +retry +~~~~~ + +The number of seconds a broker will wait for a cluster to finish a task, before it's presented again. +Only works with brokers that support delivery receipts. Defaults to 60 seconds. + compress ~~~~~~~~ @@ -100,20 +108,25 @@ The default behavior for schedules that didn't run while a cluster was down, is You can override this behavior by setting ``catch_up`` to ``False``. This will make those schedules run only once when the cluster starts and normal scheduling resumes. Defaults to ``True``. +.. _redis_configuration: + redis ~~~~~ Connection settings for Redis. Defaults:: - redis: { - 'host': 'localhost', - 'port': 6379, - 'db': 0, - 'password': None, - 'socket_timeout': None, - 'charset': 'utf-8', - 'errors': 'strict', - 'unix_socket_path': None + # redis defaults + Q_CLUSTER = { + 'redis': { + 'host': 'localhost', + 'port': 6379, + 'db': 0, + 'password': None, + 'socket_timeout': None, + 'charset': 'utf-8', + 'errors': 'strict', + 'unix_socket_path': None + } } For more information on these settings please refer to the `Redis-py `__ documentation @@ -131,7 +144,7 @@ of the cache connection you want to use:: 'name': 'DJRedis', 'workers': 4, 'timeout': 90, - 'django_redis: 'default' + 'django_redis': 'default' } @@ -139,6 +152,42 @@ of the cache connection you want to use:: .. tip:: Django Q uses your ``SECRET_KEY`` to encrypt task packages and prevent task crossover. So make sure you have it set up in your Django settings. +.. _disque_configuration: + +disque_nodes +~~~~~~~~~~~~ +If you want to use Disque as your broker, set this to a list of available Disque nodes and each cluster will randomly try to connect to them:: + + # example disque connection + Q_CLUSTER = { + 'name': 'DisqueBroker', + 'workers': 4, + 'timeout': 60, + 'retry': 60, + 'disque_nodes': ['127.0.0.1:7711', '127.0.0.1:7712'] + } + + +Django Q is also compatible with the `Tynd `__ addon on `Heroku `__:: + + # example Tynd connection + import os + + Q_CLUSTER = { + 'name': 'TyndBroker', + 'workers': 8, + 'timeout': 30, + 'retry': 60, + 'disque_nodes': os.environ['TYND_DISQUE_NODES'].split(','), + 'disque_auth': os.environ['TYND_DISQUE_AUTH'] + } + + +disque_auth +~~~~~~~~~~~ + +Optional Disque password for servers that require authentication. + cpu_affinity ~~~~~~~~~~~~ diff --git a/docs/index.rst b/docs/index.rst index 3bad6a1..5d38a22 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -20,7 +20,7 @@ Features - Django Admin integration - PaaS compatible with multiple instances - Multi cluster monitor -- Redis broker +- Redis or Disque broker - Python 2 and 3 @@ -33,6 +33,7 @@ Contents: Installation Configuration + Brokers Tasks Schedules Cluster diff --git a/docs/install.rst b/docs/install.rst index 1297921..f63f3de 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -19,7 +19,8 @@ Installation $ python manage.py migrate - Make sure you have a `Redis `__ server running - somewhere and know how to connect to it. + somewhere and know how to connect to it or configure one of the alternative :doc:`brokers`. + Requirements ------------ @@ -47,10 +48,6 @@ Django Q is tested for Python 2.7 and 3.4 This feature-filled fork of Erik Rose's blessings project provides the terminal layout of the monitor. -- `Redis server `__ - - Django Q uses Redis as a centralized hub between your Django instances and your Q clusters. - Optional ~~~~~~~~ @@ -65,3 +62,8 @@ Optional $ pip install hiredis +- `Redis `__ server is the default broker for Django Q. It provides the best performance and does not require Django's cache framework for monitoring. + +- `Disque `__ server is based on Redis by the same author, but focuses on reliable queues. Currently in Alpha, but highly recommended. You can either build it from source or use it on Heroku through the `Tynd `__ beta. + +