fork/django-q2
1
0
Fork 0
mirror of https://github.com/django-q2/django-q2.git synced 2026-09-15 13:37:56 +08:00
Code Issues Packages Projects Releases Wiki Activity

docs: adjusted for multiple brokers

Browse Source
This commit is contained in:
Ilan Steemers
2015-09-06 14:48:33 +02:00
parent 44473ca408
commit 5e5bf2cc8d
6 changed files with 171 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.rst
View File

@@ -55,7 +55,7 @@ Installation
$ python manage.py migrate
- Make sure you have a `Redis <http://redis.io/>`__ server running
somewhere
somewhere or configure one of the other `brokers <https://django-q.readthedocs.org/en/latest/brokers.html>`__.
Read the full documentation at `https://django-q.readthedocs.org <https://django-q.readthedocs.org>`__

95
docs/brokers.rst Normal file
View File

@@ -0,0 +1,95 @@
Brokers
=======
The broker sits between your Django instances and your Django Q cluster instances, accepting and delivering task packages.
Currently we only support Redis and Disque, but support for other brokers is being worked on.
Even though `Disque <https://github.com/antirez/disque>`__ is still considered Alpha software, it's been gathering a lot of support and test results are positive.
Clients for `Amazon SQS <https://aws.amazon.com/sqs/>`__ and `IronMQ <http://www.iron.io/mq/>`__ are being tested.
Redis
-----
The default broker for Django Q clusters is `Redis <http://redis.io/>`__.
* Atomic
* Does not need separate cache framework for monitoring
* Can use existing `Django-Redis <https://github.com/niwinz/django-redis>`__ connections through the :ref:`django_redis` setting
* Requires `Redis-py <https://github.com/andymccurdy/redis-py>`__ as a client
* Does not support receipts
Can be configured with :ref:`redis_configuration` configuration settings or :ref:`django_redis`.
Disque
------
Unlike Redis, Disque supports message receipts which make delivery to the cluster workers guaranteed. If a task never produces a failed or successful result, it will automatically be sent to the cluster again for a retry.
You can control the amount of time Disque should wait for completion of a task by configuring the :ref:`retry` setting.
* Supports receipts
* Atomic
* Needs Django's `Cache framework <https://docs.djangoproject.com/en/1.8/topics/cache/#setting-up-the-cache>`__ configured for monitoring
* Compatible with `Tynd <https://disque.tynd.co/>`__ Disque addon on `Heroku <https://heroku.com>`__
* Still considered Alpha software
* Requires `Redis-py <https://github.com/andymccurdy/redis-py>`__ as a client
See the :ref:`disque_configuration` configuration section for more info.
Amazon SQS
----------
*TBA*
Iron MQ
-------
*TBA*
Reference
---------
The :class:`Broker` class is used internally to communicate with the different types of brokers.
You can override this class if you want to contribute and support your own broker.
.. py:class:: Broker
.. py:method:: enqueue(task)
Sends a task package to the broker queue and returns a tracking id.
.. py:method:: dequeue()
Gets a task package from the broker.
.. py:method:: acknowledge(id)
Notifies the broker that the task has been processed.
Only works with brokers that support delivery receipts.
.. py:method:: fail(id)
Tells the broker that the message failed to be processed by the cluster.
Only available on brokers that support this.
Currently only occurs when a cluster fails to unpack a task package.
.. py:method:: delete(id)
Instructs the broker to delete this message from the queue.
.. py:method:: purge_queue()
Empties the current queue of all messages.
.. py:method:: delete_queue()
Deletes the current queue from the broker.
.. py:method:: queue_size()
Returns the amount of messages in the brokers queue.
.. py:method:: ping()
Returns True if the broker can be reached.
.. py:method:: info()
Shows the name and version of the currently configured broker.

12
docs/cluster.rst
View File

@@ -48,7 +48,7 @@ Multiple Clusters
-----------------
You can have multiple clusters on multiple machines, working on the same queue as long as:
- They connect to the same Redis server or Redis cluster.
- They connect to the same :doc:`broker<brokers>`.
- They use the same cluster name. See :doc:`configure`
- They share the same ``SECRET_KEY`` for Django.
@@ -92,15 +92,16 @@ Architecture
Signed Tasks
""""""""""""
Tasks are first pickled and then signed using Django's own :mod:`django.core.signing` module using the ``SECRET_KEY`` and cluster name as salt, before being sent to a Redis list. This ensures that task
packages on the Redis server can only be executed and read by clusters
Tasks are first pickled and then signed using Django's own :mod:`django.core.signing` module using the ``SECRET_KEY`` and cluster name as salt, before being sent to a message broker. This ensures that task
packages on the broker can only be executed and read by clusters
and django servers who share the same secret key and cluster name.
Optionally the packages can be compressed before transport
If a package fails to unpack, it will be marked failed with the broker and discarded.
Optionally the packages can be compressed before transport.
Pusher
""""""
The pusher process continuously checks the Redis list for new task
The pusher process continuously checks the broker for new task
packages. It checks the signing and unpacks the task to the Task Queue.
Worker
@@ -115,6 +116,7 @@ Monitor
The result monitor checks the Result Queue for processed packages and
saves both failed and successful packages to the Django database.
If the broker supports it, a delivery receipt is sent.
.. _sentinel:

69
docs/configure.rst
View File

@@ -52,6 +52,14 @@ timeout
The number of seconds a worker is allowed to spend on a task before it's terminated. Defaults to ``None``, meaning it will never time out.
Set this to something that makes sense for your project. Can be overridden for individual tasks.
.. _retry:
retry
~~~~~
The number of seconds a broker will wait for a cluster to finish a task, before it's presented again.
Only works with brokers that support delivery receipts. Defaults to 60 seconds.
compress
~~~~~~~~
@@ -100,20 +108,25 @@ The default behavior for schedules that didn't run while a cluster was down, is
You can override this behavior by setting ``catch_up`` to ``False``. This will make those schedules run only once when the cluster starts and normal scheduling resumes.
Defaults to ``True``.
.. _redis_configuration:
redis
~~~~~
Connection settings for Redis. Defaults::
redis: {
'host': 'localhost',
'port': 6379,
'db': 0,
'password': None,
'socket_timeout': None,
'charset': 'utf-8',
'errors': 'strict',
'unix_socket_path': None
# redis defaults
Q_CLUSTER = {
'redis': {
'host': 'localhost',
'port': 6379,
'db': 0,
'password': None,
'socket_timeout': None,
'charset': 'utf-8',
'errors': 'strict',
'unix_socket_path': None
}
}
For more information on these settings please refer to the `Redis-py <https://github.com/andymccurdy/redis-py>`__ documentation
@@ -131,7 +144,7 @@ of the cache connection you want to use::
'name': 'DJRedis',
'workers': 4,
'timeout': 90,
'django_redis: 'default'
'django_redis': 'default'
}
@@ -139,6 +152,42 @@ of the cache connection you want to use::
.. tip::
Django Q uses your ``SECRET_KEY`` to encrypt task packages and prevent task crossover. So make sure you have it set up in your Django settings.
.. _disque_configuration:
disque_nodes
~~~~~~~~~~~~
If you want to use Disque as your broker, set this to a list of available Disque nodes and each cluster will randomly try to connect to them::
# example disque connection
Q_CLUSTER = {
'name': 'DisqueBroker',
'workers': 4,
'timeout': 60,
'retry': 60,
'disque_nodes': ['127.0.0.1:7711', '127.0.0.1:7712']
}
Django Q is also compatible with the `Tynd <https://disque.tynd.co/>`__ addon on `Heroku <https://heroku.com>`__::
# example Tynd connection
import os
Q_CLUSTER = {
'name': 'TyndBroker',
'workers': 8,
'timeout': 30,
'retry': 60,
'disque_nodes': os.environ['TYND_DISQUE_NODES'].split(','),
'disque_auth': os.environ['TYND_DISQUE_AUTH']
}
disque_auth
~~~~~~~~~~~
Optional Disque password for servers that require authentication.
cpu_affinity
~~~~~~~~~~~~

3
docs/index.rst
View File

@@ -20,7 +20,7 @@ Features
- Django Admin integration
- PaaS compatible with multiple instances
- Multi cluster monitor
- Redis broker
- Redis or Disque broker
- Python 2 and 3
@@ -33,6 +33,7 @@ Contents:
Installation <install>
Configuration <configure>
Brokers <brokers>
Tasks <tasks>
Schedules <schedules>
Cluster <cluster>

12
docs/install.rst
View File

@@ -19,7 +19,8 @@ Installation
$ python manage.py migrate
- Make sure you have a `Redis <http://redis.io/>`__ server running
somewhere and know how to connect to it.
somewhere and know how to connect to it or configure one of the alternative :doc:`brokers`.
Requirements
------------
@@ -47,10 +48,6 @@ Django Q is tested for Python 2.7 and 3.4
This feature-filled fork of Erik Rose's blessings project provides the terminal layout of the monitor.
- `Redis server <http://redis.io/>`__
Django Q uses Redis as a centralized hub between your Django instances and your Q clusters.
Optional
~~~~~~~~
@@ -65,3 +62,8 @@ Optional
$ pip install hiredis
- `Redis <http://redis.io/>`__ server is the default broker for Django Q. It provides the best performance and does not require Django's cache framework for monitoring.
- `Disque <https://github.com/antirez/disque>`__ server is based on Redis by the same author, but focuses on reliable queues. Currently in Alpha, but highly recommended. You can either build it from source or use it on Heroku through the `Tynd <https://disque.tynd.co/>`__ beta.