mirror of
https://github.com/django-q2/django-q2.git
synced 2026-09-23 00:18:11 +08:00
Writing sphinx docs
This commit is contained in:
Ilan Steemers
@@ -0,0 +1,2 @@
|
||||
Admin pages
|
||||
===========
|
||||
@@ -1,65 +0,0 @@
|
||||
|
||||
Architecture
|
||||
------------
|
||||
|
||||
.. figure:: http://i.imgur.com/wTIeg2T.png
|
||||
:alt: Django Q schema
|
||||
|
||||
|
||||
Signed Tasks
|
||||
""""""""""""
|
||||
|
||||
Tasks are first pickled and then signed using Django's own
|
||||
signing module before being sent to a Redis list. This ensures that task
|
||||
packages on the Redis server can only be executed and read by clusters
|
||||
and django servers who share the same secret key.
|
||||
Optionally the packages can be compressed before transport
|
||||
|
||||
Pusher
|
||||
""""""
|
||||
|
||||
The pusher process continuously checks the Redis list for new task
|
||||
packages and pushes them on the Task Queue.
|
||||
|
||||
Worker
|
||||
""""""
|
||||
|
||||
A worker process checks the package signing, unpacks the task, executes
|
||||
it and saves the return value. Irrespective of the failure or success of
|
||||
any of these steps, the package is then pushed onto the Result Queue.
|
||||
|
||||
|
||||
Monitor
|
||||
"""""""
|
||||
|
||||
The result monitor checks the Result Queue for processed packages and
|
||||
saves both failed and successful packages to the Django database.
|
||||
|
||||
|
||||
Sentinel
|
||||
""""""""
|
||||
|
||||
The sentinel spawns all process and then checks the health of all
|
||||
workers, including the pusher and the monitor. Reincarnating processes
|
||||
if any may fail. In case of a stop signal, the sentinel will halt the
|
||||
pusher and instruct the workers and monitor to finish the remaining
|
||||
items , before exiting. see Stop procedure
|
||||
|
||||
Timeouts
|
||||
""""""""
|
||||
Before each task execution the worker resets a timer on the sentinel and resets it again after execution.
|
||||
Meanwhile the the sentinel checks if the timers don't exceed the timeout amount, in which case it will terminate the worker and reincarnate a new one.
|
||||
|
||||
Hooks
|
||||
"""""
|
||||
|
||||
Packages can be assigned a hook function, upon completion of the package
|
||||
this function will be called with the Task object as the first argument.
|
||||
|
||||
Stop procedure
|
||||
""""""""""""""
|
||||
|
||||
When a stop signal is given, the sentinel exits the guard loop and instructs the pusher to stop pushing.
|
||||
Once this is confirmed, the sentinel pushes poison pills onto the task queue and will wait for all the workers to die.
|
||||
This ensure that the task is emptied before the workers exit. Afterwards the sentinel waits for the monitor to empty the result queue before the stop flow is complete.
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 45 KiB |
@@ -0,0 +1,103 @@
|
||||
|
||||
Cluster
|
||||
=======
|
||||
Django Q uses Python's multiprocessing module to manage a pool of workers that will handle your tasks.
|
||||
Start your cluster using Django's `manage.py` command::
|
||||
|
||||
$ python manage.py qcluster
|
||||
|
||||
|
||||
You should see something like this::
|
||||
|
||||
10:57:40 [Q] INFO Q Cluster-31781 starting.
|
||||
10:57:40 [Q] INFO Process-1:1 ready for work at 31784
|
||||
10:57:40 [Q] INFO Process-1:2 ready for work at 31785
|
||||
10:57:40 [Q] INFO Process-1:3 ready for work at 31786
|
||||
10:57:40 [Q] INFO Process-1:4 ready for work at 31787
|
||||
10:57:40 [Q] INFO Process-1:5 ready for work at 31788
|
||||
10:57:40 [Q] INFO Process-1:6 ready for work at 31789
|
||||
10:57:40 [Q] INFO Process-1:7 ready for work at 31790
|
||||
10:57:40 [Q] INFO Process-1:8 ready for work at 31791
|
||||
10:57:40 [Q] INFO Process-1:9 monitoring at 31792
|
||||
10:57:40 [Q] INFO Process-1 guarding cluster at 31783
|
||||
10:57:40 [Q] INFO Process-1:10 pushing tasks at 31793
|
||||
10:57:40 [Q] INFO Q Cluster-31781 running.
|
||||
|
||||
|
||||
Stopping the cluster with ctrl-c or either the `SIGTERM` and `SIGKILL` signals, will initiate the :ref:`stop_procedure`.
|
||||
|
||||
Architecture
|
||||
------------
|
||||
|
||||
.. image:: cluster.png
|
||||
:alt: Django Q schema
|
||||
|
||||
|
||||
Signed Tasks
|
||||
""""""""""""
|
||||
|
||||
Tasks are first pickled and then signed using Django's own
|
||||
signing module before being sent to a Redis list. This ensures that task
|
||||
packages on the Redis server can only be executed and read by clusters
|
||||
and django servers who share the same secret key.
|
||||
Optionally the packages can be compressed before transport
|
||||
|
||||
Pusher
|
||||
""""""
|
||||
|
||||
The pusher process continuously checks the Redis list for new task
|
||||
packages and pushes them on the Task Queue.
|
||||
|
||||
Worker
|
||||
""""""
|
||||
|
||||
A worker process checks the package signing, unpacks the task, executes
|
||||
it and saves the return value. Irrespective of the failure or success of
|
||||
any of these steps, the package is then pushed onto the Result Queue.
|
||||
|
||||
|
||||
Monitor
|
||||
"""""""
|
||||
|
||||
The result monitor checks the Result Queue for processed packages and
|
||||
saves both failed and successful packages to the Django database.
|
||||
|
||||
|
||||
Sentinel
|
||||
""""""""
|
||||
|
||||
The sentinel spawns all process and then checks the health of all
|
||||
workers, including the pusher and the monitor. Reincarnating processes
|
||||
if any may fail. In case of a stop signal, the sentinel will halt the
|
||||
pusher and instruct the workers and monitor to finish the remaining
|
||||
items. See :ref:`stop_procedure`
|
||||
|
||||
Timeouts
|
||||
""""""""
|
||||
Before each task execution the worker resets a timer on the sentinel and resets it again after execution.
|
||||
Meanwhile the the sentinel checks if the timers don't exceed the timeout amount, in which case it will terminate the worker and reincarnate a new one.
|
||||
|
||||
Scheduler
|
||||
"""""""""
|
||||
Once a minute the scheduler checks for any scheduled task that should be starting.
|
||||
It creates and queues a new task from the schedule and sets the next run date.
|
||||
|
||||
.. _stop_procedure:
|
||||
|
||||
Stop procedure
|
||||
""""""""""""""
|
||||
|
||||
When a stop signal is given, the sentinel exits the guard loop and instructs the pusher to stop pushing.
|
||||
Once this is confirmed, the sentinel pushes poison pills onto the task queue and will wait for all the workers to die.
|
||||
This ensure that the queue is emptied before the workers exit.
|
||||
Afterwards the sentinel waits for the monitor to empty the result queue before the stop procedure is complete.
|
||||
|
||||
- Send stop event to pusher
|
||||
- Wait for pusher to exit
|
||||
- Put poison pills in the Task Queue
|
||||
- Wait for all the workers to clear the queue and stop
|
||||
- Put a poison pill on the Result Queue
|
||||
- Wait for monitor to stop
|
||||
|
||||
.. :warning::
|
||||
If you force the cluster to terminate before the stop procedure has completed, you can lose tasks and their results.
|
||||
+3
-2
@@ -127,6 +127,7 @@ html_theme_options = {
|
||||
'github_user': 'Koed00',
|
||||
'github_repo': 'django-q',
|
||||
'gittip_user': 'koed00',
|
||||
'github_banner': True,
|
||||
'travis_button': True,
|
||||
}
|
||||
html_sidebars = {
|
||||
@@ -160,7 +161,7 @@ html_theme_path = [alabaster.get_path()]
|
||||
# Add any paths that contain custom static files (such as style sheets) here,
|
||||
# relative to this directory. They are copied after the builtin static files,
|
||||
# so a file named "default.css" will overwrite the builtin "default.css".
|
||||
html_static_path = ['_static']
|
||||
#html_static_path = ['_static']
|
||||
|
||||
# Add any extra paths that contain custom files (such as robots.txt or
|
||||
# .htaccess) here, relative to this directory. These files are copied
|
||||
@@ -298,7 +299,7 @@ texinfo_documents = [
|
||||
#texinfo_appendices = []
|
||||
|
||||
# If false, no module index is generated.
|
||||
#texinfo_domain_indices = True
|
||||
texinfo_domain_indices = True
|
||||
|
||||
# How to display URL addresses: 'footnote', 'no', or 'inline'.
|
||||
#texinfo_show_urls = 'footnote'
|
||||
|
||||
+6
-8
@@ -9,17 +9,15 @@ Welcome to Django Q's documentation!
|
||||
Contents:
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
:maxdepth: 1
|
||||
|
||||
Introduction
|
||||
Installation <install>
|
||||
Usage <usage>
|
||||
Management <management>
|
||||
Tasks <tasks>
|
||||
Schedules <schedules>
|
||||
Cluster <cluster>
|
||||
Monitor <monitor>
|
||||
Admin <admin>
|
||||
Architecture <architecture>
|
||||
|
||||
|
||||
|
||||
* :ref:`genindex`
|
||||
* :ref:`genindex`
|
||||
* :ref:`search`
|
||||
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
Installation
|
||||
============
|
||||
|
||||
- Install the latest version with pip::
|
||||
|
||||
$ pip install django-q
|
||||
|
||||
|
||||
- Add :mod:`django_q` to your :const:`INSTALLED_APPS` in your projects :file:`settings.py`::
|
||||
|
||||
INSTALLED_APPS = (
|
||||
# other apps
|
||||
'django_q',
|
||||
)
|
||||
|
||||
- Run Django migrations to create the database tables::
|
||||
|
||||
$ python manage.py migrate
|
||||
|
||||
- Make sure you have a `Redis <http://redis.io/>`__ server running
|
||||
somewhere
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
Configuration is handled via the :const:`Q_ClUSTER` dictionary in your :file:`settings.py`
|
||||
|
||||
.. code:: python
|
||||
|
||||
# settings.py example
|
||||
Q_CLUSTER = {
|
||||
'name': 'myproject',
|
||||
'workers': 8,
|
||||
'recycle': 500,
|
||||
'timeout': 60,
|
||||
'compress': True,
|
||||
'save_limit': 250,
|
||||
'label': 'Django Q',
|
||||
'redis': {
|
||||
'host': '127.0.0.1',
|
||||
'port': 6379,
|
||||
'db': 0, }
|
||||
}
|
||||
|
||||
|
||||
|
||||
name
|
||||
~~~~
|
||||
|
||||
Used to differentiate between projects using the same Redis server. Defaults to ``'default'``.
|
||||
This can be useful if you have several projects using the same Redis server.
|
||||
|
||||
.. note::
|
||||
Tasks are encrypted. When a worker encounters a task it can not decrypt, it will be discarded
|
||||
|
||||
workers
|
||||
~~~~~~~
|
||||
|
||||
The number of workers to use in the cluster. Defaults to CPU count of the current host, but can be set to a custom number.
|
||||
|
||||
recycle
|
||||
~~~~~~~
|
||||
|
||||
The number of tasks a worker will process before respawning. Useful to release resources on a regular basis. Defaults to ``500``.
|
||||
|
||||
timeout
|
||||
~~~~~~~
|
||||
|
||||
The number of seconds a worker is allowed to spend on a task before it's terminated. Defaults to ``None``, meaning it will never time out.
|
||||
Set this to something that makes sense for your use
|
||||
|
||||
compress
|
||||
~~~~~~~~
|
||||
|
||||
Compress task packages to Redis. Useful for large payloads, but can add overhead when used with many small packages.
|
||||
Defaults to ``False``
|
||||
|
||||
save_limit
|
||||
~~~~~~~~~~
|
||||
|
||||
Limits the amount of successful tasks saved to Django.
|
||||
Set to ``0`` for unlimited. Set to ``-1`` for no success storage at all.
|
||||
Failures are always saved. Defaults to ``250``
|
||||
|
||||
label
|
||||
~~~~~
|
||||
|
||||
The label used for the Django Admin page. Defaults to ``'Django Q'``
|
||||
|
||||
redis
|
||||
~~~~~
|
||||
|
||||
Connection settings for Redis. Defaults::
|
||||
|
||||
redis: {
|
||||
'host': 'localhost',
|
||||
'port': 6379,
|
||||
'db': 0,
|
||||
'password': None,
|
||||
'socket_timeout': None,
|
||||
'charset': 'utf-8',
|
||||
'errors': 'strict',
|
||||
'unix_socket_path': None
|
||||
}
|
||||
|
||||
For more information on these settings please refer to the `Redis-py <https://github.com/andymccurdy/redis-py>`__ documentation
|
||||
|
||||
|
||||
.. note::
|
||||
Django Q uses your :const:`SECRET_KEY` to encrypt task packages and prevent task crossover. So make sure you have it set up in your Django settings.
|
||||
|
||||
@@ -1,20 +0,0 @@
|
||||
Management commands
|
||||
===================
|
||||
|
||||
$project offers the following Django management commands
|
||||
|
||||
|
||||
qcluster
|
||||
--------
|
||||
|
||||
.. code:: bash
|
||||
|
||||
python management.py qcluster
|
||||
|
||||
|
||||
qmonitor
|
||||
--------
|
||||
|
||||
.. code:: bash
|
||||
|
||||
python management.py qmonitor
|
||||
@@ -0,0 +1,3 @@
|
||||
Monitor
|
||||
=======
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
Schedules
|
||||
=========
|
||||
@@ -1,7 +1,7 @@
|
||||
Usage
|
||||
Tasks
|
||||
=====
|
||||
|
||||
Use :py:func:`async` from your code to quickly offload tasks:
|
||||
Use :py:func:`async` from your code to quickly offload tasks to the py:module:`cluster`:
|
||||
|
||||
.. code:: python
|
||||
|
||||
Reference in New Issue
Block a user