mirror of
https://github.com/django-q2/django-q2.git
synced 2026-09-23 09:08:12 +08:00
74 lines
2.2 KiB
ReStructuredText
74 lines
2.2 KiB
ReStructuredText
Django Q
|
|
========
|
|
|
|
A multiprocessing task queue application for Django
|
|
---------------------------------------------------
|
|
|
|
Status
|
|
~~~~~~
|
|
|
|
Currently in the pre-alpha stage and Python 3 only (for now).
|
|
|
|
Architecture
|
|
~~~~~~~~~~~~
|
|
|
|
|Django Q schema| ### Signed Tasks Tasks are first pickled to Json and
|
|
then signed using Django's own signing module before being sent to a
|
|
Redis list. This ensures that task packages on the Redis server can only
|
|
be excuted and read by clusters and django servers who share the same
|
|
secret key.
|
|
|
|
Optionally, packages can be compressed before transport by setting
|
|
``Q_COMPRESSED = True``
|
|
|
|
Pusher
|
|
~~~~~~
|
|
|
|
The pusher process continously checks the Redis list for new task
|
|
packages and pushes them on the Task Queue.
|
|
|
|
Worker
|
|
~~~~~~
|
|
|
|
A worker process checks the package signing, unpacks the task, executes
|
|
it and saves the return value. Irrespective of the failure or success of
|
|
any of these steps, the package is then pushed onto the Result Queue.
|
|
|
|
By default Django Q spawns a worker for each detected CPU on the host
|
|
system. This can be overridden by setting ``Q_WORKERS = n``. With *n*
|
|
being the numbered of desired worker processes.
|
|
|
|
Monitor
|
|
~~~~~~~
|
|
|
|
The result monitor checks the Result Queue for processed packages and
|
|
saves both failed and succesful packages to the Django database.
|
|
|
|
By default only the last 100 succesful packages are kept in the
|
|
database. This can be increased or decreased at will by settings
|
|
``Q_SAVE_LIMIT = n``. With *n* being the desired number of records. Set
|
|
``Q_SAVE_LIMIT = 0`` to save all results to the database. Failed
|
|
packages are always saved.
|
|
|
|
Sentinel
|
|
~~~~~~~~
|
|
|
|
The sentinel spawns all process and then checks the health of all
|
|
workers, including the pusher and the monitor. Reincarnating processes
|
|
if any may fail. In case of a stop signal, the sentinel will halt the
|
|
pusher and instruct the workers and monitor to finish the remaining
|
|
items , before exiting.
|
|
|
|
Hooks
|
|
~~~~~
|
|
|
|
Packages can be assigned a hook function, upon completion of the package
|
|
this function will be called with the Task object as the first argument.
|
|
|
|
Todo
|
|
~~~~
|
|
|
|
I'll add to this README while I'm developing the various parts.
|
|
|
|
.. |Django Q schema| image:: http://i.imgur.com/wTIeg2T.png
|