fork/django-q2
1
0
Fork 0
mirror of https://github.com/django-q2/django-q2.git synced 2026-09-20 11:28:34 +08:00
Code Issues Packages Projects Releases Wiki Activity

Updated README for release

Browse Source
This commit is contained in:
Ilan Steemers
2015-06-28 17:12:48 +02:00
parent 218913e626
commit b469d0a341
2 changed files with 243 additions and 203 deletions
Download Patch File Download Diff File Expand all files Collapse all files

195
README.md
View File

@@ -1,46 +1,124 @@
# Django Q
##A multiprocessing task queue application for Django
##A multiprocessing task queue for Django
[![](https://travis-ci.org/Koed00/django-q.svg?branch=master)](https://travis-ci.org/Koed00/django-q)
### Status
In Alpha.
Everything should work, but the basic structure can still change.
Main focus is on creating more tests, better coverage and stability.
###Features
* Multiprocessing worker pool
* Encrypted and compressed task packages
* Scheduled tasks
* Result hooks
* Result and Failure database
* PaaS compatible with multiple pools
* Django Admin
### Requirements
* [Redis-py](https://github.com/andymccurdy/redis-py)
* [Django](https://www.djangoproject.com) > = 1.7
* [Django-picklefield](https://github.com/gintas/django-picklefield)
* [Arrow](https://github.com/crsmithdev/arrow)
* [Blessed](https://github.com/jquast/blessed)
Tested with: Python 2.7, 3.4. Django 1.7.8, 1.8.2\*
*\*Django Q is currently in Alpha and as such not safe for production, yet.*
### Architecture
![Django Q schema](http://i.imgur.com/wTIeg2T.png)
### Installation
### Usage
Schedule the asynchronous execution of a function by calling `async` from within your Django project.
* Install the latest version with pip: `pip install django-q`
* Add `django_q` to `INSTALLED_APPS` in your settings.py:
```python
INSTALLED_APPS = (
# other apps
'django_q',
)
```
* Run `python manage.py migrate` to create the database tables
* Make sure you have a [Redis](http://redis.io/) server running somewhere
###Configuration
All configuration settings are optional. e.g:
```python
# settings.py
Q_CLUSTER = {
'name': 'myproject',
'workers': 8,
'recycle': 500,
'compress': True,
'save_limit': 250,
'label': 'Django Q',
'redis': {
'host': '127.0.0.1',
'port': 6379,
'db': 0, }
}
```
* **name**
Used to differentiatie between projects using the same Redis server\*
*default*: 'default'
* **workers**
The number of workers to use in the cluster
*default*: CPU count of host
* **recycle**
The number of tasks a worker will process before respawing. Used to release resources.
*default*: 500
* **compress**
Compress task packages to Redis. Useful for large payloads.
*default*: False
* **save_limit**
Limits the amount of successful tasks saved to Django. Set to 0 for unlimited. Set to -1 for no success storage at all.
Failures are always saved.
*default*: 250
* **label**
The label used for the Django Admin page
*default*: 'Django Q'
* **redis**
Connection settings for Redis. Follows standard Redis-Py syntax.
*default*: localhost
\**Django Q uses your SECRET_KEY to encrypt task packages and prevent task crossover*
### Managment Commands
#### qcluster
Start a cluster with: `python manage.py qcluster`
####qmonitor
Monitor your clusters with `python manage.py qmonitor`
### Creating Tasks
#### Async
```python
async(func,*args,hook=None,**kwargs)
```
####Basic example
```python
from django_q import async
# math.copysign(2,-2)
async('math.copysign', 2, -2)
# also
from math import copysign
async(copysign, 2, -2)
```
#### Result example
```python
from django_q import async, result
# create the task
task_id = async('math.copysign', 2, -2)
async('math.copysign', 2, -2)
# or with import and storing the id
import math.copysign
task_id = async(copysign, 2, -2)
# get the result
task_result = result(task_id)
# result returns None if the task has not been executed yet
# so it makes more sense to use a hook:
# so in most cases you will want to use a hook:
async('math.modf', 2.5, hook='hooks.print_result')
@@ -48,61 +126,24 @@ async('math.modf', 2.5, hook='hooks.print_result')
def print_result(task):
print(task.result)
```
### Management commands
####Schedule
Schedules are regular Django models. You can manage them through the Admin page or in your code:
```python
from django_q import Schedule
from django.utils import timezone
#### `qcluster`
Start a cluster with `./manage.py qcluster`
Schedule.create(func='math.copysign',
hook='hooks.print_result',
args='2,-2',
schedule_type=Schedule.DAILY,
next_run=timezone.now())
```
![qcluster command](http://i.imgur.com/xccUxhW.png)
##Todo
* Write sphinx documentation
* Better tests and coverage
* Get out of Alpha
* Less dependencies?
####`qmonitor`
You can monitor basic information about all the connected clusters by running `./manage.py qmonitor`
![qmonitor command](http://i.imgur.com/5cm7hdP.png)
###Admin integration
Django Q registers itself with the admin page to show failed, successful and scheduled tasks.
From there task results can be read or deleted. If necessary, failed tasks can be reintroduced to the queue.
Schedules be created and their results monitored.
![q admin](http://i.imgur.com/FBlusZB.png)
###Schedules
Scheduled tasks are a django model and can be created through the admin interface or by creating a Schedule instance directly.
Like the Async Task, a Schedule can take an optional hook keyword and is used as a template to create the actual task package at the scheduled time.
If a result task is available in the database, it can be accessed through the Schedule instance's `result()` method.
### Signed Tasks
Tasks are first pickled to Json and then signed using Django's own signing module before being sent to a Redis list. This ensures that task packages on the Redis server can only be excuted and read by clusters and django servers who share the same secret key.
Optionally, packages can be compressed before transport by setting `Q_COMPRESSED = True `
### Pusher
The pusher process continuously checks the Redis list for new task packages and pushes them on the Task Queue.
### Worker
A worker process checks the package signing, unpacks the task, executes it and saves the return value. Irrespective of the failure or success of any of these steps, the package is then pushed onto the Result Queue.
By default Django Q spawns a worker for each detected CPU on the host system.
This can be overridden by setting `Q_WORKERS = n`. With *n* being the number of desired worker processes.
### Monitor
The result monitor checks the Result Queue for processed packages and saves both failed and successful packages to the Django database.
By default only the last 100 successful packages are kept in the database.
This can be increased or decreased at will by settings `Q_SAVE_LIMIT = n`. With *n* being the desired number of records.
Set `Q_SAVE_LIMIT = 0` to save all results to the database.
Failed packages are always saved.
### Sentinel
The sentinel spawns all process and then checks the health of all workers, including the pusher and the monitor. Reincarnating processes if any may fail.
In case of a stop signal, the sentinel will halt the pusher and instruct the workers and monitor to finish the remaining items , before exiting.
### Hooks
Packages can be assigned a hook function, upon completion of the package this function will be called with the Task object as the first argument.
### Todo
I'll add to this README while I'm developing the various parts.

251
README.rst
View File

@@ -1,60 +1,135 @@
Django Q
========
A multiprocessing task queue application for Django
---------------------------------------------------
A multiprocessing task queue for Django
---------------------------------------
|image0| ### Status In Alpha. Everything should work, but the basic
structure can still change. Main focus is on creating more tests, better
coverage and stability.
|image0|
Architecture
Features
~~~~~~~~
- Multiprocessing worker pool
- Encrypted and compressed task packages
- Scheduled tasks
- Result hooks
- Result and Failure database
- PaaS compatible with multiple pools
- Django Admin
Requirements
~~~~~~~~~~~~
.. figure:: http://i.imgur.com/wTIeg2T.png
:alt: Django Q schema
- `Redis-py <https://github.com/andymccurdy/redis-py>`__
- `Django <https://www.djangoproject.com>`__ > = 1.7
- `Django-picklefield <https://github.com/gintas/django-picklefield>`__
- `Arrow <https://github.com/crsmithdev/arrow>`__
- `Blessed <https://github.com/jquast/blessed>`__
Django Q schema
Usage
~~~~~
Tested with: Python 2.7, 3.4. Django 1.7.8, 1.8.2\*
Schedule the asynchronous execution of a function by calling ``async``
from within your Django project.
*\*Django Q is currently in Alpha and as such not safe for production,
yet.*
Installation
~~~~~~~~~~~~
- Install the latest version with pip: ``pip install django-q``
- Add ``django_q`` to ``INSTALLED_APPS`` in your settings.py:
.. code:: python
INSTALLED_APPS = (
# other apps
'django_q',
)
- Run ``python manage.py migrate`` to create the database tables
- Make sure you have a `Redis <http://redis.io/>`__ server running
somewhere
Configuration
~~~~~~~~~~~~~
All configuration settings are optional. e.g:
.. code:: python
# settings.py
Q_CLUSTER = {
'name': 'myproject',
'workers': 8,
'recycle': 500,
'compress': True,
'save_limit': 250,
'label': 'Django Q',
'redis': {
'host': '127.0.0.1',
'port': 6379,
'db': 0, }
}
- **name** Used to differentiatie between projects using the same Redis
server\* *default*: 'default'
- **workers** The number of workers to use in the cluster *default*:
CPU count of host
- **recycle** The number of tasks a worker will process before
respawing. Used to release resources. *default*: 500
- **compress** Compress task packages to Redis. Useful for large
payloads. *default*: False
- **save\_limit** Limits the amount of successful tasks saved to
Django. Set to 0 for unlimited. Set to -1 for no success storage at
all. Failures are always saved. *default*: 250
- **label** The label used for the Django Admin page *default*: 'Django
Q'
- **redis** Connection settings for Redis. Follows standard Redis-Py
syntax. *default*: localhost
\*\ *Django Q uses your SECRET\_KEY to encrypt task packages and prevent
task crossover*
Managment Commands
~~~~~~~~~~~~~~~~~~
qcluster
^^^^^^^^
Start a cluster with: ``python manage.py qcluster`` ####qmonitor Monitor
your clusters with ``python manage.py qmonitor``
Creating Tasks
~~~~~~~~~~~~~~
Async
^^^^^
.. code:: python
async(func,*args,hook=None,**kwargs)
Basic example
^^^^^^^^^^^^^
.. code:: python
from django_q import async
# math.copysign(2,-2)
async('math.copysign', 2, -2)
# also
from math import copysign
async(copysign, 2, -2)
Result example
^^^^^^^^^^^^^^
.. code:: python
from django_q import async, result
# create the task
task_id = async('math.copysign', 2, -2)
async('math.copysign', 2, -2)
# or with import and storing the id
import math.copysign
task_id = async(copysign, 2, -2)
# get the result
task_result = result(task_id)
# result returns None if the task has not been executed yet
# so it makes more sense to use a hook:
# so in most cases you will want to use a hook:
async('math.modf', 2.5, hook='hooks.print_result')
@@ -62,106 +137,30 @@ Result example
def print_result(task):
print(task.result)
Management commands
~~~~~~~~~~~~~~~~~~~
Schedule
^^^^^^^^
``qcluster``
^^^^^^^^^^^^
Schedules are regular Django models. You can manage them through the
Admin page or in your code:
Start a cluster with ``./manage.py qcluster``
.. code:: python
.. figure:: http://i.imgur.com/xccUxhW.png
:alt: qcluster command
from django_q import Schedule
from django.utils import timezone
qcluster command
``qmonitor``
^^^^^^^^^^^^
You can monitor basic information about all the connected clusters by
running ``./manage.py qmonitor``
.. figure:: http://i.imgur.com/5cm7hdP.png
:alt: qmonitor command
qmonitor command
Admin integration
~~~~~~~~~~~~~~~~~
Django Q registers itself with the admin page to show failed, successful
and scheduled tasks. From there task results can be read or deleted. If
necessary, failed tasks can be reintroduced to the queue. Schedules be
created and their results monitored. |q admin|
Schedules
~~~~~~~~~
Scheduled tasks are a django model and can be created through the admin
interface or by creating a Schedule instance directly. Like the Async
Task, a Schedule can take an optional hook keyword and is used as a
template to create the actual task package at the scheduled time. If a
result task is available in the database, it can be accessed through the
Schedule instance's ``result()`` method.
Signed Tasks
~~~~~~~~~~~~
Tasks are first pickled to Json and then signed using Django's own
signing module before being sent to a Redis list. This ensures that task
packages on the Redis server can only be excuted and read by clusters
and django servers who share the same secret key.
Optionally, packages can be compressed before transport by setting
``Q_COMPRESSED = True``
Pusher
~~~~~~
The pusher process continuously checks the Redis list for new task
packages and pushes them on the Task Queue.
Worker
~~~~~~
A worker process checks the package signing, unpacks the task, executes
it and saves the return value. Irrespective of the failure or success of
any of these steps, the package is then pushed onto the Result Queue.
By default Django Q spawns a worker for each detected CPU on the host
system. This can be overridden by setting ``Q_WORKERS = n``. With *n*
being the number of desired worker processes.
Monitor
~~~~~~~
The result monitor checks the Result Queue for processed packages and
saves both failed and successful packages to the Django database.
By default only the last 100 successful packages are kept in the
database. This can be increased or decreased at will by settings
``Q_SAVE_LIMIT = n``. With *n* being the desired number of records. Set
``Q_SAVE_LIMIT = 0`` to save all results to the database. Failed
packages are always saved.
Sentinel
~~~~~~~~
The sentinel spawns all process and then checks the health of all
workers, including the pusher and the monitor. Reincarnating processes
if any may fail. In case of a stop signal, the sentinel will halt the
pusher and instruct the workers and monitor to finish the remaining
items , before exiting.
Hooks
~~~~~
Packages can be assigned a hook function, upon completion of the package
this function will be called with the Task object as the first argument.
Schedule.create(func='math.copysign',
hook='hooks.print_result',
args='2,-2',
schedule_type=Schedule.DAILY,
next_run=timezone.now())
Todo
~~~~
----
I'll add to this README while I'm developing the various parts.
- Write sphinx documentation
- Better tests and coverage
- Get out of Alpha
- Less dependencies?
.. |image0| image:: https://travis-ci.org/Koed00/django-q.svg?branch=master
:target: https://travis-ci.org/Koed00/django-q
.. |q admin| image:: http://i.imgur.com/FBlusZB.png