Writing sphinx docs

docs/admin.rst

@@ -0,0 +1,2 @@
+Admin pages
+===========

docs/architecture.rst

@@ -1,65 +0,0 @@
-
-Architecture
-------------
-
-.. figure:: http://i.imgur.com/wTIeg2T.png
-   :alt: Django Q schema
-
-
-Signed Tasks
-""""""""""""
-
-Tasks are first pickled and then signed using Django's own
-signing module before being sent to a Redis list. This ensures that task
-packages on the Redis server can only be executed and read by clusters
-and django servers who share the same secret key.
-Optionally the packages can be compressed before transport
-
-Pusher
-""""""
-
-The pusher process continuously checks the Redis list for new task
-packages and pushes them on the Task Queue.
-
-Worker
-""""""
-
-A worker process checks the package signing, unpacks the task, executes
-it and saves the return value. Irrespective of the failure or success of
-any of these steps, the package is then pushed onto the Result Queue.
-
-
-Monitor
-"""""""
-
-The result monitor checks the Result Queue for processed packages and
-saves both failed and successful packages to the Django database.
-
-
-Sentinel
-""""""""
-
-The sentinel spawns all process and then checks the health of all
-workers, including the pusher and the monitor. Reincarnating processes
-if any may fail. In case of a stop signal, the sentinel will halt the
-pusher and instruct the workers and monitor to finish the remaining
-items , before exiting. see Stop procedure
-
-Timeouts
-""""""""
-Before each task execution the worker resets a timer on the sentinel and resets it again after execution.
-Meanwhile the the sentinel checks if the timers don't exceed the timeout amount, in which case it will terminate the worker and reincarnate a new one.
-
-Hooks
-"""""
-
-Packages can be assigned a hook function, upon completion of the package
-this function will be called with the Task object as the first argument.
-
-Stop procedure
-""""""""""""""
-
-When a stop signal is given, the sentinel exits the guard loop and instructs the pusher to stop pushing.
-Once this is confirmed, the sentinel pushes poison pills onto the task queue and will wait for all the workers to die.
-This ensure that the task is emptied before the workers exit. Afterwards the sentinel waits for the monitor to empty the result queue before the stop flow is complete.
-

docs/cluster.rst

@@ -0,0 +1,103 @@
+
+Cluster
+=======
+Django Q uses Python's multiprocessing module to manage a pool of workers that will handle your tasks.
+Start your cluster using Django's `manage.py` command::
+
+    $ python manage.py qcluster
+
+
+You should  see something like this::
+
+    10:57:40 [Q] INFO Q Cluster-31781 starting.
+    10:57:40 [Q] INFO Process-1:1 ready for work at 31784
+    10:57:40 [Q] INFO Process-1:2 ready for work at 31785
+    10:57:40 [Q] INFO Process-1:3 ready for work at 31786
+    10:57:40 [Q] INFO Process-1:4 ready for work at 31787
+    10:57:40 [Q] INFO Process-1:5 ready for work at 31788
+    10:57:40 [Q] INFO Process-1:6 ready for work at 31789
+    10:57:40 [Q] INFO Process-1:7 ready for work at 31790
+    10:57:40 [Q] INFO Process-1:8 ready for work at 31791
+    10:57:40 [Q] INFO Process-1:9 monitoring at 31792
+    10:57:40 [Q] INFO Process-1 guarding cluster at 31783
+    10:57:40 [Q] INFO Process-1:10 pushing tasks at 31793
+    10:57:40 [Q] INFO Q Cluster-31781 running.
+
+
+Stopping the cluster with ctrl-c or either the `SIGTERM` and `SIGKILL` signals, will initiate the :ref:`stop_procedure`.
+
+Architecture
+------------
+
+.. image:: cluster.png
+   :alt: Django Q schema
+
+
+Signed Tasks
+""""""""""""
+
+Tasks are first pickled and then signed using Django's own
+signing module before being sent to a Redis list. This ensures that task
+packages on the Redis server can only be executed and read by clusters
+and django servers who share the same secret key.
+Optionally the packages can be compressed before transport
+
+Pusher
+""""""
+
+The pusher process continuously checks the Redis list for new task
+packages and pushes them on the Task Queue.
+
+Worker
+""""""
+
+A worker process checks the package signing, unpacks the task, executes
+it and saves the return value. Irrespective of the failure or success of
+any of these steps, the package is then pushed onto the Result Queue.
+
+
+Monitor
+"""""""
+
+The result monitor checks the Result Queue for processed packages and
+saves both failed and successful packages to the Django database.
+
+
+Sentinel
+""""""""
+
+The sentinel spawns all process and then checks the health of all
+workers, including the pusher and the monitor. Reincarnating processes
+if any may fail. In case of a stop signal, the sentinel will halt the
+pusher and instruct the workers and monitor to finish the remaining
+items. See :ref:`stop_procedure`
+
+Timeouts
+""""""""
+Before each task execution the worker resets a timer on the sentinel and resets it again after execution.
+Meanwhile the the sentinel checks if the timers don't exceed the timeout amount, in which case it will terminate the worker and reincarnate a new one.
+
+Scheduler
+"""""""""
+Once a minute the scheduler checks for any scheduled task that should be starting.
+It creates and queues a new task from the schedule and sets the next run date.
+
+.. _stop_procedure:
+
+Stop procedure
+""""""""""""""
+
+When a stop signal is given, the sentinel exits the guard loop and instructs the pusher to stop pushing.
+Once this is confirmed, the sentinel pushes poison pills onto the task queue and will wait for all the workers to die.
+This ensure that the queue is emptied before the workers exit.
+Afterwards the sentinel waits for the monitor to empty the result queue before the stop procedure is complete.
+
+- Send stop event to pusher
+- Wait for pusher to exit
+- Put poison pills in the Task Queue
+- Wait for all the workers to clear the queue and stop
+- Put a poison pill on the Result Queue
+- Wait for monitor to stop
+
+.. :warning::
+    If you force the cluster to terminate before the stop procedure has completed, you can lose tasks and their results.

docs/conf.py

@@ -127,6 +127,7 @@ html_theme_options = {
     'github_user': 'Koed00',
     'github_repo': 'django-q',
     'gittip_user': 'koed00',
+    'github_banner': True,
     'travis_button': True,
 }
 html_sidebars = {
@@ -160,7 +161,7 @@ html_theme_path = [alabaster.get_path()]
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+#html_static_path = ['_static']
 
 # Add any extra paths that contain custom files (such as robots.txt or
 # .htaccess) here, relative to this directory. These files are copied
@@ -298,7 +299,7 @@ texinfo_documents = [
 #texinfo_appendices = []
 
 # If false, no module index is generated.
-#texinfo_domain_indices = True
+texinfo_domain_indices = True
 
 # How to display URL addresses: 'footnote', 'no', or 'inline'.
 #texinfo_show_urls = 'footnote'

docs/index.rst

@@ -9,17 +9,15 @@ Welcome to Django Q's documentation!
 Contents:
 
 .. toctree::
-   :maxdepth: 2
+   :maxdepth: 1
 
-  Introduction
   Installation <install>
-  Usage <usage>
+  Tasks <tasks>
-  Management <management>
+  Schedules <schedules>
+  Cluster <cluster>
+  Monitor <monitor>
   Admin <admin>
-  Architecture <architecture>
 
-
+   * :ref:`genindex`
-
-* :ref:`genindex`
 * :ref:`search`
 

docs/install.rst

@@ -0,0 +1,111 @@
+Installation
+============
+
+-  Install the latest version with pip::
+
+    $ pip install django-q
+
+
+-  Add :mod:`django_q` to your :const:`INSTALLED_APPS` in your projects :file:`settings.py`::
+
+       INSTALLED_APPS = (
+           # other apps
+           'django_q',
+       )
+
+-  Run Django migrations to create the database tables::
+
+    $ python manage.py migrate
+
+-  Make sure you have a `Redis <http://redis.io/>`__ server running
+   somewhere
+
+Configuration
+-------------
+
+Configuration is handled via the :const:`Q_ClUSTER` dictionary in your :file:`settings.py`
+
+.. code:: python
+
+    # settings.py example
+    Q_CLUSTER = {
+        'name': 'myproject',
+        'workers': 8,
+        'recycle': 500,
+        'timeout': 60,
+        'compress': True,
+        'save_limit': 250,
+        'label': 'Django Q',
+        'redis': {
+            'host': '127.0.0.1',
+            'port': 6379,
+            'db': 0, }
+    }
+
+
+
+name
+~~~~
+
+Used to differentiate between projects using the same Redis server. Defaults to ``'default'``.
+This can be useful if you have several projects using the same Redis server.
+
+.. note::
+    Tasks are encrypted. When a worker encounters a task it can not decrypt, it will be discarded
+
+workers
+~~~~~~~
+
+The number of workers to use in the cluster. Defaults to CPU count of the current host, but can be set to a custom number.
+
+recycle
+~~~~~~~
+
+The number of tasks a worker will process before respawning. Useful to release resources on a regular basis. Defaults to ``500``.
+
+timeout
+~~~~~~~
+
+The number of seconds a worker is allowed to spend on a task before it's terminated. Defaults to ``None``, meaning it will never time out.
+Set this to something that makes sense for your use
+
+compress
+~~~~~~~~
+
+Compress task packages to Redis. Useful for large payloads, but can add overhead when used with many small packages.
+Defaults to ``False``
+
+save_limit
+~~~~~~~~~~
+
+Limits the amount of successful tasks saved to Django.
+Set to ``0`` for unlimited. Set to ``-1`` for no success storage at all.
+Failures are always saved. Defaults to ``250``
+
+label
+~~~~~
+
+The label used for the Django Admin page. Defaults to ``'Django Q'``
+
+redis
+~~~~~
+
+Connection settings for Redis. Defaults::
+
+    redis: {
+        'host': 'localhost',
+        'port': 6379,
+        'db': 0,
+        'password': None,
+        'socket_timeout': None,
+        'charset': 'utf-8',
+        'errors': 'strict',
+        'unix_socket_path': None
+    }
+
+For more information on these settings please refer to the `Redis-py <https://github.com/andymccurdy/redis-py>`__ documentation
+
+
+.. note::
+    Django Q uses your :const:`SECRET_KEY` to encrypt task packages and prevent task crossover. So make sure you have it set up in your Django settings.
+

docs/management.rst

@@ -1,20 +0,0 @@
-Management commands
-===================
-
-$project offers the following Django management commands
-
-
-qcluster
---------
-
-.. code:: bash
-
-    python management.py qcluster
-
-
-qmonitor
---------
-
-.. code:: bash
-
-    python management.py qmonitor

docs/monitor.rst

@@ -0,0 +1,3 @@
+Monitor
+=======
+

docs/schedules.rst

@@ -0,0 +1,2 @@
+Schedules
+=========

docs/tasks.rst

@@ -1,7 +1,7 @@
-Usage
+Tasks
 =====
 
-Use  :py:func:`async` from your code to quickly offload tasks:
+Use  :py:func:`async` from your code to quickly offload tasks to the py:module:`cluster`:
 
 .. code:: python