docs: moves architecture to its own main section

Also updated the architecture text a little

docs/architecture.rst

@@ -0,0 +1,91 @@
+Architecture
+------------
+
+.. image:: _static/cluster.png
+   :alt: Django Q schema
+
+
+Signed Tasks
+""""""""""""
+
+Tasks are first pickled and then signed using Django's own :mod:`django.core.signing` module using the ``SECRET_KEY`` and cluster name as salt, before being sent to a message broker. This ensures that task
+packages on the broker can only be executed and read by clusters
+and django servers who share the same secret key and cluster name.
+If a package fails to unpack, it will be marked failed with the broker and discarded.
+Optionally the packages can be compressed before transport.
+
+Broker
+""""""
+
+The broker collects task packages from the django instances and queues them for pick up by a cluster.
+If the broker supports message receipts, it will keep a copy of the tasks around until a cluster acknowledges the processing of the task.
+Otherwise it is put back in the queue after a timeout period. This ensure at-least-once delivery.
+Note that even if the task errors when processed by the cluster, this is considered a successful delivery.
+Most failed deliveries will be the result of a worker or the cluster crashing before the task was saved.
+
+Pusher
+""""""
+
+The pusher process continuously checks the broker for new task
+packages. It checks the signing and unpacks the task to the internal Task Queue.
+The amount of tasks in the Task Queue can be configured to control memory usage and minimize data loss in case of a failure.
+
+Worker
+""""""
+
+A worker process pulls a task of the Task Queue and it sets a shared countdown timer with :ref:`sentinel` indicating it is about to start work.
+The worker then tries to execute the task and afterwards the timer is reset and any results (including errors) are saved to the package.
+Irrespective of the failure or success of any of these steps, the package is then pushed onto the Result Queue.
+
+Monitor
+"""""""
+
+The result monitor checks the Result Queue for processed packages and
+saves both failed and successful packages to the Django database or cache backend.
+If the broker supports it, a delivery receipt is sent.
+In case the task was part of a chain, the next task is queued.
+
+.. _sentinel:
+
+Sentinel
+""""""""
+
+The sentinel spawns all process and then checks the health of all
+workers, including the pusher and the monitor. This includes checking timers on each worker for timeouts.
+In case of a sudden death or timeout, it will reincarnate the failing processes. When a stop signal is received, the sentinel will halt the
+pusher and instruct the workers and monitor to finish the remaining items. See :ref:`stop_procedure`
+
+Timeouts
+""""""""
+Before each task execution the worker sets a countdown timer on the sentinel and resets it again after execution.
+Meanwhile the sentinel checks if the timers don't reach zero, in which case it will terminate the worker and reincarnate a new one.
+
+Scheduler
+"""""""""
+Twice a minute the scheduler checks for any scheduled tasks that should be starting.
+
+- Creates a task from the schedule
+- Subtracts 1 from :attr:`django_q.Schedule.repeats`
+- Sets the next run time if there are repeats left or if it has a negative value.
+
+.. _stop_procedure:
+
+Stop procedure
+""""""""""""""
+
+When a stop signal is received, the sentinel exits the guard loop and instructs the pusher to stop pushing.
+Once this is confirmed, the sentinel pushes poison pills onto the task queue and will wait for all the workers to exit.
+This ensures that the task queue is emptied before the workers exit.
+Afterwards the sentinel waits for the monitor to empty the result queue and the stop procedure is complete.
+
+- Send stop event to pusher
+- Wait for pusher to exit
+- Put poison pills in the Task Queue
+- Wait for all the workers to clear the queue and stop
+- Put a poison pill on the Result Queue
+- Wait for monitor to process remaining results and exit
+- Signal that we have stopped
+
+.. warning::
+    If you force the cluster to terminate before the stop procedure has completed, you can lose tasks or results still being held in memory.
+    You can manage the amount of tasks in a clusters memory by setting the :ref:`queue_limit`.

docs/cluster.rst

@@ -82,87 +82,6 @@ An example :file:`circus.ini` ::
 Note that we only start one process. It is not a good idea to run multiple instances of the cluster in the same environment since this does nothing to increase performance and in all likelihood will diminish it.
 Control your cluster using the ``workers``, ``recycle`` and ``timeout`` settings in your :doc:`configure`
 
-Architecture
-------------
-
-.. image:: _static/cluster.png
-   :alt: Django Q schema
-
-
-Signed Tasks
-""""""""""""
-
-Tasks are first pickled and then signed using Django's own :mod:`django.core.signing` module using the ``SECRET_KEY`` and cluster name as salt, before being sent to a message broker. This ensures that task
-packages on the broker can only be executed and read by clusters
-and django servers who share the same secret key and cluster name.
-If a package fails to unpack, it will be marked failed with the broker and discarded.
-Optionally the packages can be compressed before transport.
-
-Pusher
-""""""
-
-The pusher process continuously checks the broker for new task
-packages. It checks the signing and unpacks the task to the Task Queue.
-
-Worker
-""""""
-
-A worker process pulls a task of the Task Queue and it sets a shared countdown timer with :ref:`sentinel` indicating it is about to start work.
-The worker then tries to execute the task and afterwards the timer is reset and any results (including errors) are saved to the package.
-Irrespective of the failure or success of any of these steps, the package is then pushed onto the Result Queue.
-
-Monitor
-"""""""
-
-The result monitor checks the Result Queue for processed packages and
-saves both failed and successful packages to the Django database.
-If the broker supports it, a delivery receipt is sent.
-
-.. _sentinel:
-
-Sentinel
-""""""""
-
-The sentinel spawns all process and then checks the health of all
-workers, including the pusher and the monitor. This includes checking timers on each worker for timeouts.
-In case of a sudden death or timeout, it will reincarnate the failing processes. When a stop signal is received, the sentinel will halt the
-pusher and instruct the workers and monitor to finish the remaining items. See :ref:`stop_procedure`
-
-Timeouts
-""""""""
-Before each task execution the worker sets a countdown timer on the sentinel and resets it again after execution.
-Meanwhile the sentinel checks if the timers don't reach zero, in which case it will terminate the worker and reincarnate a new one.
-
-Scheduler
-"""""""""
-Twice a minute the scheduler checks for any scheduled tasks that should be starting.
-
-- Creates a task from the schedule
-- Subtracts 1 from :attr:`django_q.Schedule.repeats`
-- Sets the next run time if there are repeats left or if it has a negative value.
-
-.. _stop_procedure:
-
-Stop procedure
-""""""""""""""
-
-When a stop signal is received, the sentinel exits the guard loop and instructs the pusher to stop pushing.
-Once this is confirmed, the sentinel pushes poison pills onto the task queue and will wait for all the workers to exit.
-This ensures that the task queue is emptied before the workers exit.
-Afterwards the sentinel waits for the monitor to empty the result queue and the stop procedure is complete.
-
-- Send stop event to pusher
-- Wait for pusher to exit
-- Put poison pills in the Task Queue
-- Wait for all the workers to clear the queue and stop
-- Put a poison pill on the Result Queue
-- Wait for monitor to process remaining results and exit
-- Signal that we have stopped
-
-.. warning::
-    If you force the cluster to terminate before the stop procedure has completed, you can lose tasks or results still being held in memory.
-    You can manage the amount of tasks in a clusters memory by setting the :ref:`queue_limit`.
-
 Reference
 ---------
 

docs/index.rst

@@ -11,7 +11,7 @@ Django Q is a native Django task queue, scheduler and worker application using P
 Features
 --------
 
--  Multiprocessing worker pool
+-  Multiprocessing worker poole7u-CG
 -  Asynchronous tasks
 -  Scheduled and repeated tasks
 -  Encrypted and compressed packages
@@ -39,6 +39,7 @@ Contents:
     Cluster <cluster>
     Monitor <monitor>
     Admin <admin>
+    Architecture <architecture>
     Examples <examples>
 
 * :ref:`genindex`