diff --git a/docs/admin.rst b/docs/admin.rst new file mode 100644 index 0000000..1a5c821 --- /dev/null +++ b/docs/admin.rst @@ -0,0 +1,2 @@ +Admin pages +=========== diff --git a/docs/architecture.rst b/docs/architecture.rst deleted file mode 100644 index 0e15ac1..0000000 --- a/docs/architecture.rst +++ /dev/null @@ -1,65 +0,0 @@ - -Architecture ------------- - -.. figure:: http://i.imgur.com/wTIeg2T.png - :alt: Django Q schema - - -Signed Tasks -"""""""""""" - -Tasks are first pickled and then signed using Django's own -signing module before being sent to a Redis list. This ensures that task -packages on the Redis server can only be executed and read by clusters -and django servers who share the same secret key. -Optionally the packages can be compressed before transport - -Pusher -"""""" - -The pusher process continuously checks the Redis list for new task -packages and pushes them on the Task Queue. - -Worker -"""""" - -A worker process checks the package signing, unpacks the task, executes -it and saves the return value. Irrespective of the failure or success of -any of these steps, the package is then pushed onto the Result Queue. - - -Monitor -""""""" - -The result monitor checks the Result Queue for processed packages and -saves both failed and successful packages to the Django database. - - -Sentinel -"""""""" - -The sentinel spawns all process and then checks the health of all -workers, including the pusher and the monitor. Reincarnating processes -if any may fail. In case of a stop signal, the sentinel will halt the -pusher and instruct the workers and monitor to finish the remaining -items , before exiting. see Stop procedure - -Timeouts -"""""""" -Before each task execution the worker resets a timer on the sentinel and resets it again after execution. -Meanwhile the the sentinel checks if the timers don't exceed the timeout amount, in which case it will terminate the worker and reincarnate a new one. - -Hooks -""""" - -Packages can be assigned a hook function, upon completion of the package -this function will be called with the Task object as the first argument. - -Stop procedure -"""""""""""""" - -When a stop signal is given, the sentinel exits the guard loop and instructs the pusher to stop pushing. -Once this is confirmed, the sentinel pushes poison pills onto the task queue and will wait for all the workers to die. -This ensure that the task is emptied before the workers exit. Afterwards the sentinel waits for the monitor to empty the result queue before the stop flow is complete. - diff --git a/docs/cluster.png b/docs/cluster.png new file mode 100644 index 0000000..69fb38f Binary files /dev/null and b/docs/cluster.png differ diff --git a/docs/cluster.rst b/docs/cluster.rst new file mode 100644 index 0000000..463298c --- /dev/null +++ b/docs/cluster.rst @@ -0,0 +1,103 @@ + +Cluster +======= +Django Q uses Python's multiprocessing module to manage a pool of workers that will handle your tasks. +Start your cluster using Django's `manage.py` command:: + + $ python manage.py qcluster + + +You should see something like this:: + + 10:57:40 [Q] INFO Q Cluster-31781 starting. + 10:57:40 [Q] INFO Process-1:1 ready for work at 31784 + 10:57:40 [Q] INFO Process-1:2 ready for work at 31785 + 10:57:40 [Q] INFO Process-1:3 ready for work at 31786 + 10:57:40 [Q] INFO Process-1:4 ready for work at 31787 + 10:57:40 [Q] INFO Process-1:5 ready for work at 31788 + 10:57:40 [Q] INFO Process-1:6 ready for work at 31789 + 10:57:40 [Q] INFO Process-1:7 ready for work at 31790 + 10:57:40 [Q] INFO Process-1:8 ready for work at 31791 + 10:57:40 [Q] INFO Process-1:9 monitoring at 31792 + 10:57:40 [Q] INFO Process-1 guarding cluster at 31783 + 10:57:40 [Q] INFO Process-1:10 pushing tasks at 31793 + 10:57:40 [Q] INFO Q Cluster-31781 running. + + +Stopping the cluster with ctrl-c or either the `SIGTERM` and `SIGKILL` signals, will initiate the :ref:`stop_procedure`. + +Architecture +------------ + +.. image:: cluster.png + :alt: Django Q schema + + +Signed Tasks +"""""""""""" + +Tasks are first pickled and then signed using Django's own +signing module before being sent to a Redis list. This ensures that task +packages on the Redis server can only be executed and read by clusters +and django servers who share the same secret key. +Optionally the packages can be compressed before transport + +Pusher +"""""" + +The pusher process continuously checks the Redis list for new task +packages and pushes them on the Task Queue. + +Worker +"""""" + +A worker process checks the package signing, unpacks the task, executes +it and saves the return value. Irrespective of the failure or success of +any of these steps, the package is then pushed onto the Result Queue. + + +Monitor +""""""" + +The result monitor checks the Result Queue for processed packages and +saves both failed and successful packages to the Django database. + + +Sentinel +"""""""" + +The sentinel spawns all process and then checks the health of all +workers, including the pusher and the monitor. Reincarnating processes +if any may fail. In case of a stop signal, the sentinel will halt the +pusher and instruct the workers and monitor to finish the remaining +items. See :ref:`stop_procedure` + +Timeouts +"""""""" +Before each task execution the worker resets a timer on the sentinel and resets it again after execution. +Meanwhile the the sentinel checks if the timers don't exceed the timeout amount, in which case it will terminate the worker and reincarnate a new one. + +Scheduler +""""""""" +Once a minute the scheduler checks for any scheduled task that should be starting. +It creates and queues a new task from the schedule and sets the next run date. + +.. _stop_procedure: + +Stop procedure +"""""""""""""" + +When a stop signal is given, the sentinel exits the guard loop and instructs the pusher to stop pushing. +Once this is confirmed, the sentinel pushes poison pills onto the task queue and will wait for all the workers to die. +This ensure that the queue is emptied before the workers exit. +Afterwards the sentinel waits for the monitor to empty the result queue before the stop procedure is complete. + +- Send stop event to pusher +- Wait for pusher to exit +- Put poison pills in the Task Queue +- Wait for all the workers to clear the queue and stop +- Put a poison pill on the Result Queue +- Wait for monitor to stop + +.. :warning:: + If you force the cluster to terminate before the stop procedure has completed, you can lose tasks and their results. \ No newline at end of file diff --git a/docs/conf.py b/docs/conf.py index c81df97..1553118 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -127,6 +127,7 @@ html_theme_options = { 'github_user': 'Koed00', 'github_repo': 'django-q', 'gittip_user': 'koed00', + 'github_banner': True, 'travis_button': True, } html_sidebars = { @@ -160,7 +161,7 @@ html_theme_path = [alabaster.get_path()] # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] +#html_static_path = ['_static'] # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied @@ -298,7 +299,7 @@ texinfo_documents = [ #texinfo_appendices = [] # If false, no module index is generated. -#texinfo_domain_indices = True +texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. #texinfo_show_urls = 'footnote' diff --git a/docs/index.rst b/docs/index.rst index b6a1abe..b9b4fd8 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -9,17 +9,15 @@ Welcome to Django Q's documentation! Contents: .. toctree:: - :maxdepth: 2 + :maxdepth: 1 - Introduction Installation - Usage - Management + Tasks + Schedules + Cluster + Monitor Admin - Architecture - - -* :ref:`genindex` + * :ref:`genindex` * :ref:`search` diff --git a/docs/install.rst b/docs/install.rst new file mode 100644 index 0000000..bbbf12b --- /dev/null +++ b/docs/install.rst @@ -0,0 +1,111 @@ +Installation +============ + +- Install the latest version with pip:: + + $ pip install django-q + + +- Add :mod:`django_q` to your :const:`INSTALLED_APPS` in your projects :file:`settings.py`:: + + INSTALLED_APPS = ( + # other apps + 'django_q', + ) + +- Run Django migrations to create the database tables:: + + $ python manage.py migrate + +- Make sure you have a `Redis `__ server running + somewhere + +Configuration +------------- + +Configuration is handled via the :const:`Q_ClUSTER` dictionary in your :file:`settings.py` + +.. code:: python + + # settings.py example + Q_CLUSTER = { + 'name': 'myproject', + 'workers': 8, + 'recycle': 500, + 'timeout': 60, + 'compress': True, + 'save_limit': 250, + 'label': 'Django Q', + 'redis': { + 'host': '127.0.0.1', + 'port': 6379, + 'db': 0, } + } + + + +name +~~~~ + +Used to differentiate between projects using the same Redis server. Defaults to ``'default'``. +This can be useful if you have several projects using the same Redis server. + +.. note:: + Tasks are encrypted. When a worker encounters a task it can not decrypt, it will be discarded + +workers +~~~~~~~ + +The number of workers to use in the cluster. Defaults to CPU count of the current host, but can be set to a custom number. + +recycle +~~~~~~~ + +The number of tasks a worker will process before respawning. Useful to release resources on a regular basis. Defaults to ``500``. + +timeout +~~~~~~~ + +The number of seconds a worker is allowed to spend on a task before it's terminated. Defaults to ``None``, meaning it will never time out. +Set this to something that makes sense for your use + +compress +~~~~~~~~ + +Compress task packages to Redis. Useful for large payloads, but can add overhead when used with many small packages. +Defaults to ``False`` + +save_limit +~~~~~~~~~~ + +Limits the amount of successful tasks saved to Django. +Set to ``0`` for unlimited. Set to ``-1`` for no success storage at all. +Failures are always saved. Defaults to ``250`` + +label +~~~~~ + +The label used for the Django Admin page. Defaults to ``'Django Q'`` + +redis +~~~~~ + +Connection settings for Redis. Defaults:: + + redis: { + 'host': 'localhost', + 'port': 6379, + 'db': 0, + 'password': None, + 'socket_timeout': None, + 'charset': 'utf-8', + 'errors': 'strict', + 'unix_socket_path': None + } + +For more information on these settings please refer to the `Redis-py `__ documentation + + +.. note:: + Django Q uses your :const:`SECRET_KEY` to encrypt task packages and prevent task crossover. So make sure you have it set up in your Django settings. + diff --git a/docs/management.rst b/docs/management.rst deleted file mode 100644 index 88e1859..0000000 --- a/docs/management.rst +++ /dev/null @@ -1,20 +0,0 @@ -Management commands -=================== - -$project offers the following Django management commands - - -qcluster --------- - -.. code:: bash - - python management.py qcluster - - -qmonitor --------- - -.. code:: bash - - python management.py qmonitor diff --git a/docs/monitor.rst b/docs/monitor.rst new file mode 100644 index 0000000..e962a1a --- /dev/null +++ b/docs/monitor.rst @@ -0,0 +1,3 @@ +Monitor +======= + diff --git a/docs/schedules.rst b/docs/schedules.rst new file mode 100644 index 0000000..867fa68 --- /dev/null +++ b/docs/schedules.rst @@ -0,0 +1,2 @@ +Schedules +========= diff --git a/docs/usage.rst b/docs/tasks.rst similarity index 93% rename from docs/usage.rst rename to docs/tasks.rst index 4132f4f..1a96ce4 100644 --- a/docs/usage.rst +++ b/docs/tasks.rst @@ -1,7 +1,7 @@ -Usage +Tasks ===== -Use :py:func:`async` from your code to quickly offload tasks: +Use :py:func:`async` from your code to quickly offload tasks to the py:module:`cluster`: .. code:: python