fork/django-q2
1
0
Fork 0
mirror of https://github.com/django-q2/django-q2.git synced 2026-09-15 13:37:56 +08:00
Code Issues Packages Projects Releases Wiki Activity

Docs updates (#6)

Browse Source 
* Update django docs links to 4.0
* Update Django Q to Django Q2
* Use different theme to make difference more clear between the original and this forked one
* Added info notice about this fork
* Some small layout/typo fixes
This commit is contained in:
Stan Triepels
2022-10-13 01:38:24 +02:00
committed by GitHub
parent f698792f60
commit d32d3f5a09
18 changed files with 115 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.readthedocs.yaml
View File

@@ -14,3 +14,7 @@ build:
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: docs/conf.py
python:
install:
- requirements: docs/requirements.txt

33
README.rst
View File

@@ -3,6 +3,10 @@ A multiprocessing distributed task queue for Django
|image0| |image1| |docs|
::
Django Q2 is a fork of Django Q. Big thanks to Ilan Steemers for starting this project. Unfortunately, development has stalled since June 2021. Django Q2 is the new updated version of Django Q, with dependencies updates, docs updates and several bug fixes. Original repository: https://github.com/Koed00/django-q
Features
~~~~~~~~
@@ -30,12 +34,12 @@ Tested with: Python 3.7, 3.8, 3.9, 3.10 Django 2.2.X and 3.2.X
Brokers
~~~~~~~
- `Redis <https://django-q.readthedocs.org/en/latest/brokers.html#redis>`__
- `Disque <https://django-q.readthedocs.org/en/latest/brokers.html#disque>`__
- `IronMQ <https://django-q.readthedocs.org/en/latest/brokers.html#ironmq>`__
- `Amazon SQS <https://django-q.readthedocs.org/en/latest/brokers.html#amazon-sqs>`__
- `MongoDB <https://django-q.readthedocs.org/en/latest/brokers.html#mongodb>`__
- `Django ORM <https://django-q.readthedocs.org/en/latest/brokers.html#django-orm>`__
- `Redis <https://django-q2.readthedocs.org/en/latest/brokers.html#redis>`__
- `Disque <https://django-q2.readthedocs.org/en/latest/brokers.html#disque>`__
- `IronMQ <https://django-q2.readthedocs.org/en/latest/brokers.html#ironmq>`__
- `Amazon SQS <https://django-q2.readthedocs.org/en/latest/brokers.html#amazon-sqs>`__
- `MongoDB <https://django-q2.readthedocs.org/en/latest/brokers.html#mongodb>`__
- `Django ORM <https://django-q2.readthedocs.org/en/latest/brokers.html#django-orm>`__
Installation
~~~~~~~~~~~~
@@ -56,9 +60,9 @@ Installation
$ python manage.py migrate
- Choose a message `broker <https://django-q.readthedocs.org/en/latest/brokers.html>`__ , configure and install the appropriate client library.
- Choose a message `broker <https://django-q2.readthedocs.org/en/latest/brokers.html>`__ , configure and install the appropriate client library.
Read the full documentation at `https://django-q.readthedocs.org <https://django-q.readthedocs.org>`__
Read the full documentation at `https://django-q2.readthedocs.org <https://django-q2.readthedocs.org>`__
Configuration
@@ -82,10 +86,11 @@ All configuration settings are optional. e.g:
'redis': {
'host': '127.0.0.1',
'port': 6379,
'db': 0, }
'db': 0,
}
}
For full configuration options, see the `configuration documentation <https://django-q.readthedocs.org/en/latest/configure.html>`__.
For full configuration options, see the `configuration documentation <https://django-q2.readthedocs.org/en/latest/configure.html>`__.
Management Commands
~~~~~~~~~~~~~~~~~~~
@@ -138,7 +143,7 @@ Use `async_task` from your code to quickly offload tasks:
def print_result(task):
print(task.result)
For more info see `Tasks <https://django-q.readthedocs.org/en/latest/tasks.html>`__
For more info see `Tasks <https://django-q2.readthedocs.org/en/latest/tasks.html>`__
Schedule
@@ -183,7 +188,7 @@ Admin page or directly from your code:
schedule_type=Schedule.CRON,
cron = '0 22 * * 1-5')
For more info check the `Schedules <https://django-q.readthedocs.org/en/latest/schedules.html>`__ documentation.
For more info check the `Schedules <https://django-q2.readthedocs.org/en/latest/schedules.html>`__ documentation.
Testing
@@ -248,8 +253,8 @@ Acknowledgements
- JetBrains for their `Open Source Support Program <https://www.jetbrains.com/community/opensource>`__
.. |image0| image:: https://github.com/GDay/django-q/workflows/Tests/badge.svg?branche=master
:target: https://github.com/Koed00/django-q/actions?query=workflow%3Atests
.. |image0| image:: https://github.com/GDay/django-q2/actions/workflows/test.yml/badge.svg?branche=master
:target: https://github.com/GDay/django-q2/actions?query=workflow%3Atests
.. |image1| image:: https://coveralls.io/repos/github/GDay/django-q2/badge.svg?branch=master
:target: https://coveralls.io/github/GDay/django-q2?branch=master
.. |docs| image:: https://readthedocs.org/projects/docs/badge/?version=latest

7
docs/Dockerfile Normal file
View File

@@ -0,0 +1,7 @@
FROM sphinxdoc/sphinx
RUN mkdir -p /docs
WORKDIR /docs
COPY . .
RUN pip3 install -r requirements.txt
RUN make html

BIN
docs/_static/logo.png vendored
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 6.0 KiB

BIN
docs/_static/logo_large.png vendored
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 28 KiB

2
docs/architecture.rst
View File

@@ -2,7 +2,7 @@ Architecture
------------
.. image:: _static/cluster.png
:alt: Django Q schema
:alt: Django Q2 schema
Signed Tasks

20
docs/brokers.rst
View File

@@ -1,7 +1,7 @@
Brokers
=======
The broker sits between your Django instances and your Django Q cluster instances; accepting, saving and delivering task packages.
The broker sits between your Django instances and your Django Q2 cluster instances; accepting, saving and delivering task packages.
Currently we support a variety of brokers from the default Redis, bleeding edge Disque to the convenient ORM and fast MongoDB.
The default Redis broker does not support message receipts.
@@ -11,7 +11,7 @@ Keep in mind this is not the same as a failing task. If a tasks code crashes, th
Even though this might be acceptable in some use cases, you might prefer brokers with message receipts support.
These guarantee delivery by waiting for the cluster to send a receipt after the task has been processed.
In case a receipt has not been received after a set time, the task package is put back in the queue.
Django Q supports this behavior by setting the :ref:`retry` timer on brokers that support message receipts.
Django Q2 supports this behavior by setting the :ref:`retry` timer on brokers that support message receipts.
Some pointers:
@@ -29,7 +29,7 @@ Support for more brokers is being worked on.
Redis
-----
The default broker for Django Q clusters.
The default broker for Django Q2 clusters.
* Atomic
* Requires `Redis-py <https://github.com/andymccurdy/redis-py>`__ client library: ``pip install redis``
@@ -47,7 +47,7 @@ Bulk task retrieval is supported via the :ref:`bulk` option.
* Delivery receipts
* Atomic
* Needs Django's `Cache framework <https://docs.djangoproject.com/en/2.2/topics/cache/#setting-up-the-cache>`__ configured for monitoring
* Needs Django's `Cache framework <https://docs.djangoproject.com/en/4.0/topics/cache/#setting-up-the-cache>`__ configured for monitoring
* Compatible with `Tynd <https://disque.tynd.co/>`__ Disque addon on `Heroku <https://heroku.com>`__
* Still considered Alpha software
* Supports bulk dequeue
@@ -60,7 +60,7 @@ This HTTP based queue service is both available directly via `Iron.io <http://ww
* Delivery receipts
* Supports bulk dequeue
* Needs Django's `Cache framework <https://docs.djangoproject.com/en/2.2/topics/cache/#setting-up-the-cache>`__ configured for monitoring
* Needs Django's `Cache framework <https://docs.djangoproject.com/en/4.0/topics/cache/#setting-up-the-cache>`__ configured for monitoring
* Requires the `iron-mq <https://github.com/iron-io/iron_mq_python>`__ client library: ``pip install iron-mq``
* See the :ref:`ironmq_configuration` configuration section for options.
@@ -72,7 +72,7 @@ Although `SQS <https://aws.amazon.com/sqs/>`__ is not the fastest, it is stable,
* Delivery receipts
* Maximum message size is 256Kb
* Supports bulk dequeue up to 10 messages with a maximum total size of 256Kb
* Needs Django's `Cache framework <https://docs.djangoproject.com/en/2.2/topics/cache/#setting-up-the-cache>`__ configured for monitoring
* Needs Django's `Cache framework <https://docs.djangoproject.com/en/4.0/topics/cache/#setting-up-the-cache>`__ configured for monitoring
* Requires the `boto3 <https://github.com/boto/boto3>`__ client library: ``pip install boto3``
* See the :ref:`sqs_configuration` configuration section for options.
@@ -83,7 +83,7 @@ This highly scalable NoSQL database makes for a very fast and reliably persisten
Usually available on most PaaS providers, as `open-source <https://www.mongodb.org/>`__ or commercial `enterprise <https://www.mongodb.com/lp/download/mongodb-enterprise>`__ edition.
* Delivery receipts
* Needs Django's `Cache framework <https://docs.djangoproject.com/en/2.2/topics/cache/#setting-up-the-cache>`__ configured for monitoring
* Needs Django's `Cache framework <https://docs.djangoproject.com/en/4.0/topics/cache/#setting-up-the-cache>`__ configured for monitoring
* Can be configured as the Django cache-backend through several open-source cache providers.
* Requires the `pymongo <https://github.com/mongodb/mongo-python-driver>`__ driver: ``pip install pymongo``
* See the :ref:`mongo_configuration` configuration section for options.
@@ -98,8 +98,8 @@ However for a medium message rate and scheduled tasks, this is the most convenie
* Delivery receipts
* Supports bulk dequeue
* Needs Django's `Cache framework <https://docs.djangoproject.com/en/2.2/topics/cache/#setting-up-the-cache>`__ configured for monitoring
* Can be `configured <https://docs.djangoproject.com/en/2.2/topics/cache/#database-caching>`__ as its own cache backend.
* Needs Django's `Cache framework <https://docs.djangoproject.com/en/4.0/topics/cache/#setting-up-the-cache>`__ configured for monitoring
* Can be `configured <https://docs.djangoproject.com/en/4.0/topics/cache/#database-caching>`__ as its own cache backend.
* Queue editable in Django Admin
* See the :ref:`orm_configuration` configuration on how to set it up.
@@ -118,7 +118,7 @@ You can override the :class:`Broker` or any of its existing derived broker types
def info(self):
return 'My Custom Broker'
Using the :ref:`broker_class` configuration setting you can then instruct Django Q to use this instead of one of the existing brokers:
Using the :ref:`broker_class` configuration setting you can then instruct Django Q2 to use this instead of one of the existing brokers:
.. code-block:: python

4
docs/cluster.rst
View File

@@ -3,7 +3,7 @@ Cluster
=======
.. py:currentmodule:: django_q
Django Q uses Python's multiprocessing module to manage a pool of workers that will handle your tasks.
Django Q2 uses Python's multiprocessing module to manage a pool of workers that will handle your tasks.
Start your cluster using Django's ``manage.py`` command::
$ python manage.py qcluster
@@ -60,7 +60,7 @@ If you host on `Heroku <https://heroku.com>`__ or you are using `Honcho <https:/
Process managers
----------------
While you certainly can run a Django Q with a process manager like `Supervisor <http://supervisord.org/>`__ or `Circus <https://circus.readthedocs.org/en/latest/>`__ it is not strictly necessary.
While you certainly can run a Django Q2 with a process manager like `Supervisor <http://supervisord.org/>`__ or `Circus <https://circus.readthedocs.org/en/latest/>`__ it is not strictly necessary.
The cluster has an internal sentinel that checks the health of all the processes and recycles or reincarnates according to your settings or in case of unexpected crashes.
Because of the multiprocessing daemonic nature of the cluster, it is impossible for a process manager to determine the clusters health and resource usage.

37
docs/conf.py
View File

@@ -16,7 +16,7 @@
import os
import sys
import alabaster
import sphinx_rtd_theme
myPath = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, myPath + '/../')
@@ -37,7 +37,7 @@ nitpick_ignore = [('py:class', 'datetime')]
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'alabaster',
'sphinx_rtd_theme',
'sphinx.ext.todo',
'sphinx.ext.intersphinx',
# 'sphinx.ext.autodoc'
@@ -62,9 +62,9 @@ source_suffix = '.rst'
master_doc = 'index'
# General information about the project.
project = 'Django Q'
copyright = '2015-2021, Ilan Steemers'
author = 'Ilan Steemers'
project = 'Django Q2'
copyright = '2015-2021, Ilan Steemers - 2022, Stan Triepels'
author = 'Ilan Steemers, Stan Triepels'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
@@ -124,19 +124,16 @@ todo_include_todos = True
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'alabaster'
html_theme = 'sphinx_rtd_theme'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.None
html_theme_options = {
'description': "A multiprocessing task queue for Django",
'logo': 'logo.png',
'github_user': 'Koed00',
'github_repo': 'django-q',
'github_banner': True,
'travis_button': True,
'analytics_id': 'UA-64807059-1'
# 'description': "A multiprocessing task queue for Django",
# 'github_user': 'GDay',
# 'github_repo': 'django-q2',
# 'github_banner': True,
}
html_sidebars = {
'**': [
@@ -147,7 +144,7 @@ html_sidebars = {
]
}
# Add any paths that contain custom themes here, relative to this directory.
html_theme_path = [alabaster.get_path()]
# html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
@@ -232,7 +229,7 @@ html_static_path = ['_static']
# html_search_scorer = 'scorer.js'
# Output file base name for HTML help builder.
htmlhelp_basename = 'DjangoQdoc'
htmlhelp_basename = 'DjangoQ2doc'
# -- Options for LaTeX output ---------------------------------------------
@@ -254,13 +251,13 @@ latex_elements = {
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'DjangoQ.tex', 'Django Q Documentation',
(master_doc, 'DjangoQ2.tex', 'Django Q2 Documentation',
'Ilan Steemers', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
latex_logo = '_static/logo_large.png'
# latex_logo = '_static/logo_large.png'
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
@@ -284,7 +281,7 @@ latex_logo = '_static/logo_large.png'
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'djangoq', 'Django Q Documentation',
(master_doc, 'djangoq2', 'Django Q2 Documentation',
[author], 1)
]
@@ -299,8 +296,8 @@ man_pages = [
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'DjangoQ', 'Django Q Documentation',
author, 'DjangoQ', 'A multiprocessing distributed task queue for Django.',
(master_doc, 'DjangoQ2', 'Django Q2 Documentation',
author, 'DjangoQ2', 'A multiprocessing distributed task queue for Django.',
'Miscellaneous'),
]

32
docs/configure.rst
View File

@@ -16,7 +16,7 @@ Configuration is handled via the ``Q_CLUSTER`` dictionary in your :file:`setting
'save_limit': 250,
'queue_limit': 500,
'cpu_affinity': 1,
'label': 'Django Q',
'label': 'Django Q2',
'redis': {
'host': '127.0.0.1',
'port': 6379,
@@ -96,7 +96,7 @@ Only works with brokers that support delivery receipts. Defaults to 60.
The value must be bigger than the time it takes to complete longest task, i.e. :ref:`timeout` must be less than retry value and all tasks must complete
in less time than the selected retry time. If this does not hold, i.e. the retry value is less than timeout or less than it takes to finish a task,
Django-Q will start the task again if the used broker supports receipts.
Django-Q2 will start the task again if the used broker supports receipts.
For example, with the following code
@@ -167,7 +167,7 @@ Defaults to ``workers**2``.
label
~~~~~
The label used for the Django Admin page. Defaults to ``'Django Q'``
The label used for the Django Admin page. Defaults to ``'Django Q2'``
.. _catch_up:
@@ -225,7 +225,7 @@ of the cache connection you want to use instead of a direct Redis connection::
.. tip::
Django Q uses your ``SECRET_KEY`` to sign task packages and prevent task crossover. So make sure you have it set up in your Django settings.
Django Q2 uses your ``SECRET_KEY`` to sign task packages and prevent task crossover. So make sure you have it set up in your Django settings.
.. _disque_configuration:
@@ -243,7 +243,7 @@ If you want to use Disque as your broker, set this to a list of available Disque
}
Django Q is also compatible with the `Tynd Disque <https://disque.tynd.co/>`__ addon on `Heroku <https://heroku.com>`__::
Django Q2 is also compatible with the `Tynd Disque <https://disque.tynd.co/>`__ addon on `Heroku <https://heroku.com>`__::
# example Tynd Disque connection
import os
@@ -339,15 +339,15 @@ Using the Django ORM backend will also enable the Queued Tasks table in the Admi
If you need better performance , you should consider using a different database backend than the main project.
Set ``orm`` to the name of that database connection and make sure you run migrations on it using the ``--database`` option.
When using the Django database as a message broker, you can set the ``has_replica`` boolean keyword to ensure Django-Q works properly letting a `Database Router <https://docs.djangoproject.com/en/3.2/topics/db/multi-db/>`__. ::
When using the Django database as a message broker, you can set the ``has_replica`` boolean keyword to ensure Django-Q2 works properly letting a `Database Router <https://docs.djangoproject.com/en/3.2/topics/db/multi-db/>`__. ::
# example ORM broker connection with replica database
Q_CLUSTER = {
...
'orm': 'default',
'has_replica': True
}
...
'orm': 'default',
'has_replica': True
}
.. _mongo_configuration:
@@ -364,8 +364,8 @@ To use MongoDB as a message broker you simply provide the connection information
'retry': 70,
'queue_limit': 100,
'mongo': {
'host': '127.0.0.1',
'port': 27017
'host': '127.0.0.1',
'port': 27017
}
}
@@ -389,7 +389,7 @@ You can use a custom broker class for your cluster workers::
'name': 'Custom',
'workers': 8,
'timeout': 60,
'broker_class: 'myapp.broker.CustomBroker'
'broker_class': 'myapp.broker.CustomBroker'
}
Make sure your ``CustomBroker`` class inherits from either the base :class:`Broker` class or one of its children.
@@ -431,7 +431,7 @@ Defaults to ``True``
error_reporter
~~~~~~~~~~~~~~
You can redirect worker exceptions directly to various error reporters (for example `Rollbar <https://rollbar.com/>`__ or `Sentry <https://docs.sentry.io/>`__) by installing Django Q with the necessary `extras <https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies>`__.
You can redirect worker exceptions directly to various error reporters (for example `Rollbar <https://rollbar.com/>`__ or `Sentry <https://docs.sentry.io/>`__) by installing Django Q2 with the necessary `extras <https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies>`__.
To enable installed error reporters, you must provide the configuration settings required by an error reporter extension::
@@ -440,12 +440,12 @@ To enable installed error reporters, you must provide the configuration settings
'error_reporter': {
'rollbar': {
'access_token': '32we33a92a5224jiww8982',
'environment': 'Django-Q'
'environment': 'Django-Q2'
}
}
}
For more information on error reporters and developing error reporting plugins for Django Q, see :doc:`errors<errors>`.
For more information on error reporters and developing error reporting plugins for Django Q2, see :doc:`errors<errors>`.
cpu_affinity
~~~~~~~~~~~~

7
docs/docker-compose.yaml Normal file
View File

@@ -0,0 +1,7 @@
version: "3.0"
services:
docs:
container_name: djangoq2-docs
build: .
volumes:
- .:/docs

4
docs/errors.rst
View File

@@ -2,8 +2,8 @@ Errors
------
.. py:currentmodule:: django_q
Django Q uses a pluggable error reporter system based upon python `extras <https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies>`__, allowing anyone to develop plugins for their favorite error reporting and monitoring integration. Currently implemented examples include `Rollbar <https://rollbar.com/>`__ and `Sentry <https://docs.sentry.io/>`__.
Django Q2 uses a pluggable error reporter system based upon python `extras <https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies>`__, allowing anyone to develop plugins for their favorite error reporting and monitoring integration. Currently implemented examples include `Rollbar <https://rollbar.com/>`__ and `Sentry <https://docs.sentry.io/>`__.
Error reporting plugins register a class which implements a ``report`` method, which is invoked when a Django Q cluster encounters an error, passing information to the particular service. Error reporters must be :ref:`configured<error_reporter>` via the ``Q_CLUSTER`` dictionary in your :file:`settings.py`. These settings are passed as kwargs upon initiation of the Error Reporter. Therefore, in order to implement a new plugin, a package must expose a class which will be instantiated with the necessary information via the ``Q_CLUSTER`` settings and implements a single ``report`` method.
Error reporting plugins register a class which implements a ``report`` method, which is invoked when a Django Q2 cluster encounters an error, passing information to the particular service. Error reporters must be :ref:`configured<error_reporter>` via the ``Q_CLUSTER`` dictionary in your :file:`settings.py`. These settings are passed as kwargs upon initiation of the Error Reporter. Therefore, in order to implement a new plugin, a package must expose a class which will be instantiated with the necessary information via the ``Q_CLUSTER`` settings and implements a single ``report`` method.
For example implementations, see `django-q-rollbar <https://github.com/danielwelch/django-q-rollbar>`__ and `django-q-sentry <https://github.com/danielwelch/django-q-sentry>`__

8
docs/examples.rst
View File

@@ -38,7 +38,7 @@ Sending an email can take a while so why not queue it:
# this schedule will erase itself after having run
Since you're only telling Django Q to take care of the emails, you can quickly move on to serving web pages to your user.
Since you're only telling Django Q2 to take care of the emails, you can quickly move on to serving web pages to your user.
Signals
=======
@@ -142,7 +142,7 @@ The hook is practical here, because it allows us to detach the sending task from
Haystack
========
If you use `Haystack <http://haystacksearch.org/>`__ as your projects search engine,
here's an example of how you can have Django Q take care of your indexes in real time using model signals:
here's an example of how you can have Django Q2 take care of your indexes in real time using model signals:
.. code-block:: python
@@ -286,7 +286,7 @@ Adapted from `Sebastian Raschka's blog <http://sebastianraschka.com/Articles/201
Django Q is not optimized for distributed computing, but this example will give you an idea of what you can do with task :doc:`group`.
Django Q2 is not optimized for distributed computing, but this example will give you an idea of what you can do with task :doc:`group`.
Alternatively the ``parzen_async()`` function can also be written with :func:`async_iter`, which automatically utilizes the cache backend and groups to return a single result from an iterable:
@@ -389,7 +389,7 @@ Requires cache to be enabled. Save file in your Django project's root directory
.. note::
If you have an example you want to share, please submit a pull request on `github <https://github.com/Koed00/django-q/>`__.
If you have an example you want to share, please submit a pull request on `github <https://github.com/GDay/django-q2/>`__.

13
docs/index.rst
View File

@@ -3,9 +3,12 @@
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to Django Q
===================
Django Q is a native Django task queue, scheduler and worker application using Python multiprocessing.
Welcome to Django Q2
====================
Django Q2 is a native Django task queue, scheduler and worker application using Python multiprocessing.
.. note::
Django Q2 is a fork of Django Q. Big thanks to Ilan Steemers for starting this project. Unfortunately, development of Django Q has stalled since June 2021. Django Q2 is the new updated version of Django Q, with dependencies updates, docs updates and several bug fixes. Original repository: https://github.com/Koed00/django-q
Features
@@ -24,7 +27,7 @@ Features
- Rollbar and Sentry support
Django Q is tested with: Python 3.7 and 3.8, Django 2.2.x and 3.2.x
Django Q is tested with: Python 3.7, 3.8, 3.9 and 3.10, Django 2.2.x and 3.2.x
Currently available in English, German and French.
@@ -50,5 +53,3 @@ Contents:
Examples <examples>
* :ref:`genindex`
* :ref:`search`

33
docs/install.rst
View File

@@ -18,20 +18,20 @@ Installation
$ python manage.py migrate
- Choose a message :doc:`broker<brokers>` , configure it and install the appropriate client library.
- Choose a message :doc:`broker<brokers>`, configure it and install the appropriate client library.
- Run Django Q cluster in order to handle tasks async::
- Run Django Q2 cluster in order to handle tasks async::
$ python manage.py qcluster
Requirements
------------
Django Q is tested for Python 3.7, 3.8 and 3.9
Django Q2 is tested for Python 3.7, 3.8, 3.9 and 3.10
- `Django <https://www.djangoproject.com>`__
Django Q aims to use as much of Django's standard offerings as possible
Django Q2 aims to use as much of Django's standard offerings as possible
The code is tested against Django versions `2.2.x` and `3.2.x`.
Please note that Django versions below 2.0 do not support Python 3.7
@@ -76,7 +76,7 @@ Optional
$ pip install pymongo
- `Redis <http://redis.io/>`__ server is the default broker for Django Q. It provides the best performance and does not require Django's cache framework for monitoring.
- `Redis <http://redis.io/>`__ server is the default broker for Django Q2. It provides the best performance and does not require Django's cache framework for monitoring.
- `Disque <https://github.com/antirez/disque>`__ server is based on Redis by the same author, but focuses on reliable queues. Currently in Alpha, but highly recommended. You can either build it from source or use it on Heroku through the `Tynd <https://disque.tynd.co/>`__ beta.
@@ -110,16 +110,13 @@ Add-ons
- `django-q-email <https://github.com/joeyespo/django-q-email>`__ is a compatible Django email backend that will automatically async queue your emails.
Compatibility
-------------
Django Q is still a young project. If you do find any incompatibilities please submit an issue on `github <https://github.com/Koed00/django-q>`__.
OS X
~~~~
Running Django Q on OS X should work fine, except for the following known issues:
Running Django Q2 on OS X should work fine, except for the following known issues:
* :meth:`multiprocessing.Queue.qsize()` is not supported. This leads to the monitor not reporting the internal queue size of clusters running under OS X.
* CPU count through :func:`multiprocessing.cpu_count()` does not work. Installing :ref:`psutil<psutil_package>` provides Django Q with an alternative way of determining the number of CPU's on your system
* CPU count through :func:`multiprocessing.cpu_count()` does not work. Installing :ref:`psutil<psutil_package>` provides Django Q2 with an alternative way of determining the number of CPU's on your system
* CPU affinity is provided by :ref:`psutil<psutil_package>` which at this time does not support this feature on OSX. The code however is aware of this and will fake the CPU affinity assignment in the logs without actually assigning it. This way you can still develop with this setting.
Windows
@@ -130,14 +127,13 @@ This will run all ``async`` calls inline through a single cluster worker without
Other known issues are:
* :func:`os.getppid()` is only supported under windows since Python 3.2. If you use an older version you need to install :ref:`psutil<psutil_package>` as an alternative.
* CPU count through :func:`multiprocessing.cpu_count()` occasionally fails on servers. Installing :ref:`psutil<psutil_package>` provides Django Q with an alternative way of determining the number of CPU's on your system
* CPU count through :func:`multiprocessing.cpu_count()` occasionally fails on servers. Installing :ref:`psutil<psutil_package>` provides Django Q2 with an alternative way of determining the number of CPU's on your system
* The monitor and info commands rely on the Curses package which is not officially supported on windows. There are however some ports available like `this one <http://www.lfd.uci.edu/~gohlke/pythonlibs/#curses>`__ by Christoph Gohlke.
Python
~~~~~~
The code is always tested against the latest version Python 3 and we try to stay compatible with the last two versions of each.
Current tests are performed with 3.7 and 3.8
If you do encounter any regressions with earlier versions, please submit an issue on `github <https://github.com/Koed00/django-q>`__
Current tests are performed with 3.7, 3.8, 3.9 and 3.10
If you do encounter any regressions with earlier versions, please submit an issue on `github <https://github.com/GDay/django-q2>`__
.. note::
@@ -146,16 +142,13 @@ If you do encounter any regressions with earlier versions, please submit an issu
Open-source packages
~~~~~~~~~~~~~~~~~~~~
Django Q is always tested with the latest versions of the required and optional Python packages. We try to keep the dependencies as up to date as possible.
You can reference the `requirements <https://github.com/Koed00/django-q/blob/master/requirements.txt>`__ file to determine which versions are currently being used for tests and development.
Django Q2 is always tested with the latest versions of the required and optional Python packages. We try to keep the dependencies as up to date as possible.
You can reference the `requirements <https://github.com/GDay/django-q2/blob/master/requirements.txt>`__ file to determine which versions are currently being used for tests and development.
Django
~~~~~~
We strive to be compatible with last two major version of Django.
At the moment this means we support the 2.2.x and 3.1.x releases.
At the moment this means we support the 2.2.x and 3.2.x releases.
Since we are now no longer supporting Python 2, we can also not support older versions of Django that do not support Python >= 3.6
For this you can always use older releases, but they are no longer maintained.

1
docs/requirements.txt Normal file
View File

@@ -0,0 +1 @@
sphinx-rtd-theme==1.0.0

4
docs/signals.rst
View File

@@ -5,7 +5,7 @@ Signals
Available signals
-----------------
Django Q emits the following signals during its lifecycle.
Django Q2 emits the following signals during its lifecycle.
Before enqueuing a task
"""""""""""""""""""""""
@@ -33,7 +33,7 @@ executed by a worker and processed by the monitor. It included the ``task`` dict
Subscribing to a signal
-----------------------
Connecting to a Django Q signal is done the same as any other Django
Connecting to a Django Q2 signal is done the same as any other Django
signal::
from django.dispatch import receiver

9
docs/tasks.rst
View File

@@ -7,9 +7,6 @@ Tasks
async_task()
------------
.. warning:: Since Python 3.7 `async` became a reserved keyword and was refactored to `async_task`
Use :func:`async_task` from your code to quickly offload tasks to the :class:`Cluster`:
.. code:: python
@@ -101,7 +98,7 @@ Please note that this will override any other option keywords.
.. note::
For tasks to be processed you will need to have a worker cluster running in the background using ``python manage.py qcluster``
or you need to configure Django Q to run in synchronous mode for testing using the :ref:`sync` option.
or you need to configure Django Q2 to run in synchronous mode for testing using the :ref:`sync` option.
AsyncTask
@@ -221,7 +218,7 @@ Instead of setting ``sync`` on each individual ``async_task`` you can also confi
Connection pooling
------------------
Django Q tries to pass broker instances around its parts as much as possible to save you from running out of connections.
Django Q2 tries to pass broker instances around its parts as much as possible to save you from running out of connections.
When you are making individual calls to :func:`async_task` a lot though, it can help to set up a broker to reuse for :func:`async_task`:
.. code:: python
@@ -236,7 +233,7 @@ When you are making individual calls to :func:`async_task` a lot though, it can
.. tip::
If you are using `django-redis <https://github.com/niwinz/django-redis>`__ and the redis broker, you can :ref:`configure <django_redis>` Django Q to use its connection pool.
If you are using `django-redis <https://github.com/niwinz/django-redis>`__ and the redis broker, you can :ref:`configure <django_redis>` Django Q2 to use its connection pool.
Reference