Multiple queue, multiple cluster in one site (#71)

* Support for multi-queue, multi-cluster configuration. API changes include:
  * Adding `cluster` to async_task() parameters and Task model
  * Adding argument --name to qcluster command
  * Necessary adjustments to Conf and Broker classes
  * Some admin improvements

* Add settings.Q_CLUSTER['ALT_CLUSTERS']: q_cluster config overrides for alternative clusters;
Add Conf.CLUSTER_NAME: separate usage from Conf.PREFIX;
QueueAdmin/OrmQ detail page enhanced: now displaying args/kwargs/q_options instead of encrypted payload.

* if `cluster` argument is not set (the default), async_task() and schedule() will be handled by the default cluster; Documentation update.

* Text cleanup

* Documentation update.

* Fix TIMEOUT setting in Windows for non-default cluster

---------

Co-authored-by: Stan Triepels <1939656+GDay@users.noreply.github.com>
This commit is contained in:
sinowood
2023-04-02 16:36:17 +02:00
committed by GitHub
co-authored by Stan Triepels
parent f8520c9cda
commit 52c04217e9
17 changed files with 215 additions and 47 deletions
+30 -1
View File
@@ -20,7 +20,18 @@ Configuration is handled via the ``Q_CLUSTER`` dictionary in your :file:`setting
'redis': {
'host': '127.0.0.1',
'port': 6379,
'db': 0, }
'db': 0, },
'ALT_CLUSTERS': {
'long': {
'timeout': 3000,
'retry': 3600,
'max_attempts': 2,
},
'short': {
'timeout': 10,
'max_attempts': 1,
},
}
}
All configuration settings are optional:
@@ -459,6 +470,24 @@ As a rule of thumb; cpu_affinity 1 favors repetitive short running tasks, while
*Psutil does not support cpu affinity on OS X at this time.*
.. _alt-clusters:
ALT_CLUSTERS
~~~~~~~~~~~~
For multiple clusters working on multiple queues to run in one Django site.
ALT_CLUSTERS should be a dict with cluster_name as its key, and the value is the configuration for the cluster
with the key as its name. The configuration items are consistent with Q_CLUSTER,
except for a few items such as name/cluster_name/ALT_CLUSTER, which are not available of course.
See :ref:`multiple-queues`.
.. note::
For a cluster, if its name is in ALT_CLUSTERS, the config item in ALT_CLUSTER will override
the same config item in the Q_CLUSTER root. Other config items in Q_CLUSTER root remain in effect for this cluster.
.. py:module:: django_q
.. rubric:: Footnotes