Adds a MongoDB broker

* adds mongo broker tests
* adds mongo brokers docs

* broker info is now cached when it needs to connect to a server, to reduce traffic.
This commit is contained in:
Ilan Steemers
2015-09-25 20:03:39 +02:00
parent ef6a60478a
commit 826178c463
11 changed files with 188 additions and 11 deletions
+11 -1
View File
@@ -2,7 +2,7 @@ Brokers
=======
The broker sits between your Django instances and your Django Q cluster instances; accepting, saving and delivering task packages.
Currently we support a variety of brokers from the default Redis, bleeding edge Disque to the convenient Amazon SQS.
Currently we support a variety of brokers from the default Redis, bleeding edge Disque to the convenient ORM and fast MongoBD.
The default Redis broker does not support message receipts.
This means that in case of a catastrophic failure of the cluster server or worker timeouts, tasks that were being executed get lost.
@@ -73,6 +73,16 @@ Although `SQS <https://aws.amazon.com/sqs/>`__ is not the fastest, it is stable,
* Requires the `boto3 <https://github.com/boto/boto3>`__ client library: ``pip install boto3``
* See the :ref:`sqs_configuration` configuration section for options.
MongoDB
-------
The
* Delivery receipts
* Needs Django's `Cache framework <https://docs.djangoproject.com/en/1.8/topics/cache/#setting-up-the-cache>`__ configured for monitoring
* Requires the `pymongo <https://github.com/mongodb/mongo-python-driver>`__ driver: ``pip install pymongo``
* See the :ref:`mongo_configuration` configuration section for options.
.. _orm_broker:
Django ORM
+28 -1
View File
@@ -264,6 +264,33 @@ Using the Django ORM backend will also enable the Queued Tasks table in the Admi
If you need better performance , you should consider using a different database backend than the main project.
Set ``orm`` to the name of that database connection and make sure you run migrations on it using the ``--database`` option.
.. _mongo_configuration:
mongo
~~~~~
To use MongoDB as a message broker you simply provide the connection information in a dictionary ::
# example MongoDB broker connection
Q_CLUSTER = {
'name': 'MongoDB',
'workers': 8,
'timeout': 60,
'retry': 70,
'queue_limit': 100,
'mongo': {
'host': '127.0.0.1',
'port': 27017
}
}
The ``mongo`` dictionary can contain any of the parameters exposed by pymongo's`MongoClient<https://api.mongodb.org/python/current/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient>`__
mongo_db
~~~~~~~~
When using the MongoDB broker you can optionally provide a database name to use for the queues.
Defaults to ``django-q``
.. _bulk:
bulk
@@ -273,7 +300,7 @@ Especially HTTP based or very high latency servers can benefit from bulk dequeue
Keep in mind however that settings this too high can degrade performance with multiple clusters or very large task packages.
Not supported by the default Redis broker.
Defaults to 1.
Defaults to ``1``.
cache
~~~~~