Add option for acknowledging failed tasks (globally and per-task)

If a task fails with an exception, it is retried until it
succeeds. This is contrary to what is said in the documentation: under
the "Architecture" section, heading "Broker" it says that even when a
task errors, it's still considered a successful delivery. Failed tasks
never get acknowledged however, thereby being retried after the
timeout period. See also issues #238 and #194.

This patch adds an option to acknowledge failures, thereby closing
issue #238. Issue #194 would require some more work. The default of
this option is set to `False`, thereby maintaining backwards
compatibility.

docs/architecture.rst

@@ -20,9 +20,13 @@ Broker
 The broker collects task packages from the django instances and queues them for pick up by a cluster.
 If the broker supports message receipts, it will keep a copy of the tasks around until a cluster acknowledges the processing of the task.
 Otherwise it is put back in the queue after a timeout period. This ensure at-least-once delivery.
-Note that even if the task errors when processed by the cluster, this is considered a successful delivery.
 Most failed deliveries will be the result of a worker or the cluster crashing before the task was saved.
 
+.. note::
+   When the :ref:`ack_failures` option is set to ``False`` (the default), a task is
+   considered a failed delivery when it raises an ``Exception``. Set
+   this option to ``True`` to acknowledge failed tasks as successful.
+
 Pusher
 """"""
 

docs/configure.rst

@@ -59,6 +59,13 @@ timeout
 The number of seconds a worker is allowed to spend on a task before it's terminated. Defaults to ``None``, meaning it will never time out.
 Set this to something that makes sense for your project. Can be overridden for individual tasks.
 
+.. _ack_failures:
+
+ack_failures
+~~~~~~~~~~~~
+
+When set to ``True``, also acknowledge unsuccessful tasks. This causes failed tasks to be considered as successful deliveries, thereby removing them from the task queue. Can also be set per-task by passing the ``ack_failure`` option to :func:`async`. Defaults to ``False``.
+
 .. _retry:
 
 retry

docs/tasks.rst

@@ -54,6 +54,10 @@ timeout
 """""""
 Overrides the cluster's timeout setting for this task.
 
+ack_failure
+"""""""""""
+Overrides the cluster's :ref:`ack_failures` setting for this task.
+
 sync
 """"
 Simulates a task execution synchronously. Useful for testing.
@@ -244,6 +248,7 @@ Reference
    :param str group: An optional group identifier
    :param int timeout: Overrides global cluster :ref:`timeout`.
    :param bool save: Overrides global save setting for this task.
+   :param bool ack_failure: Overrides the global :ref:`ack_failures` setting for this task.
    :param bool sync: If set to True, async will simulate a task execution
    :param cached: Output the result to the cache backend. Bool or timeout in seconds
    :param broker: Optional broker connection from :func:`brokers.get_broker`