Merge pull request #84 from Koed00/dev

Adds cached result backend

docs/configure.rst

@@ -308,6 +308,13 @@ cache
 For some brokers, you will need to set up the Django `cache framework <https://docs.djangoproject.com/en/1.8/topics/cache/#setting-up-the-cache>`__
 to gather statistics for the monitor. You can indicate which cache to use by setting this value. Defaults to ``default``.
 
+.. _cached:
+
+cached
+~~~~~~
+Switches all task and result functions from using the database backend to the cache backend. This is the same as setting the keyword ``cached=True`` on all task functions.
+Instead of a bool this can also be set to the number of seconds you want the cache to retain results. e.g. ``cached=60``
+
 scheduler
 ~~~~~~~~~
 You can disable the scheduler by setting this option to ``False``. This will reduce a little overhead if you're not using schedules, but is most useful if you want to temporarily disable all schedules.

docs/examples.rst

@@ -253,8 +253,7 @@ Adapted from `Sebastian Raschka's blog <http://sebastianraschka.com/Articles/201
     # Group example with Parzen-window estimation
     import numpy
 
-    from django_q.tasks import async, result_group,\
-        count_group, delete_group
+    from django_q.tasks import async, result_group, delete_group
 
     # the estimation function
     def parzen_estimation(x_samples, point_x, h):
@@ -268,30 +267,47 @@ Adapted from `Sebastian Raschka's blog <http://sebastianraschka.com/Articles/201
                 k_n += 1
         return h, (k_n / len(x_samples)) / (h ** point_x.shape[1])
 
-
     # create 100 calculations and send them to the cluster
     def parzen_async():
         # clear the previous results
-        delete_group('parzen', tasks=True)
+        delete_group('parzen', cached=True)
         mu_vec = numpy.array([0, 0])
         cov_mat = numpy.array([[1, 0], [0, 1]])
-        sample = numpy.random.\
+        sample = numpy.random. \
             multivariate_normal(mu_vec, cov_mat, 10000)
         widths = numpy.linspace(1.0, 1.2, 100)
         x = numpy.array([[0], [0]])
-        # async them with a group label and a hook
+        # async them with a group label to the cache backend
         for w in widths:
             async(parzen_estimation, sample, x, w,
-             group='parzen', hook=parzen_hook)
+                  group='parzen', cached=True)
+        # return after 100 results
+        return result_group('parzen', count=100, cached=True)
 
-    # wait for 100 results to return and print it.
-    def parzen_hook(task):
-        if task.group_count() == 100:
-            print(task.group_result())
 
 
 Django Q is not optimized for distributed computing, but this example will give you an idea of what you can do with task :ref:`groups`.
 
+Alternatively the ``parzen_async()`` function can also be written with :func:`async_iter`, which automatically utilizes the cache backend and groups to return a single result from an iterable:
+
+.. code-block:: python
+
+    # create 100 calculations and send them to the cluster
+    # with async_iter
+    def parzen_async():
+        mu_vec = numpy.array([0, 0])
+        cov_mat = numpy.array([[1, 0], [0, 1]])
+        sample = numpy.random. \
+            multivariate_normal(mu_vec, cov_mat, 10000)
+        widths = numpy.linspace(1.0, 1.2, 100)
+        x = numpy.array([[0], [0]])
+        # async them with async iterable
+        args = [(sample, x, w) for w in widths]
+        result_id = async_iter(parzen_estimation, args)
+        # return the result or timeout after 10 seconds
+        return result(result_id, wait=10000)
+
+
 .. note::
 
     If you have an example you want to share, please submit a pull request on `github <https://github.com/Koed00/django-q/>`__.

docs/index.rst

@@ -15,7 +15,7 @@ Features
 -  Asynchronous tasks
 -  Scheduled and repeated tasks
 -  Encrypted and compressed packages
--  Failure and success database
+-  Failure and success database or cache
 -  Result hooks and groups
 -  Django Admin integration
 -  PaaS compatible with multiple instances

docs/tasks.rst

@@ -59,6 +59,12 @@ sync
 Simulates a task execution synchronously. Useful for testing.
 Can also be forced globally via the :ref:`sync` configuration option.
 
+cached
+""""""
+Redirects the result to the cache backend instead of the database if set to ``True`` or to an integer indicating the cache timeout in seconds.
+e.g. ``cached=60``. Especially useful with large and group operations where you don't need the all results in your
+database and want to take advantage of the speed of your cache backend.
+
 broker
 """"""
 A broker instance, in case you want to control your own connections.
@@ -83,6 +89,28 @@ Please not that this will override any other option keywords.
     For tasks to be processed you will need to have a worker cluster running in the background using ``python manage.py qcluster``
     or you need to configure Django Q to run in synchronous mode for testing using the :ref:`sync` option.
 
+
+
+Async Iterable
+--------------
+If you have an iterable object with arguments for a function, you can use :func:`async_iter` to async them with a single command::
+
+    # Async Iterable example
+    from django_q.tasks import async_iter, result
+
+    # set up a list of arguments for math.floor
+    iter = [i for i in range(100)]
+
+    # async iter them
+    id=async_iter('math.floor',iter)
+
+    # wait for the collated result for 1 second
+    result_list = result(id, wait=1000)
+
+This will individually queue 100 tasks to the worker cluster, which will save their results in the cache backend for speed.
+Once all the 100 results are in the cache, they are collated into a list and saved as a single result in the database. The cache results are then cleared.
+Needs the Django cache framework.
+
 .. _groups:
 
 Groups
@@ -97,14 +125,16 @@ You can group together results by passing :func:`async` the optional ``group`` k
     for i in range(4):
         async('math.modf', i, group='modf')
 
-    # after the tasks have finished you can get the group results
-    result = result_group('modf')
+    # wait until the group has 4 results
+    result = result_group('modf', count=4)
     print(result)
 
 .. code-block:: python
 
     [(0.0, 0.0), (0.0, 1.0), (0.0, 2.0), (0.0, 3.0)]
 
+Note that the same can be achieved much faster with :func:`async_iter`
+
 Take care to not limit your results database too much and call :func:`delete_group` before each run, unless you want your results to keep adding up.
 Instead of :func:`result_group` you can also use :func:`fetch_group` to return a queryset of :class:`Task` objects.:
 
@@ -149,6 +179,54 @@ You can also access group functions from a task result instance:
         task.group_delete()
         print('Deleted group {}'.format(task.group))
 
+Cached operations
+-----------------
+You can run your tasks results against the Django cache backend instead of the database backend by either using the global :ref:`cached` setting or by supplying the ``cached`` keyword to individual functions.
+This can be useful if you are not interested in persistent results or if you run large group tasks where you only want the final result.
+By using a cache backend like Redis or Memcached you can speed up access to your task results significantly compared to a relational database.
+
+When you set ``cached=True``, results will be saved permanently in the cache and you will have to rely on your backend's cleanup strategies (like LRU) to
+manage stale results.
+You can also opt to set a manual timeout on the results, by setting ``cached=60``. Meaning the result will be evicted from the cache after 60 seconds.
+This works both globally or on individual async executions.::
+
+    # simple cached example
+    from django_q.tasks import async, result
+
+    # cache the result for 10 seconds
+    id = async('math.floor', 100, cached=10)
+
+    # wait max 50ms for the result to appear in the cache
+    result(id, wait=50, cached=True)
+
+    # o fetch the task object
+    task = fetch(id, cache=True)
+
+    # and then save it to the database
+    task.save()
+
+This also works for group actions::
+
+    # cached group example
+    from django_q.tasks import async, result_group
+    from django_q.brokers import get_broker
+
+    # set up a broker instance for better performance
+    broker = get_broker()
+
+    # async a hundred functions under a group label
+    for i in range(100):
+        async('math.frexp',
+              i,
+              group='frexp',
+              cached=True,
+              broker=broker)
+
+    # wait max 50ms for one hundred results to return
+    result_group('frexp', wait=50, count=100, cached=True)
+
+Note that exact same result can be achieved by using the more convenient :func:`async_iter` in this case, but without hook support.
+
 Synchronous testing
 -------------------
 
@@ -199,7 +277,7 @@ Reference
 ---------
 
 .. py:function:: async(func, *args, hook=None, group=None, timeout=None,\
-    save=None, sync=False, broker=None, q_options=None, **kwargs)
+    save=None, sync=False, cached=False, broker=None, q_options=None, **kwargs)
 
     Puts a task in the cluster queue
 
@@ -210,26 +288,29 @@ Reference
    :param int timeout: Overrides global cluster :ref:`timeout`.
    :param bool save: Overrides global save setting for this task.
    :param bool sync: If set to True, async will simulate a task execution
+   :param cached: Output the result to the cache backend. Bool or timeout in seconds
    :param broker: Optional broker connection from :func:`brokers.get_broker`
    :param dict q_options: Options dict, overrides option keywords
    :param dict kwargs: Keyword arguments for the task function
    :returns: The uuid of the task
    :rtype: str
 
-.. py:function:: result(task_id, wait=0)
+.. py:function:: result(task_id, wait=0, cached=False)
 
     Gets the result of a previously executed task
 
     :param str task_id: the uuid or name of the task
     :param int wait: optional milliseconds to wait for a result
+    :param bool cached: run this against the cache backend.
     :returns: The result of the executed task
 
-.. py:function:: fetch(task_id, wait=0)
+.. py:function:: fetch(task_id, wait=0, cached=False)
 
     Returns a previously executed task
 
     :param str name: the uuid or name of the task
     :param int wait: optional milliseconds to wait for a result
+    :param bool cached: run this against the cache backend.
     :returns: A task object
     :rtype: Task
 
@@ -237,6 +318,16 @@ Reference
 
     Renamed from get_task
 
+.. py:function:: async_iter(func, args_iter,**kwargs)
+
+   Runs iterable arguments against the cache backend and returns a single collated result
+
+   :param object func: The task function to execute
+   :param args: An iterable containing arguments for the task function
+   :param dict kwargs: Keyword arguments for the task function. Ignores ``cached`` and ``hook``.
+   :returns: The uuid of the task
+   :rtype: str
+
 .. py:function:: queue_size()
 
     Returns the size of the broker queue.
@@ -245,42 +336,58 @@ Reference
     :returns: The amount of task packages in the broker
     :rtype: int
 
-.. py:function:: result_group(group_id, failures=False)
+.. py:function:: result_group(group_id, failures=False, wait=0, count=None, cached=False)
 
     Returns the results of a task group
 
     :param str group_id: the group identifier
     :param bool failures: set this to ``True`` to include failed results
+    :param int wait: optional milliseconds to wait for a result or count
+    :param int count: block until there are this many results in the group
+    :param bool cached: run this against the cache backend
     :returns: a list of results
     :rtype: list
 
-.. py:function:: fetch_group(group_id, failures=True)
+.. py:function:: fetch_group(group_id, failures=True, wait=0, count=None, cached=False)
 
     Returns a list of tasks in a group
 
     :param str group_id: the group identifier
     :param bool failures: set this to ``False`` to exclude failed tasks
-    :returns: a list of Tasks
+    :param int wait: optional milliseconds to wait for a task or count
+    :param int count: block until there are this many tasks in the group
+    :param bool cached: run this against the cache backend.
+    :returns: a list of :class:`Task`
     :rtype: list
 
-.. py:function:: count_group(group_id, failures=False)
+.. py:function:: count_group(group_id, failures=False, cached=False)
 
     Counts the number of task results in a group.
 
     :param str group_id: the group identifier
     :param bool failures: counts the number of failures if ``True``
+    :param bool cached: run this against the cache backend.
     :returns: the number of tasks or failures in a group
     :rtype: int
 
-.. py:function:: delete_group(group_id, tasks=False)
+.. py:function:: delete_group(group_id, tasks=False, cached=False)
 
     Deletes a group label from the database.
 
     :param str group_id: the group identifier
     :param bool tasks: also deletes the associated tasks if ``True``
+    :param bool cached: run this against the cache backend.
     :returns: the numbers of tasks affected
     :rtype: int
 
+.. py:function:: delete_cached(task_id, broker=None)
+
+    Deletes a task from the cache backend
+
+    :param task_id: the uuid of the task
+    :param broker: an optional broker instance
+
+
 .. py:class:: Task
 
     Database model describing an executed task