Renames async/enqueue to async_task.

This commit is contained in:
Ilan Steemers
2018-08-01 15:20:34 +02:00
parent 887870bcdb
commit acebdaa850
17 changed files with 154 additions and 154 deletions
+1 -1
View File
@@ -140,7 +140,7 @@ You can override this class if you want to contribute and support your own broke
.. py:class:: Broker
.. py:method:: enqueue(task)
.. py:method:: async_task(task)
Sends a task package to the broker queue and returns a tracking id if available.
+9 -9
View File
@@ -2,16 +2,16 @@
Chains
======
Sometimes you want to run tasks sequentially. For that you can use the :func:`enqueue_chain` function:
Sometimes you want to run tasks sequentially. For that you can use the :func:`async_chain` function:
.. code-block:: python
# enqueue a chain of tasks
from django_q.tasks import enqueue_chain, result_group
# async a chain of tasks
from django_q.tasks import async_chain, result_group
# the chain must be in the format
# [(func,(args),{kwargs}),(func,(args),{kwargs}),..]
group_id = enqueue_chain([('math.copysign', (1, -1)),
group_id = async_chain([('math.copysign', (1, -1)),
('math.floor', (1,))])
# get group result
@@ -21,7 +21,7 @@ A slightly more convenient way is to use a :class:`Chain` instance:
.. code-block:: python
# Chain enqueue
# Chain async
from django_q.tasks import Chain
# create a chain that uses the cache backend
@@ -41,9 +41,9 @@ A slightly more convenient way is to use a :class:`Chain` instance:
Reference
---------
.. py:function:: enqueue_chain(chain, group=None, cached=Conf.CACHED, sync=Conf.SYNC, broker=None)
.. py:function:: async_chain(chain, group=None, cached=Conf.CACHED, sync=Conf.SYNC, broker=None)
enqueue a chain of tasks. See also the :class:`Chain` class.
Async a chain of tasks. See also the :class:`Chain` class.
:param list chain: a list of tasks in the format [(func,(args),{kwargs}), (func,(args),{kwargs})]
:param str group: an optional group name.
@@ -52,7 +52,7 @@ Reference
.. py:class:: Chain(chain=None, group=None, cached=Conf.CACHED, sync=Conf.SYNC)
A sequential chain of tasks. Acts as a convenient wrapper for :func:`enqueue_chain`
A sequential chain of tasks. Acts as a convenient wrapper for :func:`async_chain`
You can pass the task chain at construction or you can append individual tasks before running them.
:param list chain: a list of task in the format [(func,(args),{kwargs}), (func,(args),{kwargs})]
@@ -63,7 +63,7 @@ Reference
.. py:method:: append(func, *args, **kwargs)
Append a task to the chain. Takes the same arguments as :func:`enqueue`
Append a task to the chain. Takes the same arguments as :func:`async_task`
:return: the current number of tasks in the chain
:rtype: int
+2 -2
View File
@@ -64,7 +64,7 @@ Set this to something that makes sense for your project. Can be overridden for i
ack_failures
~~~~~~~~~~~~
When set to ``True``, also acknowledge unsuccessful tasks. This causes failed tasks to be considered as successful deliveries, thereby removing them from the task queue. Can also be set per-task by passing the ``ack_failure`` option to :func:`enqueue`. Defaults to ``False``.
When set to ``True``, also acknowledge unsuccessful tasks. This causes failed tasks to be considered as successful deliveries, thereby removing them from the task queue. Can also be set per-task by passing the ``ack_failure`` option to :func:`async_task`. Defaults to ``False``.
.. _retry:
@@ -101,7 +101,7 @@ Guard loop sleep in seconds, must be greater than 0 and less than 60.
sync
~~~~
When set to ``True`` this configuration option forces all :func:`enqueue` calls to be run with ``sync=True``.
When set to ``True`` this configuration option forces all :func:`async_task` calls to be run with ``sync=True``.
Effectively making everything synchronous. Useful for testing. Defaults to ``False``.
.. _queue_limit:
+30 -30
View File
@@ -12,14 +12,14 @@ Sending an email can take a while so why not queue it:
# Welcome mail with follow up example
from datetime import timedelta
from django.utils import timezone
from django_q.tasks import enqueue, schedule
from django_q.tasks import async_task, schedule
from django_q.models import Schedule
def welcome_mail(user):
msg = 'Welcome to our website'
# send this message right away
enqueue('django.core.mail.send_mail',
async_task('django.core.mail.send_mail',
'Welcome',
msg,
'from@example.com',
@@ -51,7 +51,7 @@ A good place to use async tasks are Django's model signals. You don't want to de
from django.contrib.auth.models import User
from django.db.models.signals import pre_save
from django.dispatch import receiver
from django_q.tasks import enqueue
from django_q.tasks import async_task
# set up the pre_save signal for our user
@receiver(pre_save, sender=User)
@@ -64,7 +64,7 @@ A good place to use async tasks are Django's model signals. You don't want to de
# has his email changed?
if not user.email == instance.email:
# tell everyone
enqueue('tasks.inform_everyone', instance)
async_task('tasks.inform_everyone', instance)
The task will send a message to everyone else informing them that the users email address has changed. Note that this adds almost no overhead to the save action:
@@ -87,7 +87,7 @@ The task will send a message to everyone else informing them that the users emai
for u in User.objects.exclude(pk=user.pk):
msg = 'Dear {}, {} has a new email address: {}'
msg = msg.format(u.username, user.username, user.email)
enqueue('django.core.mail.send_mail',
async_task('django.core.mail.send_mail',
'New email', msg, 'from@example.com', [u.email])
@@ -104,19 +104,19 @@ In this example the user requests a report and we let the cluster do the generat
.. code-block:: python
# Report generation with hook example
from django_q.tasks import enqueue
from django_q.tasks import async_task
# views.py
# user requests a report.
def create_report(request):
enqueue('tasks.create_html_report',
async_task('tasks.create_html_report',
request.user,
hook='tasks.email_report')
.. code-block:: python
# tasks.py
from django_q.tasks import enqueue
from django_q.tasks import async_task
# report generator
def create_html_report(user):
@@ -127,14 +127,14 @@ In this example the user requests a report and we let the cluster do the generat
def email_report(task):
if task.success:
# Email the report
enqueue('django.core.mail.send_mail',
async_task('django.core.mail.send_mail',
'The report you requested',
task.result,
'from@example.com',
task.args[0].email)
else:
# Tell the admins something went wrong
enqueue('django.core.mail.mail_admins',
async_task('django.core.mail.mail_admins',
'Report generation failed',
task.result)
@@ -152,12 +152,12 @@ here's an example of how you can have Django Q take care of your indexes in real
from .models import Document
from django.db.models.signals import post_save
from django.dispatch import receiver
from django_q.tasks import enqueue
from django_q.tasks import async_task
# hook up the post save handler
@receiver(post_save, sender=Document)
def document_changed(sender, instance, **kwargs):
enqueue('tasks.index_object', sender, instance, save=False)
async_task('tasks.index_object', sender, instance, save=False)
# turn off result saving to not flood your database
.. code-block:: python
@@ -177,7 +177,7 @@ here's an example of how you can have Django Q take care of your indexes in real
index.update_object(instance, using=backend)
Now every time a Document is saved, your indexes will be updated without causing a delay in your save action.
You could expand this to dealing with deletes, by adding a ``post_delete`` signal and calling ``index.remove_object`` in the enqueue function.
You could expand this to dealing with deletes, by adding a ``post_delete`` signal and calling ``index.remove_object`` in the async_task function.
.. _shell:
@@ -187,13 +187,13 @@ You can execute or schedule shell commands using Pythons :mod:`subprocess` modul
.. code-block:: python
from django_q.tasks import enqueue, result
from django_q.tasks import async_task, result
# make a backup copy of setup.py
enqueue('subprocess.call', ['cp', 'setup.py', 'setup.py.bak'])
async_task('subprocess.call', ['cp', 'setup.py', 'setup.py.bak'])
# call ls -l and dump the output
task_id=enqueue('subprocess.check_output', ['ls', '-l'])
task_id=async_task('subprocess.check_output', ['ls', '-l'])
# get the result
dir_list = result(task_id)
@@ -202,10 +202,10 @@ In Python 3.5 the subprocess module has changed quite a bit and returns a :class
.. code-block:: python
from django_q.tasks import enqueue, result
from django_q.tasks import async_task, result
# make a backup copy of setup.py
tid = enqueue('subprocess.run', ['cp', 'setup.py', 'setup.py.bak'])
tid = async_task('subprocess.run', ['cp', 'setup.py', 'setup.py.bak'])
# get the result
r=result(tid, 500)
@@ -220,22 +220,22 @@ In Python 3.5 the subprocess module has changed quite a bit and returns a :class
from subprocess import PIPE
# call ls -l and pipe the output
tid = enqueue('subprocess.run', ['ls', '-l'], stdout=PIPE)
tid = async_task('subprocess.run', ['ls', '-l'], stdout=PIPE)
# get the result
res = result(tid, 500)
# print the output
print(res.stdout)
Instead of :func:`enqueue` you can of course also use :func:`schedule` to schedule commands.
Instead of :func:`async_task` you can of course also use :func:`schedule` to schedule commands.
For regular Django management commands, it is easier to call them directly:
.. code-block:: python
from django_q.tasks import enqueue, schedule
from django_q.tasks import async_task, schedule
enqueue('django.core.management.call_command','clearsessions')
async_task('django.core.management.call_command','clearsessions')
# or clear those sessions every hour
@@ -255,7 +255,7 @@ Adapted from `Sebastian Raschka's blog <http://sebastianraschka.com/Articles/201
# Group example with Parzen-window estimation
import numpy
from django_q.tasks import enqueue, result_group, delete_group
from django_q.tasks import async_task, result_group, delete_group
# the estimation function
def parzen_estimation(x_samples, point_x, h):
@@ -270,7 +270,7 @@ Adapted from `Sebastian Raschka's blog <http://sebastianraschka.com/Articles/201
return h, (k_n / len(x_samples)) / (h ** point_x.shape[1])
# create 100 calculations and return the collated result
def parzen_enqueue():
def parzen_async():
# clear the previous results
delete_group('parzen', cached=True)
mu_vec = numpy.array([0, 0])
@@ -279,9 +279,9 @@ Adapted from `Sebastian Raschka's blog <http://sebastianraschka.com/Articles/201
multivariate_normal(mu_vec, cov_mat, 10000)
widths = numpy.linspace(1.0, 1.2, 100)
x = numpy.array([[0], [0]])
# enqueue them with a group label to the cache backend
# async_task them with a group label to the cache backend
for w in widths:
enqueue(parzen_estimation, sample, x, w,
async_task(parzen_estimation, sample, x, w,
group='parzen', cached=True)
# return after 100 results
return result_group('parzen', count=100, cached=True)
@@ -290,21 +290,21 @@ Adapted from `Sebastian Raschka's blog <http://sebastianraschka.com/Articles/201
Django Q is not optimized for distributed computing, but this example will give you an idea of what you can do with task :doc:`group`.
Alternatively the ``parzen_enqueue()`` function can also be written with :func:`enqueue_iter`, which automatically utilizes the cache backend and groups to return a single result from an iterable:
Alternatively the ``parzen_async()`` function can also be written with :func:`async_iter`, which automatically utilizes the cache backend and groups to return a single result from an iterable:
.. code-block:: python
# create 100 calculations and return the collated result
def parzen_enqueue():
def parzen_async():
mu_vec = numpy.array([0, 0])
cov_mat = numpy.array([[1, 0], [0, 1]])
sample = numpy.random. \
multivariate_normal(mu_vec, cov_mat, 10000)
widths = numpy.linspace(1.0, 1.2, 100)
x = numpy.array([[0], [0]])
# enqueue them with enqueue iterable
# async_task them with async_task iterable
args = [(sample, x, w) for w in widths]
result_id = enqueue_iter(parzen_estimation, args, cached=True)
result_id = async_iter(parzen_estimation, args, cached=True)
# return the cached result or timeout after 10 seconds
return result(result_id, wait=10000, cached=True)
+5 -5
View File
@@ -2,15 +2,15 @@
Groups
======
You can group together results by passing :func:`enqueue` the optional ``group`` keyword:
You can group together results by passing :func:`async_task` the optional ``group`` keyword:
.. code-block:: python
# result group example
from django_q.tasks import enqueue, result_group
from django_q.tasks import async_task, result_group
for i in range(4):
enqueue('math.modf', i, group='modf')
async_task('math.modf', i, group='modf')
# wait until the group has 4 results
result = result_group('modf', count=4)
@@ -70,10 +70,10 @@ or call them directly on :class:`AsyncTask` object:
.. code-block:: python
from django_q.tasks import enqueue
from django_q.tasks import async_task
# add a task to the math group and run it cached
a = enqueue('math.floor', 2.5, group='math', cached=True)
a = async_task('math.floor', 2.5, group='math', cached=True)
# wait until this tasks group has 10 results
result = a.result_group(count=10)
+7 -7
View File
@@ -2,16 +2,16 @@
Iterable
========
If you have an iterable object with arguments for a function, you can use :func:`enqueue_iter` to async them with a single command::
If you have an iterable object with arguments for a function, you can use :func:`async_iter` to async them with a single command::
# Async Iterable example
from django_q.tasks import enqueue_iter, result
from django_q.tasks import async_iter, result
# set up a list of arguments for math.floor
iter = [i for i in range(100)]
# enqueue iter them
id=enqueue_iter('math.floor',iter)
# async_task iter them
id=async_iter('math.floor',iter)
# wait for the collated result for 1 second
result_list = result(id, wait=1000)
@@ -45,10 +45,10 @@ You can also use an :class:`Iter` instance which can sometimes be more convenien
Reference
---------
.. py:function:: enqueue_iter(func, args_iter,**kwargs)
.. py:function:: async_iter(func, args_iter,**kwargs)
Runs iterable arguments against the cache backend and returns a single collated result.
Accepts the same options as :func:`enqueue` except ``hook``. See also the :class:`Iter` class.
Accepts the same options as :func:`async_task` except ``hook``. See also the :class:`Iter` class.
:param object func: The task function to execute
:param args: An iterable containing arguments for the task function
@@ -58,7 +58,7 @@ Reference
.. py:class:: Iter(func=None, args=None, kwargs=None, cached=Conf.CACHED, sync=Conf.SYNC, broker=None)
An async task with iterable arguments. Serves as a convenient wrapper for :func:`enqueue_iter`
An async task with iterable arguments. Serves as a convenient wrapper for :func:`async_iter`
You can pass the iterable arguments at construction or you can append individual argument tuples.
:param func: the function to execute
+1 -1
View File
@@ -103,7 +103,7 @@ Reference
:param int minutes: Number of minutes for the Minutes type.
:param int repeats: Number of times to repeat schedule. -1=Always, 0=Never, n =n.
:param datetime next_run: Next or first scheduled execution datetime.
:param dict q_options: options passed to enqueue for this schedule
:param dict q_options: options passed to async_task for this schedule
:param kwargs: optional keyword arguments for the scheduled function.
.. class:: Schedule
+28 -28
View File
@@ -4,22 +4,22 @@ Tasks
.. _async:
enqueue()
async_task()
---------
Use :func:`enqueue` from your code to quickly offload tasks to the :class:`Cluster`:
Use :func:`async_task` from your code to quickly offload tasks to the :class:`Cluster`:
.. code:: python
from django_q.tasks import enqueue, result
from django_q.tasks import async_task, result
# create the task
enqueue('math.copysign', 2, -2)
async_task('math.copysign', 2, -2)
# or with import and storing the id
import math.copysign
task_id = enqueue(copysign, 2, -2)
task_id = async_task(copysign, 2, -2)
# get the result
task_result = result(task_id)
@@ -30,13 +30,13 @@ Use :func:`enqueue` from your code to quickly offload tasks to the :class:`Clust
# but in most cases you will want to use a hook:
enqueue('math.modf', 2.5, hook='hooks.print_result')
async_task('math.modf', 2.5, hook='hooks.print_result')
# hooks.py
def print_result(task):
print(task.result)
:func:`enqueue` can take the following optional keyword arguments:
:func:`async_task` can take the following optional keyword arguments:
hook
""""
@@ -84,13 +84,13 @@ None of the option keywords get passed on to the task function.
As an alternative you can also put them in
a single keyword dict named ``q_options``. This enables you to use these keywords for your function call::
# Enqueue options in a dict
# Async options in a dict
opts = {'hook': 'hooks.print_result',
'group': 'math',
'timeout': 30}
enqueue('math.modf', 2.5, q_options=opts)
async_task('math.modf', 2.5, q_options=opts)
Please note that this will override any other option keywords.
@@ -150,10 +150,10 @@ You can also opt to set a manual timeout on the results, by setting e.g. ``cache
This works both globally or on individual async executions.::
# simple cached example
from django_q.tasks import enqueue, result
from django_q.tasks import async_task, result
# cache the result for 10 seconds
id = enqueue('math.floor', 100, cached=10)
id = async_task('math.floor', 100, cached=10)
# wait max 50ms for the result to appear in the cache
result(id, wait=50, cached=True)
@@ -169,15 +169,15 @@ As you can see you can easily turn a cached result into a permanent database res
This also works for group actions::
# cached group example
from django_q.tasks import enqueue, result_group
from django_q.tasks import async_task, result_group
from django_q.brokers import get_broker
# set up a broker instance for better performance
broker = get_broker()
# enqueue a hundred functions under a group label
# Async a hundred functions under a group label
for i in range(100):
enqueue('math.frexp',
async_task('math.frexp',
i,
group='frexp',
cached=True,
@@ -186,18 +186,18 @@ This also works for group actions::
# wait max 50ms for one hundred results to return
result_group('frexp', wait=50, count=100, cached=True)
If you don't need hooks, that exact same result can be achieved by using the more convenient :func:`enqueue_iter`.
If you don't need hooks, that exact same result can be achieved by using the more convenient :func:`async_iter`.
Synchronous testing
-------------------
:func:`enqueue` can be instructed to execute a task immediately by setting the optional keyword ``sync=True``.
:func:`async_task` can be instructed to execute a task immediately by setting the optional keyword ``sync=True``.
The task will then be injected straight into a worker and the result saved by a monitor instance::
from django_q.tasks import enqueue, fetch
from django_q.tasks import async_task, fetch
# create a synchronous task
task_id = enqueue('my.buggy.code', sync=True)
task_id = async_task('my.buggy.code', sync=True)
# the task will then be available immediately
task = fetch(task_id)
@@ -210,24 +210,24 @@ The task will then be injected straight into a worker and the result saved by a
An error occurred: ImportError("No module named 'my'",)
Note that :func:`enqueue` will block until the task is executed and saved. This feature bypasses the broker and is intended for debugging and development.
Instead of setting ``sync`` on each individual ``enqueue`` you can also configure :ref:`sync` as a global override.
Note that :func:`async_task` will block until the task is executed and saved. This feature bypasses the broker and is intended for debugging and development.
Instead of setting ``sync`` on each individual ``async_task`` you can also configure :ref:`sync` as a global override.
Connection pooling
------------------
Django Q tries to pass broker instances around its parts as much as possible to save you from running out of connections.
When you are making individual calls to :func:`enqueue` a lot though, it can help to set up a broker to reuse for :func:`enqueue`:
When you are making individual calls to :func:`async_task` a lot though, it can help to set up a broker to reuse for :func:`async_task`:
.. code:: python
# broker connection economy example
from django_q.tasks import enqueue
from django_q.tasks import async_task
from django_q.brokers import get_broker
broker = get_broker()
for i in range(50):
enqueue('math.modf', 2.5, broker=broker)
async_task('math.modf', 2.5, broker=broker)
.. tip::
@@ -237,7 +237,7 @@ When you are making individual calls to :func:`enqueue` a lot though, it can hel
Reference
---------
.. py:function:: enqueue(func, *args, hook=None, group=None, timeout=None,\
.. py:function:: async_task(func, *args, hook=None, group=None, timeout=None,\
save=None, sync=False, cached=False, broker=None, q_options=None, **kwargs)
Puts a task in the cluster queue
@@ -249,7 +249,7 @@ Reference
:param int timeout: Overrides global cluster :ref:`timeout`.
:param bool save: Overrides global save setting for this task.
:param bool ack_failure: Overrides the global :ref:`ack_failures` setting for this task.
:param bool sync: If set to True, enqueue will simulate a task execution
:param bool sync: If set to True, async_task will simulate a task execution
:param cached: Output the result to the cache backend. Bool or timeout in seconds
:param broker: Optional broker connection from :func:`brokers.get_broker`
:param dict q_options: Options dict, overrides option keywords
@@ -410,11 +410,11 @@ Reference
.. py:class:: AsyncTask(func, *args, **kwargs)
A class wrapper for the :func:`enqueue` function.
A class wrapper for the :func:`async_task` function.
:param object func: The task function to execute
:param tuple args: The arguments for the task function
:param dict kwargs: Keyword arguments for the task function, including enqueue options
:param dict kwargs: Keyword arguments for the task function, including async_task options
.. py:attribute:: id
@@ -434,7 +434,7 @@ Reference
.. py:attribute:: kwargs
Keyword arguments for the function. Can include any of the optional enqueue keyword attributes directly or in a `q_options` dictionary.
Keyword arguments for the function. Can include any of the optional async_task keyword attributes directly or in a `q_options` dictionary.
.. py:attribute:: broker