Commit Graph

68 Commits

Author SHA1 Message Date
sinowood
52c04217e9 Multiple queue, multiple cluster in one site (#71)
* Support for multi-queue, multi-cluster configuration. API changes include:
  * Adding `cluster` to async_task() parameters and Task model
  * Adding argument --name to qcluster command
  * Necessary adjustments to Conf and Broker classes
  * Some admin improvements

* Add settings.Q_CLUSTER['ALT_CLUSTERS']: q_cluster config overrides for alternative clusters;
Add Conf.CLUSTER_NAME: separate usage from Conf.PREFIX;
QueueAdmin/OrmQ detail page enhanced: now displaying args/kwargs/q_options instead of encrypted payload.

* if `cluster` argument is not set (the default), async_task() and schedule() will be handled by the default cluster; Documentation update.

* Text cleanup

* Documentation update.

* Fix TIMEOUT setting in Windows for non-default cluster

---------

Co-authored-by: Stan Triepels <1939656+GDay@users.noreply.github.com>
2023-04-02 16:36:17 +02:00
msabatier
a013591de5 Add intended_date_kwarg field to Schedule (#62)
Co-authored-by: Marc Sabatier <marc.sabatier@eco-adapt.com>
2023-01-17 22:52:11 +01:00
Stan Triepels
3c0c32758d Chore: Flake8, isort and Black (#40) 2022-11-16 01:45:40 +01:00
Ilan Steemers
15155c7a99 Build improvements (#569)
* Black linting

* Adding Black to dev dependencies and upping minimal python to 3.6.2 for compatibility

* Updating packages

* Removing pip-tools input and exporting requirements with poetry

* Deleting old setup files and test runner

* Trying 1.3.7

* Looser extras requirements to prevent conflicts

* Sorted imports with isort

* Added iSort to dev dependencies

* Fixes localtime for naive setups
2021-05-30 17:10:07 +02:00
Dimitri
957f807598 Allow tasks to be scheduled on a specific cluster (#555)
* Allows schedule by cluster

* Update docs

* Add example in doc

* Fixes cluster field must be blank

* Adds cluster arg to schedule func

* Adds cluster field in list_display/filter for admin

* Uses Q filter to fetch schedule

* Fixes ref in doc

* Adds tests
2021-05-14 13:42:54 +02:00
Ilan Steemers
7d03536e89 Migrate to Github Action CI (#507)
* Some small refactors

* Initial test script

* Fixes django matrix

* Trying to set up Disque

* Disque as a docker service

* lowercase action name

* Remove Travis

* Replaces Travis badge with Github actions badge

* Show django version in step

* Typo
2021-02-14 18:59:04 +01:00
Ilan Steemers
532199fff3 Trigger cron validation 2020-07-01 14:18:04 +02:00
Ilan Steemers
5fe5b5fe37 Adds cron kwarg 2020-06-29 11:38:50 +02:00
Bastien Vallet
3569d2ea8c [cleanup] Fix some linting issues 2020-06-10 12:00:55 +02:00
Bastien Vallet
68d95a2924 [cleanup] Remove python2 artifacts 2020-06-10 11:48:31 +02:00
Ilan
0bca3cbd7a Replacing format with f-strings 2020-02-17 16:59:12 +01:00
Ilan Steemers
c27938ee62 Merge pull request #333 from tremby/fix-timeout-override
Fix timeout override
2019-08-13 14:16:03 +02:00
Chris Lamb
5c8243903f Inline import to prevent circular imports under some toolchain combinations. (Closes: #331) 2019-04-26 09:19:21 +01:00
Bart Nagel
4c2a2f3279 Fix timeout override
Closes https://github.com/Koed00/django-q/issues/332
2019-01-21 16:23:45 -08:00
Ilan Steemers
e0824802ca Use isinstance instead of typeof 2018-08-05 15:12:18 +02:00
Ilan Steemers
83d90664fb Frees resources on sync run - @abompard 2018-08-05 11:12:14 +02:00
Ilan Steemers
acebdaa850 Renames async/enqueue to async_task. 2018-08-01 15:20:34 +02:00
Pierre-Elliott Bécue
ae75a84f7f Replaces async occurrences with alternatives
* async is now a reserved word in python3.7
 * Rename async function to enqueue
 * Rename all async_ functions to enqueue_
 * Rename Async class to AsyncTask
 * Updates the docs.
2018-07-07 21:33:43 +02:00
Ilan Steemers
d5e430be44 Merge pull request #298 from Balletie/master
Add option for acknowledging failed tasks (globally and per-task)
2018-03-13 15:24:31 +01:00
Eagllus
430c177b92 Move import to the function that uses it
Only Python 2.x is having problems with this import so for now
the fix would be to move the import to the function that uses cluster
2018-03-13 09:37:13 +01:00
Eagllus
8f87bc352f Change imports for cluster and tasks 2018-03-13 09:28:17 +01:00
Eagllus
9c39546417 Change imports signing classes explicit 2018-03-13 09:22:09 +01:00
Balletie
03abbc960f Add option for acknowledging failed tasks (globally and per-task)
If a task fails with an exception, it is retried until it
succeeds. This is contrary to what is said in the documentation: under
the "Architecture" section, heading "Broker" it says that even when a
task errors, it's still considered a successful delivery. Failed tasks
never get acknowledged however, thereby being retried after the
timeout period. See also issues #238 and #194.

This patch adds an option to acknowledge failures, thereby closing
issue #238. Issue #194 would require some more work. The default of
this option is set to `False`, thereby maintaining backwards
compatibility.
2018-03-09 22:09:46 +01:00
Ronald van Zon
dd4f02f122 Add custom SharedCounter / Queue
This will not fix anything related to Django 2.0 but will allow MAC
users to run Django-Q allowing tests to be run locally.
2017-12-13 14:36:12 +01:00
Ilan Steemers
f3ae8a86dc Merge pull request #148 from xoxzo/log-enqueue-id
Log id returned by broker
2017-04-05 16:31:55 +03:00
Aurélien Bompard
77c5bb0737 Add signals
Fixes #219
2017-03-03 14:25:04 +01:00
Ilan Steemers
d4e0646097 Also fetches name override from q_options 2016-02-28 15:11:48 +01:00
Ilan Steemers
1e5509de45 Also fetches name override from q_options 2016-02-28 15:05:50 +01:00
Ilan Steemers
397503f5b2 Adds optional task_name argument 2016-02-24 10:52:25 +01:00
Ilan Steemers
0a8f2e7e72 Adds optional task_name argument
Makes it possible to name the task yourself.
2016-02-24 10:44:47 +01:00
k4ml
cb459cd0a1 Log id returned by broker
This will make it easy to inspect broker directly for particular message.
2016-02-19 17:01:42 +09:00
Ilan Steemers
d398685345 Adds a check for duplicate schedule names.
This should prevent the accidental repeated creation of schedules.
2015-11-19 17:57:31 +01:00
Ilan Steemers
3cc4021aad Optimizes option iteration for async
Simplified option iteration to only use a single loop. This makes the async command about twice as fast in optimal conditions.
2015-10-27 10:29:35 +01:00
Ilan Steemers
12bfd47917 Splits options from keywords through iteration
Popping options from async kwargs will make any reference to them void. Which can lead to unwanted situations. Unfortunately deepcopy clashes with the multiprocessing so we resort to simple re-iteration.
2015-10-26 15:33:42 +01:00
Ilan Steemers
728ba45ef0 Adds Async class wrapper for the async function 2015-10-26 13:48:23 +01:00
Ilan Steemers
62337a9d06 Adds infinite wait to task result functions
* the `wait` keyword now can take the value `-1` which will make the functions wait indefinitely for a result.
2015-10-25 16:40:49 +01:00
Ilan Steemers
7eefb8cb64 returns the size of the chain or iter on append 2015-10-18 15:57:23 +02:00
Ilan Steemers
eb72eb33d9 Adds an Iter class
The Iter class serves as a convenience rwrapper around the `async_iter` function
2015-10-18 13:53:32 +02:00
Ilan Steemers
083715f9bd Replaces list.copy with a slice for python 2.7 2015-10-17 18:59:17 +02:00
Ilan Steemers
2df29b908c Adds tests for task chains
adds several small improvements based on the problems that surfaced during writing the tests
2015-10-17 18:49:45 +02:00
Ilan Steemers
6dcead7310 Adds task chains 2015-10-17 12:13:29 +02:00
Ilan Steemers
2ba88f2d17 adds cached option to async_iter 2015-10-06 19:37:02 +02:00
Ilan Steemers
f73ae66c0f Adds async_iter command
With async iter you can quickly run the same function on an iterable set of arguments. The results are held in the cache until all are done and collated into a database result.
2015-10-03 18:07:57 +02:00
Ilan Steemers
81d5abbd00 adds count and wait option to group result and fetch
result_group and fetch_group can now be told to block for a certain number of results or a number of miliseconds or a combination of both.
2015-10-02 18:29:03 +02:00
Ilan Steemers
9cfc41f1cb fixed global cached for fetch group
And added docstrings
2015-10-02 16:40:31 +02:00
Ilan Steemers
f6d14bc7b3 adds delete_cached 2015-10-01 19:25:43 +02:00
Ilan Steemers
6127b551ae fixes tests for cache with better coverage 2015-10-01 17:21:26 +02:00
Ilan Steemers
7107e9ea35 fetch group should default to non cached 2015-10-01 15:06:43 +02:00
Ilan Steemers
33282cb2e5 cached result backend
first version of a result backend using django's cache framework
2015-10-01 14:58:28 +02:00
Ilan Steemers
a493f9cc78 Adds wait option to result() and fetch() 2015-09-18 10:57:09 +02:00