Ilan Steemers
b5a01a9e93
Removes old rollbar support - relies on the pluggable error reporters now
2018-08-10 13:03:58 +02:00
Ilan Steemers
7d5c9f8b46
removes xrange compat
2018-08-05 13:39:04 +02:00
Ilan Steemers
bd062503c5
Adds tracebacks in error logs @strets123
2018-08-05 11:13:16 +02:00
Ilan Steemers
acebdaa850
Renames async/enqueue to async_task.
2018-08-01 15:20:34 +02:00
Pierre-Elliott Bécue
ae75a84f7f
Replaces async occurrences with alternatives
...
* async is now a reserved word in python3.7
* Rename async function to enqueue
* Rename all async_ functions to enqueue_
* Rename Async class to AsyncTask
* Updates the docs.
2018-07-07 21:33:43 +02:00
Ilan Steemers
d5e430be44
Merge pull request #298 from Balletie/master
...
Add option for acknowledging failed tasks (globally and per-task)
2018-03-13 15:24:31 +01:00
Eagllus
8f87bc352f
Change imports for cluster and tasks
2018-03-13 09:28:17 +01:00
Eagllus
9c39546417
Change imports signing classes explicit
2018-03-13 09:22:09 +01:00
Balletie
03abbc960f
Add option for acknowledging failed tasks (globally and per-task)
...
If a task fails with an exception, it is retried until it
succeeds. This is contrary to what is said in the documentation: under
the "Architecture" section, heading "Broker" it says that even when a
task errors, it's still considered a successful delivery. Failed tasks
never get acknowledged however, thereby being retried after the
timeout period. See also issues #238 and #194 .
This patch adds an option to acknowledge failures, thereby closing
issue #238 . Issue #194 would require some more work. The default of
this option is set to `False`, thereby maintaining backwards
compatibility.
2018-03-09 22:09:46 +01:00
Ronald van Zon
dd4f02f122
Add custom SharedCounter / Queue
...
This will not fix anything related to Django 2.0 but will allow MAC
users to run Django-Q allowing tests to be run locally.
2017-12-13 14:36:12 +01:00
Daniel Welch
4530b05d86
putting back support for backwards compatibility
2017-10-06 13:26:43 -04:00
Daniel Welch
f90c4f5747
merge conflict resolution
2017-10-03 16:55:28 -04:00
Benjamin Bach
5ea55e3718
Import python3 version of range
2017-07-09 21:44:25 +02:00
Benjamin Bach
ab48bad0f6
Remove unused dependency and sys module patching
2017-07-09 21:14:08 +02:00
Daniel Welch
aac6984ec3
replace rollbar reporting with generic error_reporter reporting in cluster.py
2017-03-30 22:02:31 -04:00
Aurélien Bompard
77c5bb0737
Add signals
...
Fixes #219
2017-03-03 14:25:04 +01:00
Yann Pomarede
78b997e88f
fix typo
2016-12-05 14:58:42 +01:00
Yann Pomarede
aa667b40d4
daemonize_workers option
2016-12-05 14:55:42 +01:00
yannpom
3b8199cb8f
set daemon to False
2016-12-05 12:21:41 +01:00
Vladimir Rychkov
1a1385dfba
Guard loop sleep made configurable
2016-07-12 13:08:17 +01:00
Matt Bone
823ca0c51b
allow scheduler to schedule all pending tasks
2016-04-21 16:31:34 -05:00
Ilan Steemers
6026953132
Fixes typo in Save limit
2016-04-06 20:33:26 +02:00
Ilan Steemers
763fff34c0
Makes it possible to set a schedules group name via q_options dict
2016-02-24 11:22:58 +01:00
Ilan Steemers
2e965b1631
Closes db connections on reincarnate
2016-02-24 10:08:09 +01:00
Ilan Steemers
c66f0b74ed
Closes all db connections on worker recycle
2016-02-16 11:29:43 +01:00
Ilan Steemers
c75ab4e7f0
Only updates existing tasks if it was failing
...
updates stopped, result and success only for existing task results if the original result was failing. Otherwise the result is discarded.
2016-01-20 14:11:53 +01:00
Ilan Steemers
66771f274a
Save task now creates or updates
...
This way failed tasks that get requeued can have a changed status and result
2016-01-19 22:49:28 +01:00
Ilan Steemers
0e8ea25396
Only acks task if it's succesful
2016-01-18 10:51:28 +01:00
Ilan Steemers
d29c861d09
Adds Rollbar support for exceptions
...
Also formats exceptions in case they are unprintable
2016-01-07 18:50:55 +01:00
Ilan Steemers
02bd14c169
Fixes bug where task with a group only has a group result.
2015-10-26 19:35:07 +01:00
Ilan Steemers
69551793c0
adds timeout to group_key cache object
2015-10-19 15:42:05 +02:00
Ilan Steemers
2df29b908c
Adds tests for task chains
...
adds several small improvements based on the problems that surfaced during writing the tests
2015-10-17 18:49:45 +02:00
Ilan Steemers
6dcead7310
Adds task chains
2015-10-17 12:13:29 +02:00
Ilan Steemers
2ba88f2d17
adds cached option to async_iter
2015-10-06 19:37:02 +02:00
Ilan Steemers
f73ae66c0f
Adds async_iter command
...
With async iter you can quickly run the same function on an iterable set of arguments. The results are held in the cache until all are done and collated into a database result.
2015-10-03 18:07:57 +02:00
Ilan Steemers
33282cb2e5
cached result backend
...
first version of a result backend using django's cache framework
2015-10-01 14:58:28 +02:00
Ilan Steemers
1aed7c2d00
#79 close django db connection before fork
...
The sentinel saves a statistic to the cache provider just before forking the workers. If this cache provider is a database backend, the connection stays open and gets forked and it crashes. Solution is to close any db connection before forking.
2015-09-29 10:47:22 +02:00
Ilan Steemers
765c84301f
#76 checks for stale connections always
...
It turns out that checking stale connections on a timer takes between 1-2 times as long as just checking them always. This also has the benefit of catching timeouts that happen between timer loops.
2015-09-28 12:48:11 +02:00
Ilan Steemers
7075a91d37
#76 adds connection checked with timeout
...
Adds a check for old connections in both the workers and the monitor , every DB_TIMEOUT seconds
2015-09-28 11:42:48 +02:00
Ilan Steemers
4ceee42be9
#76 resets stale db connection on every scheduler ping
2015-09-28 10:42:08 +02:00
Ilan Steemers
131a37daf3
Removes Pycharm stdin bug workaround
2015-09-24 11:21:50 +02:00
Ilan Steemers
7fb1142e4e
Shares Sentinel broker with Monitor
2015-09-15 20:07:33 +02:00
Ilan Steemers
a060b3508a
Moves bulk iteration to pusher
2015-09-15 14:45:06 +02:00
Ilan Steemers
2bffcf554b
Django ORM broker
2015-09-14 13:24:43 +02:00
Ilan Steemers
cc178df2b1
#56 fakes setting cpu_affinity on unsupported platforms
2015-09-09 13:25:27 +02:00
Ilan Steemers
1ccfc66254
adds fail method to broker
2015-09-03 16:34:00 +02:00
Ilan Steemers
da20ebe734
fixes connection creation on import
2015-08-31 15:15:24 +02:00
Ilan Steemers
0740c90148
adds pluggable brokers
...
* added redis broker
* added django_redis broker
* added task acknowledgement
2015-08-29 18:54:07 +02:00
Ilan Steemers
439c8a2f77
Moves Status to it's own file
...
This way only qmonitor and qinfo need Curses and you can still run the cluster without it.
2015-08-26 13:11:58 +02:00
Ilan Steemers
3e92fddcdd
#50 removes redis connection argument from pusher process
2015-08-24 10:08:43 +02:00
Ilan Steemers