Ilan Steemers
c17468ef86
docs: Adds Custom broker class documentation
...
Also changes some of the variable and settings names, to make them less ambiguous.
2016-01-27 12:06:49 +01:00
Ilan Steemers
8298f2319f
Fixes issue with rollbar configuration
2016-01-27 10:32:48 +01:00
Ilan Steemers
af3b1da08f
Makes DB polling time configurable via poll setting
2016-01-26 22:29:00 +01:00
Ilan Steemers
91bb1c69c7
Fixes double log output
2016-01-08 13:37:58 +01:00
Ilan Steemers
d29c861d09
Adds Rollbar support for exceptions
...
Also formats exceptions in case they are unprintable
2016-01-07 18:50:55 +01:00
Ilan Steemers
33282cb2e5
cached result backend
...
first version of a result backend using django's cache framework
2015-10-01 14:58:28 +02:00
Ilan Steemers
b8bb186471
check for default database on mongo connection
2015-09-26 13:01:15 +02:00
Ilan Steemers
826178c463
Adds a MongoDB broker
...
* adds mongo broker tests
* adds mongo brokers docs
* broker info is now cached when it needs to connect to a server, to reduce traffic.
2015-09-25 20:03:39 +02:00
Ilan Steemers
2bffcf554b
Django ORM broker
2015-09-14 13:24:43 +02:00
Ilan Steemers
8543c5b328
Adds Fastack option to Disque broker
2015-09-11 17:58:16 +02:00
Ilan Steemers
d7ce945f03
Amazon SQS broker
2015-09-09 18:10:03 +02:00
Ilan Steemers
83548dcfbb
Makes bulk a general setting
2015-09-08 13:42:15 +02:00
Ilan Steemers
0ffae4519b
Adds bulk get to ironmq
2015-09-08 13:37:33 +02:00
Ilan Steemers
b56cc9da8f
IronMQ broker
2015-09-08 11:20:31 +02:00
Ilan Steemers
73d15c3608
docs: minor updates
2015-09-06 16:54:20 +02:00
Ilan Steemers
b201979eae
defaults queue_limit to workers squared.
2015-09-06 15:45:17 +02:00
Ilan Steemers
3f75e7fa05
removed SQS and IronMQ brokers for now
...
Both brokers are not atomic and cause all kinds of problems during testing. Needs more time.
Meanwhile I want to release the pluggable broker backend.
2015-09-04 14:49:17 +02:00
Ilan Steemers
c626c3137c
Defaults RETRY to 60
2015-09-03 14:40:32 +02:00
Ilan Steemers
b1f2143a3d
uses disque_nodes convention
2015-09-03 12:55:49 +02:00
Ilan Steemers
8ce441bed5
Adds more brokers
...
* added IronMQ
* added Aws SQS
* removed SafeRedis.
During tests the 'safe' part wasn't consistent enough
2015-09-02 19:04:03 +02:00
Ilan Steemers
4f39062785
Adds django cache as monitor cache
...
Not the best solution yet, but it solves monitoring problems with brokers that don't have a pattern getter for cluster stats.
2015-08-30 18:28:26 +02:00
Ilan Steemers
d3283571a6
First draft of a Disque broker
...
TODO monitoring doesn't work yet
2015-08-30 14:33:35 +02:00
Ilan Steemers
0740c90148
adds pluggable brokers
...
* added redis broker
* added django_redis broker
* added task acknowledgement
2015-08-29 18:54:07 +02:00
Ilan Steemers
b414b637eb
#50 adds psutil as alternative os.getppid provider
2015-08-23 21:19:35 +02:00
Ilan
da5e265333
Updated comment on catch up
2015-08-19 12:39:56 +02:00
Ilan
b99e16f108
adds catch_up configuration option
...
When a cluster hasn't run for a while, the default behavior is to keep executing schedules until they are caught up with the present time.
By setting `catch_up` to False, schedules will not play catch up and instead get rescheduled to the future.
2015-08-18 11:57:15 +02:00
Ilan
6c37b09151
adds global sync configuration option
...
Setting the `sync` configuration option will force all async() calls to be run with `sync=True`. Useful for testing.
2015-08-13 18:02:38 +02:00
Ilan
ae8745ab39
Checks if cpu_count is implemented
...
On rare occasions, usually on win32, cpu_count will raise a NotImplemented error. If psutil is installed we will use this to get the correct count, otherwise we default to 4
2015-08-04 16:28:19 +02:00
Ilan
4bae1fc99f
#40 typo
2015-08-04 12:59:24 +02:00
Ilan
af61270f62
#40 qsize not implemented
2015-08-04 12:26:23 +02:00
Ilan
d31e9e2b97
adds a limit to the queue
2015-07-24 01:04:58 +02:00
Ilan
a8b562bc04
some minor linting
2015-07-22 17:13:29 +02:00
Ilan
9857f33e3e
Workaround for Travis
...
fixed typo
2015-07-15 22:34:40 +02:00
Ilan
a5580c75a1
Workaround for Travis
...
Travis doesn't support cpu affinity in their virtuals so I've added a setting TESTING that will bypass the actual affinity setting, but will still cover most of the code.
Might be useful for future workarounds.
2015-07-15 22:17:13 +02:00
Ilan
2d0820e80b
Adds cpu affinity for the workers. Courtesy of the optional psutil module.
2015-07-15 19:53:13 +02:00
Ilan Steemers
e590628612
linting some blank lines
2015-07-10 12:14:57 +02:00
Ilan Steemers
4ee47fc38f
upping the coverage
2015-07-07 20:20:28 +02:00
Ilan Steemers
831427bcc0
Upped version to 0.2.1
...
adds support for django-redis connections
2015-07-06 13:52:03 +02:00
Ilan Steemers
b113a52825
Restructured all the modules
...
The core module was becoming too large.
By splitting it in cluster, monitor and tasks it becomes more manageable. It also reflects the documentation structure.
2015-07-03 15:12:11 +02:00
Ilan Steemers
4e38925cea
Adds timeout mechanism for workers
...
Workers now share a ctype with the sentinel to indicate if they are currently executing a task. This timer is incremented each guard loop until the worker resets it after finishing a job or until it reaches the TIMEOUT value and the worker is terminated by the sentinel.
2015-07-01 10:54:36 +02:00
Ilan Steemers
c4b80c4b65
Removed old version id
2015-06-30 11:41:08 +02:00
Ilan Steemers
482569b312
Rework - configuration now in class - better stop handling under heavy load. - scheduler test - better monitoring - better status reporting
2015-06-30 10:58:24 +02:00
Ilan Steemers
218913e626
Default recycle to 500
2015-06-28 17:12:30 +02:00
Ilan Steemers
d0b950f9d6
Configuration is now through a dict.
2015-06-28 14:26:55 +02:00
Ilan Steemers
d8020f0c0b
Redis server now configurable with Q_REDIS. Moved constants to conf
2015-06-27 12:08:33 +02:00
Ilan Steemers
0ed1da8f05
Refactoring to make testing more stable
2015-06-23 21:12:08 +02:00
Ilan Steemers
5adf92eb27
hooks now work
...
refactored task as dict instead of list for readability.
2015-06-18 15:23:21 +02:00
Ilan Steemers