Commit Graph
28 Commits
Author SHA1 Message Date
Ilan Steemers 8ce441bed5 Adds more brokers
* added IronMQ
* added Aws SQS
* removed SafeRedis.
 During tests the 'safe' part wasn't consistent enough
2015-09-02 19:04:03 +02:00
Ilan Steemers 4f39062785 Adds django cache as monitor cache
Not the best solution yet, but it solves monitoring problems with brokers that don't have a pattern getter for cluster stats.
2015-08-30 18:28:26 +02:00
Ilan Steemers d3283571a6 First draft of a Disque broker
TODO monitoring doesn't work yet
2015-08-30 14:33:35 +02:00
Ilan Steemers 0740c90148 adds pluggable brokers
* added redis broker
 * added django_redis broker
 * added task acknowledgement
2015-08-29 18:54:07 +02:00
Ilan Steemers b414b637eb #50 adds psutil as alternative os.getppid provider 2015-08-23 21:19:35 +02:00
Ilan da5e265333 Updated comment on catch up 2015-08-19 12:39:56 +02:00
Ilan b99e16f108 adds catch_up configuration option
When a cluster hasn't run for a while, the default behavior is to keep executing schedules until they are caught up with the present time.
By setting `catch_up` to False, schedules will not play catch up and instead get rescheduled to the future.
2015-08-18 11:57:15 +02:00
Ilan 6c37b09151 adds global sync configuration option
Setting the `sync` configuration option will force all async() calls to be run with `sync=True`. Useful for testing.
2015-08-13 18:02:38 +02:00
Ilan ae8745ab39 Checks if cpu_count is implemented
On rare occasions, usually on win32, cpu_count will raise a NotImplemented error.  If psutil is installed we will use this to get the correct count, otherwise we default to 4
2015-08-04 16:28:19 +02:00
Ilan 4bae1fc99f #40 typo 2015-08-04 12:59:24 +02:00
Ilan af61270f62 #40 qsize not implemented 2015-08-04 12:26:23 +02:00
Ilan d31e9e2b97 adds a limit to the queue 2015-07-24 01:04:58 +02:00
Ilan a8b562bc04 some minor linting 2015-07-22 17:13:29 +02:00
Ilan 9857f33e3e Workaround for Travis
fixed typo
2015-07-15 22:34:40 +02:00
Ilan a5580c75a1 Workaround for Travis
Travis doesn't support cpu affinity in their virtuals so I've added a setting TESTING that will bypass the actual affinity setting, but will still cover most of the code.
Might be useful for future workarounds.
2015-07-15 22:17:13 +02:00
Ilan 2d0820e80b Adds cpu affinity for the workers. Courtesy of the optional psutil module. 2015-07-15 19:53:13 +02:00
Ilan Steemers e590628612 linting some blank lines 2015-07-10 12:14:57 +02:00
Ilan Steemers 4ee47fc38f upping the coverage 2015-07-07 20:20:28 +02:00
Ilan Steemers 831427bcc0 Upped version to 0.2.1
adds support for django-redis connections
2015-07-06 13:52:03 +02:00
Ilan Steemers b113a52825 Restructured all the modules
The core module was becoming too large.
By splitting it in  cluster, monitor and tasks it becomes more manageable. It also reflects the documentation structure.
2015-07-03 15:12:11 +02:00
Ilan Steemers 4e38925cea Adds timeout mechanism for workers
Workers now share a ctype with the sentinel to indicate if they are currently executing a task.  This timer is incremented each guard loop until the worker resets it after finishing a job or until it reaches the TIMEOUT value and the worker is terminated by the sentinel.
2015-07-01 10:54:36 +02:00
Ilan Steemers c4b80c4b65 Removed old version id 2015-06-30 11:41:08 +02:00
Ilan Steemers 482569b312 Rework - configuration now in class - better stop handling under heavy load. - scheduler test - better monitoring - better status reporting 2015-06-30 10:58:24 +02:00
Ilan Steemers 218913e626 Default recycle to 500 2015-06-28 17:12:30 +02:00
Ilan Steemers d0b950f9d6 Configuration is now through a dict. 2015-06-28 14:26:55 +02:00
Ilan Steemers d8020f0c0b Redis server now configurable with Q_REDIS. Moved constants to conf 2015-06-27 12:08:33 +02:00
Ilan Steemers 0ed1da8f05 Refactoring to make testing more stable 2015-06-23 21:12:08 +02:00
Ilan Steemers 5adf92eb27 hooks now work
refactored task as dict instead of list for readability.
2015-06-18 15:23:21 +02:00