Ilan Steemers
6026953132
Fixes typo in Save limit
2016-04-06 20:33:26 +02:00
Ilan Steemers
763fff34c0
Makes it possible to set a schedules group name via q_options dict
2016-02-24 11:22:58 +01:00
Ilan Steemers
2e965b1631
Closes db connections on reincarnate
2016-02-24 10:08:09 +01:00
Ilan Steemers
c66f0b74ed
Closes all db connections on worker recycle
2016-02-16 11:29:43 +01:00
Ilan Steemers
c75ab4e7f0
Only updates existing tasks if it was failing
...
updates stopped, result and success only for existing task results if the original result was failing. Otherwise the result is discarded.
2016-01-20 14:11:53 +01:00
Ilan Steemers
66771f274a
Save task now creates or updates
...
This way failed tasks that get requeued can have a changed status and result
2016-01-19 22:49:28 +01:00
Ilan Steemers
0e8ea25396
Only acks task if it's succesful
2016-01-18 10:51:28 +01:00
Ilan Steemers
d29c861d09
Adds Rollbar support for exceptions
...
Also formats exceptions in case they are unprintable
2016-01-07 18:50:55 +01:00
Ilan Steemers
02bd14c169
Fixes bug where task with a group only has a group result.
2015-10-26 19:35:07 +01:00
Ilan Steemers
69551793c0
adds timeout to group_key cache object
2015-10-19 15:42:05 +02:00
Ilan Steemers
2df29b908c
Adds tests for task chains
...
adds several small improvements based on the problems that surfaced during writing the tests
2015-10-17 18:49:45 +02:00
Ilan Steemers
6dcead7310
Adds task chains
2015-10-17 12:13:29 +02:00
Ilan Steemers
2ba88f2d17
adds cached option to async_iter
2015-10-06 19:37:02 +02:00
Ilan Steemers
f73ae66c0f
Adds async_iter command
...
With async iter you can quickly run the same function on an iterable set of arguments. The results are held in the cache until all are done and collated into a database result.
2015-10-03 18:07:57 +02:00
Ilan Steemers
33282cb2e5
cached result backend
...
first version of a result backend using django's cache framework
2015-10-01 14:58:28 +02:00
Ilan Steemers
1aed7c2d00
#79 close django db connection before fork
...
The sentinel saves a statistic to the cache provider just before forking the workers. If this cache provider is a database backend, the connection stays open and gets forked and it crashes. Solution is to close any db connection before forking.
2015-09-29 10:47:22 +02:00
Ilan Steemers
765c84301f
#76 checks for stale connections always
...
It turns out that checking stale connections on a timer takes between 1-2 times as long as just checking them always. This also has the benefit of catching timeouts that happen between timer loops.
2015-09-28 12:48:11 +02:00
Ilan Steemers
7075a91d37
#76 adds connection checked with timeout
...
Adds a check for old connections in both the workers and the monitor , every DB_TIMEOUT seconds
2015-09-28 11:42:48 +02:00
Ilan Steemers
4ceee42be9
#76 resets stale db connection on every scheduler ping
2015-09-28 10:42:08 +02:00
Ilan Steemers
131a37daf3
Removes Pycharm stdin bug workaround
2015-09-24 11:21:50 +02:00
Ilan Steemers
7fb1142e4e
Shares Sentinel broker with Monitor
2015-09-15 20:07:33 +02:00
Ilan Steemers
a060b3508a
Moves bulk iteration to pusher
2015-09-15 14:45:06 +02:00
Ilan Steemers
2bffcf554b
Django ORM broker
2015-09-14 13:24:43 +02:00
Ilan Steemers
cc178df2b1
#56 fakes setting cpu_affinity on unsupported platforms
2015-09-09 13:25:27 +02:00
Ilan Steemers
1ccfc66254
adds fail method to broker
2015-09-03 16:34:00 +02:00
Ilan Steemers
da20ebe734
fixes connection creation on import
2015-08-31 15:15:24 +02:00
Ilan Steemers
0740c90148
adds pluggable brokers
...
* added redis broker
* added django_redis broker
* added task acknowledgement
2015-08-29 18:54:07 +02:00
Ilan Steemers
439c8a2f77
Moves Status to it's own file
...
This way only qmonitor and qinfo need Curses and you can still run the cluster without it.
2015-08-26 13:11:58 +02:00
Ilan Steemers
3e92fddcdd
#50 removes redis connection argument from pusher process
2015-08-24 10:08:43 +02:00
Ilan Steemers
b414b637eb
#50 adds psutil as alternative os.getppid provider
2015-08-23 21:19:35 +02:00
Ilan
b99e16f108
adds catch_up configuration option
...
When a cluster hasn't run for a while, the default behavior is to keep executing schedules until they are caught up with the present time.
By setting `catch_up` to False, schedules will not play catch up and instead get rescheduled to the future.
2015-08-18 11:57:15 +02:00
Ilan
ac2dc02b4c
consolidated redis ping
2015-08-17 14:25:32 +02:00
Ilan
47a92f8e56
added to translations
...
fixed _ conflict
2015-08-17 12:57:59 +02:00
Ilan
de5d0f2725
Adds a 'minutes' option to the scheduler
2015-08-05 20:30:29 +02:00
Ilan
f470fbf975
uses empty() instead of qsize() == 0
2015-08-04 18:30:07 +02:00
Ilan
ae8745ab39
Checks if cpu_count is implemented
...
On rare occasions, usually on win32, cpu_count will raise a NotImplemented error. If psutil is installed we will use this to get the correct count, otherwise we default to 4
2015-08-04 16:28:19 +02:00
Ilan
af61270f62
#40 qsize not implemented
2015-08-04 12:26:23 +02:00
Ilan
9899c42740
* fixes save_limit +1
...
* adds save_limit test
* expands recycle test
2015-07-31 18:39:54 +02:00
Ilan Steemers
f75593aebb
Moves unpacking task from Worker to Pusher.
...
Makes more sense and gives us more options to work with control queues from the pusher in the future. Does not seem to affect performance.
2015-07-31 12:29:10 +02:00
Ilan
ba58b4face
closes db connection on worker and monitor spawn
...
This should prevent some problems with Postgresql where the workers would use stale db connections and cause errors.
2015-07-27 12:31:38 +02:00
Ilan
cf325f8199
queue block is true by default.
2015-07-24 10:52:30 +02:00
Ilan
d31e9e2b97
adds a limit to the queue
2015-07-24 01:04:58 +02:00
Ilan
a8b562bc04
some minor linting
2015-07-22 17:13:29 +02:00
Ilan Steemers
a26ffbf03b
tuning timeout
2015-07-22 14:25:32 +02:00
Ilan
f6cbd9817a
fix for timeout
2015-07-22 13:11:52 +02:00
Ilan
c5b8b800f6
Adds save override option for tasks
...
Also adds `q_options` dict for async and schedule options
2015-07-20 19:43:27 +02:00
Ilan
c127604f10
Adds task groups
...
* task now takes an optional keyword `group`
* added result_group() and fetch_group()
* added group column to Success admin
* group column in searchable
* added `name` field to Schedules
* Scheduled tasks take schedule name as group
2015-07-18 17:20:50 +02:00
Ilan
b876b1e06d
Tests now run with logging level DEBUG
2015-07-17 14:17:40 +02:00
Ilan
af5e99249f
Paste accidents happen
2015-07-16 20:47:14 +02:00
Ilan
af735c328e
Adds a timeout override per task
...
* added a `timeout` keyword to tasks.
* timeouts now count down instead of up
* `timeout` in task overrides global timeout setting
* added tests for timeout override
* added a test for recycling
2015-07-16 20:35:29 +02:00
Ilan Steemers