Commit Graph

136 Commits

Author SHA1 Message Date
Alexandre Xavier
71d4b726ea Feature/improves multiple databases support (#561)
* feat: adds docker local files to ignore

* feat: adds replica key to django q settings

* feat: adds multiple database routers testing utility

* feat: ensures scheduler works with multiple databases and replicas

* feat: adds docker image to support containerized development

* refactor: removes unnecessary import

* style: improves import styling

* docs: adds replica setting to configure documentation
2021-05-23 15:10:31 +02:00
nittolese
8d49825d40 Fix #422 (#556) 2021-05-14 13:49:50 +02:00
Kenny Heinonen
1eb5cf4b9b Fixes #314 - Convert func to string before saving task in databse so that resubmitting failed task works (#554) 2021-05-14 13:48:08 +02:00
Dimitri
957f807598 Allow tasks to be scheduled on a specific cluster (#555)
* Allows schedule by cluster

* Update docs

* Add example in doc

* Fixes cluster field must be blank

* Adds cluster arg to schedule func

* Adds cluster field in list_display/filter for admin

* Uses Q filter to fetch schedule

* Fixes ref in doc

* Adds tests
2021-05-14 13:42:54 +02:00
Kenny Heinonen
ad77ad7512 Fixes #225 (#552) 2021-05-07 17:26:41 +02:00
weiyang
1869a94118 Use 'timezone.localtime()' when calculating the next run time (#520)
Signed-off-by: weiyang <weiyang.ones@gmail.com>
2021-03-21 12:02:22 +01:00
Ilan Steemers
7d03536e89 Migrate to Github Action CI (#507)
* Some small refactors

* Initial test script

* Fixes django matrix

* Trying to set up Disque

* Disque as a docker service

* lowercase action name

* Remove Travis

* Replaces Travis badge with Github actions badge

* Show django version in step

* Typo
2021-02-14 18:59:04 +01:00
Telmo Barros
1d6d996fd0 enhanced admin ormq panel fix and schedules with broker_name 2021-02-06 18:52:17 +00:00
ihuk
8b3d6b7d2a Initial work on fix for #424 - TypeError: can't pickle _thread.lock objects.
Made `Broker` and `Queue` classes pickable which fixes the the problem with new default `spawn` proces start method.
2020-10-14 22:21:40 +02:00
Tim O'Meara
655f0aadb5 rename config var to max_retries, remove attempt_count form admin , add admin ui customization to docs 2020-08-13 11:58:00 -05:00
Tim O'Meara
45b0b2cc27 add attempt_count to config docs, set value to 1 in inital task save 2020-08-11 10:42:28 -05:00
Tim O'Meara
3c3395f24b add attempt_count to task 2020-08-11 10:16:31 -05:00
Ilan Steemers
c94f2cb816 Adds psutil support for max_rss on other platforms 2020-07-05 18:18:43 +02:00
Ilan Steemers
b16878f36f Allow for platforms not supporting resource module 2020-07-05 17:45:07 +02:00
Ilan Steemers
83b8b7c0d7 Adds max_rss memory limit for recycle 2020-07-05 17:04:16 +02:00
Ilan Steemers
f0061dccd6 Moves optional import to root 2020-06-29 19:08:33 +02:00
Ilan Steemers
5bf2cd49eb Fixes deprecation 2020-06-28 20:23:46 +02:00
Ilan Steemers
ace3d01da7 Optional Cron expression scheduling - inital commit 2020-06-28 15:26:53 +02:00
Ilan Steemers
6e95dc2d9b Hints and lints 2020-06-15 16:02:14 +02:00
Ilan Steemers
ef4c7876ae Linting and hints 2020-06-12 12:33:04 +02:00
Bastien Vallet
68d95a2924 [cleanup] Remove python2 artifacts 2020-06-10 11:48:31 +02:00
Ilan Steemers
a6eda43c72 Merge pull request #428 from hhyo/hhyo-patch-1
Fix issue when using USE_TZ=False with MySQL
2020-05-27 17:49:38 +02:00
Ilan Steemers
bef944af61 Merge pull request #440 from maerteijn/fix-database-router
Fix issues when using multiple databases with a database router
2020-05-27 17:48:20 +02:00
Martijn Jacobs
f0546bc91c Lock the Scheduler model for the database it's using
This can happen when you use a database router for example
2020-05-01 17:30:19 +02:00
小圈圈
25d1e3e3a0 Fix issue when using USE_TZ=False with MySQL #353 2020-04-15 20:57:27 +08:00
Ryan Branche
2f14c122e4 When sync=True, re-raise exceptions from the worker.
This commit adds code to detect if the task is running synchronously in
the worker and re-raise exceptions that were originally raised by the
underlying task function.
2020-02-28 14:26:20 -08:00
Ilan
0bca3cbd7a Replacing format with f-strings 2020-02-17 16:59:12 +01:00
Jason McVetta
4529c38067 log cluster_id instead of pid 2020-02-12 18:26:53 +07:00
Jason McVetta
14a602ee8b use both pid and cluster_id 2020-02-12 18:23:47 +07:00
Jason McVetta
d22c916bfc add cluster_id to Sentinel 2020-02-12 17:53:52 +07:00
Jason McVetta
ea30c1d43f add UUID4-based cluster_id 2020-02-12 17:41:10 +07:00
Ilan Steemers
681375364d Merge pull request #393 from ossobv/sync
Preserve database connection when sync=True
2020-01-17 15:34:46 +01:00
Jake Bell
dc65f5e1de Replacing ugettext_ functions with gettext_ for Django 3 2020-01-07 15:57:37 -06:00
Harm Geerts
66588a28e7 Preserve database connection when sync=True
When tasks run synchronously django-q should only affect the calling
context by running the tasks.
Fixes https://github.com/Koed00/django-q/issues/326
2019-11-27 12:37:31 +01:00
Ilan Steemers
c27938ee62 Merge pull request #333 from tremby/fix-timeout-override
Fix timeout override
2019-08-13 14:16:03 +02:00
Martijn Jacobs
924ff928b0 Lock the Schedule objects to ensure tasks are created once
Tested this with PostgreSQL and MySQL
2019-08-12 14:28:26 +02:00
Martijn Jacobs
abaa0ba39b Don't call the scheduler twice: respect Conf.SCHEDULER 2019-08-12 14:27:56 +02:00
Ilan Steemers
7d437ea5cf Merge pull request #353 from maerteijn/fix-mysql-naive-datetime-in-scheduler
Fix issue when using USE_TZ=False with MySQL
2019-08-10 14:16:11 +02:00
Ilan Steemers
16fa6fa8d9 Fixes deprecated arrow syntax 2019-08-10 14:03:03 +02:00
Martijn Jacobs
c0cda97864 Don't save the next_run field as TZ aware when USE_TZ is False
arrow always returns a tz aware datetime, and we don't want this when we explicitly configured django with USE_TZ=False. This causes issues with MySQL.
2019-04-17 15:02:20 +02:00
Janne Rönkkö
bca22054df Fix concurrency issue in timeout timer value processing
According to multiprocessing documentation for Value
(https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Value)
reads and writes are protected with lock when the lock argument is True
(the default) or the lock argument is an instance of Lock or RLock. The
documentation states that operations like += are not atomic as that
involves reading and writing.

On the worker side the critical section includes also storing finished task
result because the timeout could happen after the task function has
finished but before the result has been stored and timer.value has been
updated to tell the guard process that the task has been finished.

On the guard side the critical section includes all checks done to see
if the worker has timed out or died and the actual reincarnation function
because the worker could update timer value to -1 (idle) or -2 (recycle)
after the guard has seen timer value 0 (timeout) and is going to terminate
the worker.
2019-01-28 08:39:56 +02:00
Ilan Steemers
44d963b716 Merge pull request #327 from mattaw/master
Modified django_q imports to support Python 3.4 again in cluster.py. …
2019-01-27 18:34:30 +01:00
Ilan Steemers
198ab59555 Merge branch 'master' into fix-timeout-override 2019-01-27 18:30:05 +01:00
Janne Rönkkö
0e2df88d92 Fix async_task timeout parameter handling when cluster timeout is set to None (the default)
The cluster timeout configuration default value None is documented to mean
tasks never timeout out. Also the documentation states that the timeout
can be overridden for individual tasks.

With the old implementation the timeout parameter given to async_task was
not honored if the cluster timeout was set to None.

Fixes: #335
2019-01-27 19:09:22 +02:00
Bart Nagel
4c2a2f3279 Fix timeout override
Closes https://github.com/Koed00/django-q/issues/332
2019-01-21 16:23:45 -08:00
Matthew Swabey
e84be5f33b Modified django_q imports to support Python 3.4 again in cluster.py. Closes #315 2018-12-09 12:28:04 -05:00
Ilan Steemers
b5a01a9e93 Removes old rollbar support - relies on the pluggable error reporters now 2018-08-10 13:03:58 +02:00
Ilan Steemers
7d5c9f8b46 removes xrange compat 2018-08-05 13:39:04 +02:00
Ilan Steemers
bd062503c5 Adds tracebacks in error logs @strets123 2018-08-05 11:13:16 +02:00
Ilan Steemers
acebdaa850 Renames async/enqueue to async_task. 2018-08-01 15:20:34 +02:00