Commit Graph

85 Commits

Author SHA1 Message Date
sinowood
52c04217e9 Multiple queue, multiple cluster in one site (#71)
* Support for multi-queue, multi-cluster configuration. API changes include:
  * Adding `cluster` to async_task() parameters and Task model
  * Adding argument --name to qcluster command
  * Necessary adjustments to Conf and Broker classes
  * Some admin improvements

* Add settings.Q_CLUSTER['ALT_CLUSTERS']: q_cluster config overrides for alternative clusters;
Add Conf.CLUSTER_NAME: separate usage from Conf.PREFIX;
QueueAdmin/OrmQ detail page enhanced: now displaying args/kwargs/q_options instead of encrypted payload.

* if `cluster` argument is not set (the default), async_task() and schedule() will be handled by the default cluster; Documentation update.

* Text cleanup

* Documentation update.

* Fix TIMEOUT setting in Windows for non-default cluster

---------

Co-authored-by: Stan Triepels <1939656+GDay@users.noreply.github.com>
2023-04-02 16:36:17 +02:00
GDay
86467dcf03 Revert "Merge pull request #15 from django-q2/error-on-timeout"
This reverts commit d97dca5bcb, reversing
changes made to 01cb652071.
2023-03-31 02:22:38 +02:00
Jim Jagielski
f5ecd337ad Merge branch 'master' into error-on-timeout 2023-02-17 12:33:47 -05:00
msabatier
17c1609f10 Add meaningfull process titles with currently running task name (#57)
* Customize process names (in ps/top) and add task name in logs

* Increase severity of reincarnate log messages and add task name

* Document setproctitle optional dependency

Co-authored-by: Marc Sabatier <37879561+msabatier@users.noreply.github.com>
Co-authored-by: Stan Triepels <1939656+GDay@users.noreply.github.com>
2023-01-26 02:21:56 +01:00
msabatier
650c3b1524 Fix use of database router for write queries and remove Conf.HAS_REPLICA (#61)
Co-authored-by: Marc Sabatier
2023-01-18 02:12:13 +01:00
Stan Triepels
4d454e4001 Fix: Daylight saving time issue with scheduler (#47) 2022-12-21 01:44:34 +01:00
Stan Triepels
3c0c32758d Chore: Flake8, isort and Black (#40) 2022-11-16 01:45:40 +01:00
Stan Triepels
542e04db54 Feat: Add biweekly and bimonthly and fix translations (#36) 2022-11-13 01:00:11 +01:00
Noortheen Raja
64d73a77b7 feat: option to fail-on-timeout
fixes #251
2022-10-19 02:27:12 +02:00
Stan Triepels
c59f036158 Remove arrow dependency, update packages, set up testing with compose (#17) 2022-10-19 02:01:16 +02:00
Stan Triepels
00fe611b86 Use logger.hasHandlers() to setup fallback logging (#13)
Instead of checking if the django-q logger has any handlers directly,
use the hasHandlers [1] method, which will check if any parent loggers
have handlers as well.

This fixes a bug where django-q configures it's own logging even though
the root logger has a handler set

 1. https://docs.python.org/3/library/logging.html#logging.Logger.hasHandlers

Co-authored-by: Wesley Ellis <wellis@benchsci.com>
2022-10-15 17:39:51 +02:00
Stan Triepels
9ace854290 Save limit per group/func/name (#12)
* feat: option to save tasks per group/func/name

* test: update tests for save limit

* fix: convert func when save-limit checking

when passed as function it needs to be converted to work

* Add warning if option is not valid

Co-authored-by: Noortheen Raja <jnoortheen@gmail.com>
2022-10-15 01:56:41 +02:00
Stan Triepels
e3f959c258 Admin improvements (#11)
* Add the group column to OrmQ admin page

* Add group field to tasks list and search_field

* Add search by name to Schedule admin page

* Limit succesful task history per schedule with SAVE_LIMIT_GROUP config

* Improve ScheduleAdmin list view performance

Co-authored-by: Marc Sabatier <marc@sabatier.online>
2022-10-14 17:13:48 +02:00
Ilan Steemers
15155c7a99 Build improvements (#569)
* Black linting

* Adding Black to dev dependencies and upping minimal python to 3.6.2 for compatibility

* Updating packages

* Removing pip-tools input and exporting requirements with poetry

* Deleting old setup files and test runner

* Trying 1.3.7

* Looser extras requirements to prevent conflicts

* Sorted imports with isort

* Added iSort to dev dependencies

* Fixes localtime for naive setups
2021-05-30 17:10:07 +02:00
Alexandre Xavier
71d4b726ea Feature/improves multiple databases support (#561)
* feat: adds docker local files to ignore

* feat: adds replica key to django q settings

* feat: adds multiple database routers testing utility

* feat: ensures scheduler works with multiple databases and replicas

* feat: adds docker image to support containerized development

* refactor: removes unnecessary import

* style: improves import styling

* docs: adds replica setting to configure documentation
2021-05-23 15:10:31 +02:00
Ruben
7892aef70a Add a warning for misconfiguration. (#509)
* Add a warning  for misconfiguration.

As specified in the docs, by default the settings result in a configuration where slow tasks that run for more than 60s cause repeated running of tasks, which has caused a few issues. This warning doesn't change the defaults, which would be a marge larger change, but instead provides a, hopefully, helpful message to the user.

* fixed missing bracket

* Update conf.py

fixed spelling

* Update conf.py

Made bool work for python 3.6+

* Fixing typo

Co-authored-by: Ilan Steemers <koed00@gmail.com>
2021-02-26 13:54:56 +01:00
Tim O'Meara
655f0aadb5 rename config var to max_retries, remove attempt_count form admin , add admin ui customization to docs 2020-08-13 11:58:00 -05:00
Tim O'Meara
3c3395f24b add attempt_count to task 2020-08-11 10:16:31 -05:00
Ilan Steemers
c94f2cb816 Adds psutil support for max_rss on other platforms 2020-07-05 18:18:43 +02:00
Ilan Steemers
b16878f36f Allow for platforms not supporting resource module 2020-07-05 17:45:07 +02:00
Ilan Steemers
83b8b7c0d7 Adds max_rss memory limit for recycle 2020-07-05 17:04:16 +02:00
Ilan Steemers
6e95dc2d9b Hints and lints 2020-06-15 16:02:14 +02:00
Bastien Vallet
68d95a2924 [cleanup] Remove python2 artifacts 2020-06-10 11:48:31 +02:00
Ilan
0bca3cbd7a Replacing format with f-strings 2020-02-17 16:59:12 +01:00
Kevin Lord
0d9ebf94ee Fix startup error in AWS Lambda 2020-01-20 22:39:40 -06:00
Jake Bell
dc65f5e1de Replacing ugettext_ functions with gettext_ for Django 3 2020-01-07 15:57:37 -06:00
Ilan Steemers
b5a01a9e93 Removes old rollbar support - relies on the pluggable error reporters now 2018-08-10 13:03:58 +02:00
Balletie
03abbc960f Add option for acknowledging failed tasks (globally and per-task)
If a task fails with an exception, it is retried until it
succeeds. This is contrary to what is said in the documentation: under
the "Architecture" section, heading "Broker" it says that even when a
task errors, it's still considered a successful delivery. Failed tasks
never get acknowledged however, thereby being retried after the
timeout period. See also issues #238 and #194.

This patch adds an option to acknowledge failures, thereby closing
issue #238. Issue #194 would require some more work. The default of
this option is set to `False`, thereby maintaining backwards
compatibility.
2018-03-09 22:09:46 +01:00
Ronald van Zon
dd4f02f122 Add custom SharedCounter / Queue
This will not fix anything related to Django 2.0 but will allow MAC
users to run Django-Q allowing tests to be run locally.
2017-12-13 14:36:12 +01:00
Daniel Welch
c6466868d8 fix usage of iter_entry_points 2017-12-07 18:30:54 -05:00
Daniel Welch
4530b05d86 putting back support for backwards compatibility 2017-10-06 13:26:43 -04:00
Daniel Welch
cd038f6f61 remove old rollbar stuff from config 2017-10-03 17:08:26 -04:00
Daniel Welch
c82ef6a177 fixing export from conf 2017-03-30 22:12:31 -04:00
Daniel Welch
9e97d48012 registering entry points for sentry and rollbar plugins, maintained in seperate repos. start of a generic error handling implementation 2017-03-30 21:51:57 -04:00
Yann Pomarede
aa667b40d4 daemonize_workers option 2016-12-05 14:55:42 +01:00
ilan
6734f5b96c Removes value check for guard cycle 2016-07-21 16:41:30 +02:00
Vladimir Rychkov
cadbc9f220 exception raised if guard_cycle configured not in range (0, 60) 2016-07-13 09:44:26 +01:00
Vladimir Rychkov
1a1385dfba Guard loop sleep made configurable 2016-07-12 13:08:17 +01:00
Ilan Steemers
c17468ef86 docs: Adds Custom broker class documentation
Also changes some of the variable and settings names, to make them less ambiguous.
2016-01-27 12:06:49 +01:00
Ilan Steemers
8298f2319f Fixes issue with rollbar configuration 2016-01-27 10:32:48 +01:00
Ilan Steemers
af3b1da08f Makes DB polling time configurable via poll setting 2016-01-26 22:29:00 +01:00
Ilan Steemers
91bb1c69c7 Fixes double log output 2016-01-08 13:37:58 +01:00
Ilan Steemers
d29c861d09 Adds Rollbar support for exceptions
Also formats exceptions in case they are unprintable
2016-01-07 18:50:55 +01:00
Ilan Steemers
33282cb2e5 cached result backend
first version of a result backend using django's cache framework
2015-10-01 14:58:28 +02:00
Ilan Steemers
b8bb186471 check for default database on mongo connection 2015-09-26 13:01:15 +02:00
Ilan Steemers
826178c463 Adds a MongoDB broker
* adds mongo broker tests
* adds mongo brokers docs

* broker info is now cached when it needs to connect to a server, to reduce traffic.
2015-09-25 20:03:39 +02:00
Ilan Steemers
2bffcf554b Django ORM broker 2015-09-14 13:24:43 +02:00
Ilan Steemers
8543c5b328 Adds Fastack option to Disque broker 2015-09-11 17:58:16 +02:00
Ilan Steemers
d7ce945f03 Amazon SQS broker 2015-09-09 18:10:03 +02:00
Ilan Steemers
83548dcfbb Makes bulk a general setting 2015-09-08 13:42:15 +02:00