Commit Graph

225 Commits

Author SHA1 Message Date
sinowood
52c04217e9 Multiple queue, multiple cluster in one site (#71)
* Support for multi-queue, multi-cluster configuration. API changes include:
  * Adding `cluster` to async_task() parameters and Task model
  * Adding argument --name to qcluster command
  * Necessary adjustments to Conf and Broker classes
  * Some admin improvements

* Add settings.Q_CLUSTER['ALT_CLUSTERS']: q_cluster config overrides for alternative clusters;
Add Conf.CLUSTER_NAME: separate usage from Conf.PREFIX;
QueueAdmin/OrmQ detail page enhanced: now displaying args/kwargs/q_options instead of encrypted payload.

* if `cluster` argument is not set (the default), async_task() and schedule() will be handled by the default cluster; Documentation update.

* Text cleanup

* Documentation update.

* Fix TIMEOUT setting in Windows for non-default cluster

---------

Co-authored-by: Stan Triepels <1939656+GDay@users.noreply.github.com>
2023-04-02 16:36:17 +02:00
msabatier
650c3b1524 Fix use of database router for write queries and remove Conf.HAS_REPLICA (#61)
Co-authored-by: Marc Sabatier
2023-01-18 02:12:13 +01:00
msabatier
a013591de5 Add intended_date_kwarg field to Schedule (#62)
Co-authored-by: Marc Sabatier <marc.sabatier@eco-adapt.com>
2023-01-17 22:52:11 +01:00
Stan Triepels
b66dfbd3af Fix DST timezone change (move from DST to normal jump) (#56) 2022-12-22 03:02:53 +01:00
Stan Triepels
ce81059da8 Fix: allow both ZoneInfo and Pytz depending on django version (#55) 2022-12-21 03:42:38 +01:00
Stan Triepels
e78e473be3 Fix: handling exceptions inside job function (#51) 2022-12-21 01:44:59 +01:00
Stan Triepels
4d454e4001 Fix: Daylight saving time issue with scheduler (#47) 2022-12-21 01:44:34 +01:00
Stan Triepels
3c0c32758d Chore: Flake8, isort and Black (#40) 2022-11-16 01:45:40 +01:00
Stan Triepels
542e04db54 Feat: Add biweekly and bimonthly and fix translations (#36) 2022-11-13 01:00:11 +01:00
Stan Triepels
0aac4e7b0d Fix unclear error when function is not called correctly (#19) 2022-10-20 17:24:17 +02:00
Stan Triepels
c59f036158 Remove arrow dependency, update packages, set up testing with compose (#17) 2022-10-19 02:01:16 +02:00
Stan Triepels
9ace854290 Save limit per group/func/name (#12)
* feat: option to save tasks per group/func/name

* test: update tests for save limit

* fix: convert func when save-limit checking

when passed as function it needs to be converted to work

* Add warning if option is not valid

Co-authored-by: Noortheen Raja <jnoortheen@gmail.com>
2022-10-15 01:56:41 +02:00
Ilan Steemers
f8cd136668 Post execute signal and tests (#580)
* adds post_execute signal and tests

* Updates signal docts

* Fixes typo in docs
2021-06-20 15:43:08 +02:00
Ilan Steemers
4df036dc9c Autofield (#574)
* Adds autofield default

* linting

* Updates packages
2021-06-10 15:42:21 +02:00
Ilan Steemers
15155c7a99 Build improvements (#569)
* Black linting

* Adding Black to dev dependencies and upping minimal python to 3.6.2 for compatibility

* Updating packages

* Removing pip-tools input and exporting requirements with poetry

* Deleting old setup files and test runner

* Trying 1.3.7

* Looser extras requirements to prevent conflicts

* Sorted imports with isort

* Added iSort to dev dependencies

* Fixes localtime for naive setups
2021-05-30 17:10:07 +02:00
Alexandre Xavier
71d4b726ea Feature/improves multiple databases support (#561)
* feat: adds docker local files to ignore

* feat: adds replica key to django q settings

* feat: adds multiple database routers testing utility

* feat: ensures scheduler works with multiple databases and replicas

* feat: adds docker image to support containerized development

* refactor: removes unnecessary import

* style: improves import styling

* docs: adds replica setting to configure documentation
2021-05-23 15:10:31 +02:00
Kenny Heinonen
728a1b85d0 Add "qmemory" command (#553)
* Add qmemory command

* Update README, add test

* Redesign qmemory

* Add optional argument to show each worker's memory usage

* Add --workers argument to test_qmemory
2021-05-14 13:44:40 +02:00
Dimitri
957f807598 Allow tasks to be scheduled on a specific cluster (#555)
* Allows schedule by cluster

* Update docs

* Add example in doc

* Fixes cluster field must be blank

* Adds cluster arg to schedule func

* Adds cluster field in list_display/filter for admin

* Uses Q filter to fetch schedule

* Fixes ref in doc

* Adds tests
2021-05-14 13:42:54 +02:00
Ilan Steemers
de9ba04602 Fixes deprecated count method (#549)
* Replaces deprecated count with count_documents

* Uses new re_path instead of url()
2021-05-01 16:23:37 +02:00
Gouri Javed
e205674d31 added long polling support (#506)
* added long pooling support for  sqs broker

* changed the parameter name

* added long polling support ,

introduce new parameter in broker sqs block
parameter for long polling is receive_message_wait_time_seconds

eg :
Q_CLUSTER = {
      'name': 'test-queue',
      'sqs': {
         'aws_region': AWS_S3_REGION_NAME,
         'aws_access_key_id': AWS_ACCESS_KEY_ID,
         'aws_secret_access_key': AWS_SECRET_ACCESS_KEY,
         'receive_message_wait_time_seconds':20
     }
}

* added receive_message_wait_time_seconds parameter in test_broker.py

Co-authored-by: Javed Gouri <you@example.com>
2021-03-21 12:04:16 +01:00
Ilan Steemers
7d03536e89 Migrate to Github Action CI (#507)
* Some small refactors

* Initial test script

* Fixes django matrix

* Trying to set up Disque

* Disque as a docker service

* lowercase action name

* Remove Travis

* Replaces Travis badge with Github actions badge

* Show django version in step

* Typo
2021-02-14 18:59:04 +01:00
Ilan Steemers
d97e7d772f Merge pull request #479 from alx-sdv/bugs/model-str
Model.__unicode__() has no effect in Python 3.X
2020-10-20 14:01:40 +02:00
Daniel Trembath
3e8fc91d46 Change Django documentation URLs from v1.8 to v2.2
URLs were linking to a now unsupported version of Django. Django v2.2 is a LTS version, and also matches the documentation intersphinx_mapping in docs/conf.py.
2020-10-07 17:56:23 +11:00
alx-sdv
28197dc4fa Model.__unicode__() has no effect 2020-10-05 12:30:11 +03:00
Tim O'Meara
655f0aadb5 rename config var to max_retries, remove attempt_count form admin , add admin ui customization to docs 2020-08-13 11:58:00 -05:00
Tim O'Meara
f26de929d6 adds tests 2020-08-11 11:48:23 -05:00
Ilan Steemers
0095a93b45 Adds max rss test 2020-07-08 22:52:07 +02:00
Ilan Steemers
572f810651 Adds max rss test 2020-07-08 22:47:35 +02:00
Ilan Steemers
f223cfcb30 Removing value check 2020-07-01 14:25:51 +02:00
Ilan Steemers
532199fff3 Trigger cron validation 2020-07-01 14:18:04 +02:00
Ilan Steemers
e6f82c5158 test model unicode 2020-06-30 22:00:31 +02:00
Ilan Steemers
b17fd24682 Test for no nodes config 2020-06-30 21:35:57 +02:00
Ilan Steemers
d46d08e94e Adds qinfo ids check 2020-06-30 21:19:15 +02:00
Ilan Steemers
dda0c3c44d Test get_ids 2020-06-30 20:56:11 +02:00
Ilan Steemers
34293cfcad Test cron validator 2020-06-29 19:08:08 +02:00
Ilan Steemers
a02ded0ade Get correct type name 2020-06-28 20:24:22 +02:00
Ilan Steemers
5509fbb496 Fixes test for cron 2020-06-28 19:58:03 +02:00
Ilan Steemers
1906be92fd Adds test for cron 2020-06-28 19:52:24 +02:00
Bastien Vallet
3569d2ea8c [cleanup] Fix some linting issues 2020-06-10 12:00:55 +02:00
Bastien Vallet
68d95a2924 [cleanup] Remove python2 artifacts 2020-06-10 11:48:31 +02:00
Ryan Branche
d7d0b0dfb4 Add unit test for sync=True change.
When sync=True, re-raise exceptions from the worker.
2020-02-28 14:47:28 -08:00
Ilan
7c31c2408c Replaces format with f-strings 2020-02-18 18:47:05 +01:00
Jason McVetta
35848cc80d fix typo 2020-02-13 18:17:00 +07:00
Jason McVetta
14a602ee8b use both pid and cluster_id 2020-02-12 18:23:47 +07:00
Jason McVetta
d22c916bfc add cluster_id to Sentinel 2020-02-12 17:53:52 +07:00
valentinogagliardi
c12e6184eb add test for Redis URL 2020-01-12 20:10:06 +01:00
Ilan Steemers
16fa6fa8d9 Fixes deprecated arrow syntax 2019-08-10 14:03:03 +02:00
Janne Rönkkö
0e2df88d92 Fix async_task timeout parameter handling when cluster timeout is set to None (the default)
The cluster timeout configuration default value None is documented to mean
tasks never timeout out. Also the documentation states that the timeout
can be overridden for individual tasks.

With the old implementation the timeout parameter given to async_task was
not honored if the cluster timeout was set to None.

Fixes: #335
2019-01-27 19:09:22 +02:00
Janne Rönkkö
2650659da9 Allow timeout unit tests to fail properly
The old implementation used count_forever test task that never finishes.
If the timeout implementation does not work in the test, the test never
end and you was required to kill the test runner.
2019-01-27 15:35:17 +02:00
Janne Rönkkö
d09e2658be Add test for task finishes before timeout 2019-01-27 15:35:17 +02:00