49 Commits

Author SHA1 Message Date
GDay
fb960104c3 Release v1.5.2 2023-04-13 01:31:59 +02:00
GDay
88bbfd8967 Mention support for 4.2.x in docs 2023-04-13 01:21:46 +02:00
Martijn Jacobs
1f31725f43 Added Django 4.2 to the test matrix, fixed deprecation warning (#89)
* Added Django 4.2 to the CI test matrix

* Positional arguments with TimestampSigner are deprecated
2023-04-12 18:18:03 +02:00
GDay
322a53f06d Release v1.5.1 2023-04-02 17:53:23 +02:00
GDay
77821b1de7 Fix release workflow after name change 2023-04-02 17:51:17 +02:00
GDay
38d48231d6 Release v1.5.0 2023-04-02 17:43:17 +02:00
Stan Triepels
d13e9d3d64 Update dependencies (2023-04-02) (#88) 2023-04-02 17:15:39 +02:00
sinowood
52c04217e9 Multiple queue, multiple cluster in one site (#71)
* Support for multi-queue, multi-cluster configuration. API changes include:
  * Adding `cluster` to async_task() parameters and Task model
  * Adding argument --name to qcluster command
  * Necessary adjustments to Conf and Broker classes
  * Some admin improvements

* Add settings.Q_CLUSTER['ALT_CLUSTERS']: q_cluster config overrides for alternative clusters;
Add Conf.CLUSTER_NAME: separate usage from Conf.PREFIX;
QueueAdmin/OrmQ detail page enhanced: now displaying args/kwargs/q_options instead of encrypted payload.

* if `cluster` argument is not set (the default), async_task() and schedule() will be handled by the default cluster; Documentation update.

* Text cleanup

* Documentation update.

* Fix TIMEOUT setting in Windows for non-default cluster

---------

Co-authored-by: Stan Triepels <1939656+GDay@users.noreply.github.com>
2023-04-02 16:36:17 +02:00
Stan Triepels
f8520c9cda Remove version locking on poetry_core to fix regex error (#87) 2023-03-31 02:43:48 +02:00
GDay
86467dcf03 Revert "Merge pull request #15 from django-q2/error-on-timeout"
This reverts commit d97dca5bcb, reversing
changes made to 01cb652071.
2023-03-31 02:22:38 +02:00
Stan Triepels
69a40344f0 Allow building docs all formats (#77) 2023-02-23 01:20:19 +01:00
Jim Jagielski
d97dca5bcb Merge pull request #15 from django-q2/error-on-timeout
Error on timeout
2023-02-17 12:41:27 -05:00
Jim Jagielski
f5ecd337ad Merge branch 'master' into error-on-timeout 2023-02-17 12:33:47 -05:00
GDay
01cb652071 Release v1.4.11 2023-01-31 00:24:44 +01:00
Stan Triepels
b9bdd64568 Remove custom build (revert to auto create setup file) (#70) 2023-01-31 00:21:37 +01:00
Stan Triepels
b2bcffbcab Fix missing setup file for "No matching distribution" error (#69) 2023-01-31 00:09:33 +01:00
GDay
25cd180410 Release v1.4.10 2023-01-27 02:10:20 +01:00
Stan Triepels
f9e01699c5 Adding translation mo files automatically on build (#65) 2023-01-26 16:45:44 +01:00
Stan Triepels
2f3b0d0a31 Update all dependencies (#64) 2023-01-26 02:56:20 +01:00
Stan Triepels
264607f59c Bump translations to latest changes (#63) 2023-01-26 02:49:46 +01:00
msabatier
17c1609f10 Add meaningfull process titles with currently running task name (#57)
* Customize process names (in ps/top) and add task name in logs

* Increase severity of reincarnate log messages and add task name

* Document setproctitle optional dependency

Co-authored-by: Marc Sabatier <37879561+msabatier@users.noreply.github.com>
Co-authored-by: Stan Triepels <1939656+GDay@users.noreply.github.com>
2023-01-26 02:21:56 +01:00
msabatier
650c3b1524 Fix use of database router for write queries and remove Conf.HAS_REPLICA (#61)
Co-authored-by: Marc Sabatier
2023-01-18 02:12:13 +01:00
msabatier
a013591de5 Add intended_date_kwarg field to Schedule (#62)
Co-authored-by: Marc Sabatier <marc.sabatier@eco-adapt.com>
2023-01-17 22:52:11 +01:00
msabatier
865b1a5ba8 Change task timeout logic to have now() as execution time (#58) 2023-01-11 01:48:57 +01:00
msabatier
8cd1028391 More explicit log messages in exception handling (#59) 2023-01-11 00:36:35 +01:00
GDay
31e82ad028 Release v1.4.9 2022-12-22 03:06:01 +01:00
Stan Triepels
b66dfbd3af Fix DST timezone change (move from DST to normal jump) (#56) 2022-12-22 03:02:53 +01:00
GDay
08baaab125 Release v1.4.8 2022-12-21 03:46:01 +01:00
Stan Triepels
ce81059da8 Fix: allow both ZoneInfo and Pytz depending on django version (#55) 2022-12-21 03:42:38 +01:00
Stan Triepels
cf33275d92 Release v1.4.7 (#54) 2022-12-21 01:53:10 +01:00
Stan Triepels
e78e473be3 Fix: handling exceptions inside job function (#51) 2022-12-21 01:44:59 +01:00
Stan Triepels
4d454e4001 Fix: Daylight saving time issue with scheduler (#47) 2022-12-21 01:44:34 +01:00
Stan Triepels
cf33891291 Chore: Remove release drafter (#53) 2022-12-21 01:44:10 +01:00
Stan Triepels
134a54dbeb Chore: Fix badge and add download badge (#52) 2022-12-21 01:43:51 +01:00
Stan Triepels
c1d4858f62 Release 1.4.6 (#43) 2022-11-30 15:09:49 +01:00
Andreas Andersen
2566955782 Log exceptions with logger.exception (#42)
Co-authored-by: Andreas Bok Andersen <andreas.bokandersen@glooko.com>
2022-11-30 15:03:20 +01:00
Stan Triepels
3c0c32758d Chore: Flake8, isort and Black (#40) 2022-11-16 01:45:40 +01:00
GDay
b73fea49b5 Release v1.4.5 2022-11-13 01:12:52 +01:00
GDay
cf458304cc [v1.4.4] add sudo to workflow 2022-11-13 01:08:54 +01:00
Stan Triepels
06ad6154e3 Release 1.4.4 (#37) 2022-11-13 01:06:19 +01:00
Stan Triepels
542e04db54 Feat: Add biweekly and bimonthly and fix translations (#36) 2022-11-13 01:00:11 +01:00
Stan Triepels
17205549eb Fix baseconv/base62 deprecation warning for Django 5.x (#34) 2022-11-09 00:41:12 +01:00
Stan Triepels
1764c5c8ae Release 1.4.3 (#32)
Update changelog and bump version number
2022-11-07 17:47:05 +01:00
Mikhail Krassavin
f0cb10bc37 Add Python 3.11 support and tests, remove Python 3.7 (#31) 2022-11-07 17:29:29 +01:00
Stan Triepels
167c348509 Update changelog and readme (#29)
Added changes from the previous versions and diff between Q and Q2
2022-11-05 23:06:34 +01:00
Stan Triepels
6f19b3d270 Fix: func reference in admin (#28) 2022-11-05 02:45:44 +01:00
Stan Triepels
f62bd7b6b4 Fix install docs: update q to q2 (#25) 2022-10-27 02:08:08 +02:00
Noortheen Raja
64d73a77b7 feat: option to fail-on-timeout
fixes #251
2022-10-19 02:27:12 +02:00
Noortheen Raja
596842e692 chore: pip install support
having build-system config helps installing  this without setup.py built
2022-10-19 02:27:10 +02:00
65 changed files with 3155 additions and 1898 deletions

2
.git-blame-ignore-revs Normal file
View File

@@ -0,0 +1,2 @@
# flake8, black, isort
b1d000d007f3f77069719523268a0c6256dc0860

View File

@@ -1,37 +0,0 @@
categories:
-
label: breaking
title: Breaking
-
label: feature
title: New
-
label: bug
title: "Bug Fixes"
-
label: dependencies
title: "Dependency Updates"
-
label: security
title: Security
name-template: v$NEXT_PATCH_VERSION
tag-template: v$NEXT_PATCH_VERSION
template: |
# Changes
$CHANGES
version-resolver:
major:
labels:
- breaking
- major
minor:
labels:
- feature
- minor
patch:
labels:
- bug
- dependencies
- security
- patch
default: patch

View File

@@ -7,7 +7,7 @@ on:
jobs:
build:
if: github.repository == 'GDay/django-q2'
if: github.repository == 'django-q2/django-q2'
runs-on: ubuntu-latest
steps:
@@ -22,8 +22,11 @@ jobs:
- name: Install dependencies
run: |
python -m pip install -U pip setuptools poetry
python -m pip install poetry
sudo apt-get update
sudo apt-get -y install gettext
python -m pip install pip setuptools django poetry
# compile messages to get .mo files
django-admin compilemessages
- name: Build and publish package
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
run: poetry --build --username=__token__ --password=${{ secrets.PYPI_TOKEN }} publish

View File

@@ -1,14 +0,0 @@
name: Update release draft
on:
push:
branches:
- master
jobs:
update_release_draft:
runs-on: ubuntu-latest
steps:
- uses: release-drafter/release-drafter@v5
with:
config-name: release-drafter.yml
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

View File

@@ -11,8 +11,8 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [ "3.8", "3.9", "3.10" ]
django: [ "3.2", "4.1" ]
python-version: [ "3.8", "3.9", "3.10", "3.11" ]
django: [ "3.2", "4.1", "4.2" ]
services:
disque:
@@ -73,7 +73,11 @@ jobs:
- name: Upload to coveralls
run: |
python -m pip install --upgrade pip
python -m pip install coveralls
python -m pip install coveralls flake8 black
coveralls --service=github --finish
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Check flake8/black
run: |
flake8 .
black --check .

View File

@@ -15,6 +15,8 @@ build:
sphinx:
configuration: docs/conf.py
formats: all
python:
install:
- requirements: docs/requirements.txt

View File

@@ -1,8 +1,126 @@
# Changelog
## [Unreleased](https://github.com/koed00/django-q/tree/HEAD)
## [Unreleased](https://github.com/GDay/django-q2/tree/HEAD)
[Full Changelog](https://github.com/koed00/django-q/compare/v1.3.9...HEAD)
## [v1.5.2](https://github.com/GDay/django-q2/tree/v1.5.2) (2023-04-13)
**Merged pull requests:**
- Added Django 4.2 to the test matrix, fixed deprecation warning https://github.com/GDay/django-q2/pull/89
- Updated docs to show support for 4.2
## [v1.5.1](https://github.com/GDay/django-q2/tree/v1.5.1) (2023-04-02)
- Fix release to pipy due to changed org name
## [v1.5.0](https://github.com/GDay/django-q2/tree/v1.5.0) (2023-04-02)
**Merged pull requests:**
- Multiple queue, multiple cluster in one site https://github.com/GDay/django-q2/pull/71
- Allow building docs all formats https://github.com/GDay/django-q2/pull/77
- Remove version locking on `poetry_core` to fix regex error https://github.com/GDay/django-q2/pull/87
- Update dependencies (2023-04-02) https://github.com/GDay/django-q2/pull/88
## [v1.4.11](https://github.com/GDay/django-q2/tree/v1.4.11) (2023-01-30)
**Merged pull requests:**
- Fix missing setup file for "No matching distribution" error https://github.com/GDay/django-q2/pull/69
- Remove custom build (revert to auto create setup file) https://github.com/GDay/django-q2/pull/70
## [v1.4.10](https://github.com/GDay/django-q2/tree/v1.4.10) (2023-01-26)
**Merged pull requests:**
- Adding translation mo files automatically on build https://github.com/GDay/django-q2/pull/65
- Update all dependencies https://github.com/GDay/django-q2/pull/64
- Bump translations to latest changes https://github.com/GDay/django-q2/pull/63
- Add meaningfull process titles with currently running task name https://github.com/GDay/django-q2/pull/57
- Fix use of database router for write queries and remove Conf.HAS_REPLICA https://github.com/GDay/django-q2/pull/61
- Add intended_date_kwarg field to Schedule https://github.com/GDay/django-q2/pull/62
- Change task timeout logic to have now() as execution time https://github.com/GDay/django-q2/pull/58
- More explicit log messages in exception handling https://github.com/GDay/django-q2/pull/59
## [v1.4.9](https://github.com/GDay/django-q2/tree/v1.4.9) (2022-12-22)
**Merged pull requests:**
- Fix DST timezone change (move from DST to normal jump) https://github.com/GDay/django-q2/pull/56
## [v1.4.8](https://github.com/GDay/django-q2/tree/v1.4.8) (2022-12-21)
**Merged pull requests:**
- Fix: allow both ZoneInfo and Pytz depending on django version https://github.com/GDay/django-q2/pull/55
## [v1.4.7](https://github.com/GDay/django-q2/tree/v1.4.7) (2022-12-21)
**Merged pull requests:**
- Fix: handling exceptions inside job function https://github.com/GDay/django-q2/pull/51
- Fix: Daylight saving time issue with scheduler https://github.com/GDay/django-q2/pull/47
- Chore: Fix badge and add download badge https://github.com/GDay/django-q2/pull/52
- Chore: Remove release drafter https://github.com/GDay/django-q2/pull/53
## [v1.4.6](https://github.com/GDay/django-q2/tree/v1.4.6) (2022-11-30)
**Merged pull requests:**
- Fix: Log exceptions with logger.exception https://github.com/GDay/django-q2/pull/42
- Chore: flake8, isort, black https://github.com/GDay/django-q2/pull/40
## [v1.4.5](https://github.com/GDay/django-q2/tree/v1.4.5) (2022-11-13)
- Fix release workflow
## [v1.4.4](https://github.com/GDay/django-q2/tree/v1.4.4) (2022-11-13)
**Merged pull requests:**
- Fix: Deprecation warning for Django 5.x https://github.com/GDay/django-q2/pull/34
- Feat: Add biweekly and bimonthly https://github.com/GDay/django-q2/pull/36
- Fix: Fix all translation strings and remove compiled https://github.com/GDay/django-q2/pull/36
## [v1.4.3](https://github.com/GDay/django-q2/tree/v1.4.3) (2022-11-07)
**Merged pull requests:**
- Fix: func reference in admin https://github.com/GDay/django-q2/pull/28
- Add python 3.11 support and remove 3.7 (as it was never supported by this package anyway)
## [v1.4.2](https://github.com/GDay/django-q2/tree/v1.4.2) (2022-10-22)
**Merged pull requests:**
- Make redis dependency optional and update boto3 #22
## [v1.4.1](https://github.com/GDay/django-q2/tree/v1.4.1) (2022-10-21)
**Merged pull requests:**
- Fix typo configure.rst https://github.com/GDay/django-q2/pull/1
- Update dependencies https://github.com/GDay/django-q2/pull/2
- Show function name in log when running task https://github.com/GDay/django-q2/pull/3
- Codecov -> Coveralls https://github.com/GDay/django-q2/pull/4
- Fix: readthedocs config file https://github.com/GDay/django-q2/pull/5
- Docs updates https://github.com/GDay/django-q2/pull/6
- Feat: Release plan to pypi https://github.com/GDay/django-q2/pull/7
- Feat: Turkish translations https://github.com/GDay/django-q2/pull/8
- Fix: Connection issues with CONN_MAX_AGE > 0 https://github.com/GDay/django-q2/pull/9
- Replace use of eval() by ast.parse() + ast.literal_eval() https://github.com/GDay/django-q2/pull/10
- Admin improvements https://github.com/GDay/django-q2/pull/11
- Save limit per group/func/name https://github.com/GDay/django-q2/pull/12
- Use logger.hasHandlers() to setup fallback logging https://github.com/GDay/django-q2/pull/13
- allow atomic on external db https://github.com/GDay/django-q2/pull/14
- Fix install command and remove old warnings https://github.com/GDay/django-q2/pull/16
- Remove arrow dependency https://github.com/GDay/django-q2/pull/17
- Remove funding file, add docker/compose for development project and fix https://github.com/GDay/django-q2/pull/18
- Fix unclear error when function is not called correctly https://github.com/GDay/django-q2/pull/19
- Remove blessed dependency https://github.com/GDay/django-q2/pull/20
- Release new version and docs fixes https://github.com/GDay/django-q2/pull/21
## v1.4.0
**Closed issues:**

View File

@@ -1,7 +1,7 @@
A multiprocessing distributed task queue for Django
---------------------------------------------------
|image0| |image1| |docs|
|image0| |image1| |docs| |downloads|
::
@@ -22,13 +22,25 @@ Features
- Redis, IronMQ, SQS, MongoDB or ORM
- Rollbar and Sentry support
Changes compared to the original Django-Q:
- Dropped support for Disque (hasn't been updated in a long time)
- Dropped Redis, Arrow and Blessed dependencies
- Updated all current dependencies
- Added tests for Django 4.x
- Added Turkish language
- Improved admin area
- Fixed a lot of issues
See the `changelog <https://github.com/GDay/django-q2/blob/master/CHANGELOG.md>`__ for all changes.
Requirements
~~~~~~~~~~~~
- `Django <https://www.djangoproject.com>`__ > = 3.2
- `Django-picklefield <https://github.com/gintas/django-picklefield>`__
Tested with: Python 3.7, 3.8, 3.9, 3.10 Django 3.2.X and 4.1.X
Tested with: Python 3.8, 3.9, 3.10 and 3.11. Works with Django 3.2.X, 4.1.X and 4.2.X.
Brokers
~~~~~~~
@@ -233,4 +245,6 @@ Acknowledgements
.. |docs| image:: https://readthedocs.org/projects/docs/badge/?version=latest
:alt: Documentation Status
:scale: 100
:target: https://django-q.readthedocs.org/
:target: https://django-q2.readthedocs.org/
.. |downloads| image:: https://img.shields.io/pypi/dm/django-q2
:target: https://img.shields.io/pypi/dm/django-q2

View File

@@ -0,0 +1,9 @@
import subprocess
def generate_mo_files():
subprocess.run(["django-admin", "compilemessages"])
if __name__ == "__main__":
generate_mo_files()

View File

@@ -1,7 +1,7 @@
VERSION = (1, 3, 9)
import django
VERSION = (1, 5, 2)
if django.VERSION < (3, 2):
default_app_config = "django_q.apps.DjangoQConfig"

View File

@@ -1,9 +1,9 @@
"""Admin module for Django."""
from django.contrib import admin
from django.db.models.expressions import OuterRef, Subquery
from django.urls import reverse
from django.utils.html import format_html
from django.contrib import admin
from django.utils.translation import gettext_lazy as _
from django.db.models.expressions import OuterRef, Subquery
from django_q.conf import Conf, croniter
from django_q.models import Failure, OrmQ, Schedule, Success, Task
@@ -13,7 +13,7 @@ from django_q.tasks import async_task
class TaskAdmin(admin.ModelAdmin):
"""model admin for success tasks."""
list_display = ("name", "group", "func", "started", "stopped", "time_taken")
list_display = ("name", "group", "func", "cluster", "started", "stopped", "time_taken")
def has_add_permission(self, request):
"""Don't allow adds."""
@@ -26,7 +26,7 @@ class TaskAdmin(admin.ModelAdmin):
search_fields = ("name", "func", "group")
readonly_fields = []
list_filter = ("group",)
list_filter = ("group", "cluster")
def get_readonly_fields(self, request, obj=None):
"""Set all fields readonly."""
@@ -36,7 +36,8 @@ class TaskAdmin(admin.ModelAdmin):
def retry_failed(FailAdmin, request, queryset):
"""Submit selected tasks back to the queue."""
for task in queryset:
async_task(task.func, *task.args or (), hook=task.hook, **task.kwargs or {})
async_task(task.func, *task.args or (), hook=task.hook,
group=task.group, cluster=task.cluster, **task.kwargs or {})
task.delete()
@@ -46,7 +47,7 @@ retry_failed.short_description = _("Resubmit selected tasks to queue")
class FailAdmin(admin.ModelAdmin):
"""model admin for failed tasks."""
list_display = ("name", "group", "func", "started", "stopped", "short_result")
list_display = ("name", "group", "func", "cluster", "started", "stopped", "short_result")
def has_add_permission(self, request):
"""Don't allow adds."""
@@ -54,7 +55,7 @@ class FailAdmin(admin.ModelAdmin):
actions = [retry_failed]
search_fields = ("name", "func", "group")
list_filter = ("group",)
list_filter = ("group", "cluster")
readonly_fields = []
def get_readonly_fields(self, request, obj=None):
@@ -82,18 +83,27 @@ class ScheduleAdmin(admin.ModelAdmin):
readonly_fields = ("cron",)
list_filter = ("next_run", "schedule_type", "cluster")
search_fields = ("name", "func",)
search_fields = (
"name",
"func",
)
list_display_links = ("id", "name")
def get_queryset(self, request):
qs = super().get_queryset(request)
task_query = Task.objects.filter(id=OuterRef('task')).values('id', 'name', 'success')
qs = qs.annotate(task_id=Subquery(task_query.values('id')), task_name=Subquery(task_query.values('name')),
task_success=Subquery(task_query.values('success')))
task_query = Task.objects.filter(id=OuterRef("task")).values(
"id", "name", "success"
)
qs = qs.annotate(
task_id=Subquery(task_query.values("id")),
task_name=Subquery(task_query.values("name")),
task_success=Subquery(task_query.values("success")),
)
return qs
def get_success(self, obj):
return obj.task_success
get_success.boolean = True
get_success.short_description = _("success")
@@ -105,6 +115,7 @@ class ScheduleAdmin(admin.ModelAdmin):
url = reverse("admin:django_q_failure_change", args=(obj.task_id,))
return format_html(f'<a href="{url}">[{obj.task_name}]</a>')
return None
get_last_run.allow_tags = True
get_last_run.short_description = _("last_run")
@@ -112,7 +123,9 @@ class ScheduleAdmin(admin.ModelAdmin):
class QueueAdmin(admin.ModelAdmin):
"""queue admin for ORM broker"""
list_display = ("id", "key", "name", "group", "func", "lock", "task_id")
list_display = ("id", "key", "name", "group", "func", "lock", "task_id")
fields = ("key", "lock", "task_id", "name", "group", "func", "args", "kwargs", "q_options")
readonly_fields = fields[2:]
def save_model(self, request, obj, form, change):
obj.save(using=Conf.ORM)

View File

@@ -9,4 +9,4 @@ class DjangoQConfig(AppConfig):
default_auto_field = "django.db.models.AutoField"
def ready(self):
from django_q.signals import call_hook
from django_q.signals import call_hook # noqa: F401

View File

@@ -7,7 +7,9 @@ from django_q.conf import Conf
class Broker:
def __init__(self, list_key: str = Conf.PREFIX):
def __init__(self, list_key: str = None):
# With same BROKER_CLASS, `list_key` is just a synonym for `queue_name` except for RedisBroker
list_key = list_key or Conf.CLUSTER_NAME
self.connection = self.get_connection(list_key)
self.list_key = list_key
self.cache = self.get_cache()
@@ -151,7 +153,7 @@ class Broker:
return None
@staticmethod
def get_connection(list_key: str = Conf.PREFIX):
def get_connection(list_key: str = None):
"""
Gets a connection to the broker
:param list_key: Optional queue name
@@ -160,13 +162,14 @@ class Broker:
return 0
def get_broker(list_key: str = Conf.PREFIX) -> Broker:
def get_broker(list_key: str = None) -> Broker:
"""
Gets the configured broker type
:param list_key: optional queue name
:type list_key: str
:return: a broker instance
"""
list_key = list_key or Conf.CLUSTER_NAME
# custom
if Conf.BROKER_CLASS:
module, func = Conf.BROKER_CLASS.rsplit(".", 1)

View File

@@ -8,7 +8,7 @@ QUEUE_DOES_NOT_EXIST = "AWS.SimpleQueueService.NonExistentQueue"
class Sqs(Broker):
def __init__(self, list_key: str = Conf.PREFIX):
def __init__(self, list_key: str = None):
self.sqs = None
super(Sqs, self).__init__(list_key)
self.queue = self.get_queue()
@@ -39,7 +39,8 @@ class Sqs(Broker):
raise ValueError("receive_message_wait_time_seconds should be int")
if wait_time_second > 20:
raise ValueError(
"receive_message_wait_time_seconds is invalid. Reason: Must be >= 0 and <= 20"
"receive_message_wait_time_seconds is invalid. Reason: Must be >= 0"
" and <= 20"
)
params.update({"WaitTimeSeconds": wait_time_second})
@@ -76,7 +77,7 @@ class Sqs(Broker):
return "AWS SQS"
@staticmethod
def get_connection(list_key: str = Conf.PREFIX) -> Session:
def get_connection(list_key: str = None) -> Session:
config = Conf.SQS
if "aws_region" in config:
config["region_name"] = config["aws_region"]

View File

@@ -46,6 +46,7 @@ class IronMQBroker(Broker):
return self.delete(task_id)
@staticmethod
def get_connection(list_key: str = Conf.PREFIX) -> Queue:
def get_connection(list_key: str = None) -> Queue:
list_key = list_key or Conf.CLUSTER_NAME
ironmq = IronMQ(name=None, **Conf.IRON_MQ)
return ironmq.queue(queue_name=list_key)

View File

@@ -15,7 +15,7 @@ def _timeout():
class Mongo(Broker):
def __init__(self, list_key=Conf.PREFIX):
def __init__(self, list_key: str = None):
super(Mongo, self).__init__(list_key)
self.collection = self.get_collection()
@@ -24,7 +24,7 @@ class Mongo(Broker):
self.collection = self.get_collection()
@staticmethod
def get_connection(list_key: str = Conf.PREFIX) -> MongoClient:
def get_connection(list_key: str = None) -> MongoClient:
return MongoClient(**Conf.MONGO)
def get_collection(self):

View File

@@ -11,12 +11,12 @@ from django_q.models import OrmQ
def _timeout():
return timezone.now() - timedelta(seconds=Conf.RETRY)
return timezone.now() + timedelta(seconds=Conf.RETRY)
class ORM(Broker):
@staticmethod
def get_connection(list_key: str = Conf.PREFIX):
def get_connection(list_key: str = None):
if transaction.get_autocommit(
using=Conf.ORM
): # Only True when not in an atomic block
@@ -31,13 +31,13 @@ class ORM(Broker):
def queue_size(self) -> int:
return (
self.get_connection()
.filter(key=self.list_key, lock__lte=_timeout())
.filter(key=self.list_key, lock__lte=timezone.now())
.count()
)
def lock_size(self) -> int:
return (
self.get_connection().filter(key=self.list_key, lock__gt=_timeout()).count()
self.get_connection().filter(key=self.list_key, lock__gt=timezone.now()).count()
)
def purge_queue(self):
@@ -55,14 +55,15 @@ class ORM(Broker):
self.delete(task_id)
def enqueue(self, task):
# list_key might be null (e.g. in a test setup) but OrmQ.key has not-null constraint
package = self.get_connection().create(
key=self.list_key, payload=task, lock=_timeout()
key=self.list_key or Conf.CLUSTER_NAME, payload=task, lock=timezone.now()
)
return package.pk
def dequeue(self):
tasks = self.get_connection().filter(key=self.list_key, lock__lt=_timeout())[
0 : Conf.BULK
tasks = self.get_connection().filter(key=self.list_key, lock__lt=timezone.now())[
0 : Conf.BULK # noqa: E203
]
if tasks:
task_list = []
@@ -70,10 +71,11 @@ class ORM(Broker):
if (
self.get_connection()
.filter(id=task.id, lock=task.lock)
.update(lock=timezone.now())
.update(lock=_timeout())
):
task_list.append((task.pk, task.payload))
# else don't process, as another cluster has been faster than us on that task
# else don't process, as another cluster has been faster than us on
# that task
return task_list
# empty queue, spare the cpu
sleep(Conf.POLL)

View File

@@ -11,7 +11,8 @@ except ImportError:
class Redis(Broker):
def __init__(self, list_key: str = Conf.PREFIX):
def __init__(self, list_key: str = None):
list_key = list_key or Conf.CLUSTER_NAME
super(Redis, self).__init__(list_key=f"django_q:{list_key}:q")
def enqueue(self, task):
@@ -57,7 +58,7 @@ class Redis(Broker):
return self.connection.mget(keys)
@staticmethod
def get_connection(list_key: str = Conf.PREFIX) -> Redis:
def get_connection(list_key: str = None) -> Redis:
if django_redis and Conf.DJANGO_REDIS:
return django_redis.get_redis_connection(Conf.DJANGO_REDIS)
if isinstance(Conf.REDIS, str):

View File

@@ -1,6 +1,5 @@
# Standard
import ast
import inspect
import pydoc
import signal
import socket
@@ -21,7 +20,6 @@ except core.exceptions.AppRegistryNotReady:
django.setup()
from django.conf import settings
from django.utils import timezone
from django.utils.translation import gettext_lazy as _
@@ -35,6 +33,7 @@ from django_q.conf import (
get_ppid,
logger,
psutil,
setproctitle,
resource,
)
from django_q.humanhash import humanize
@@ -44,23 +43,27 @@ from django_q.signals import post_execute, pre_execute
from django_q.signing import BadSignature, SignedPackage
from django_q.status import Stat, Status
from .utils import add_months, add_years
from .utils import get_func_repr, localtime
class Cluster:
def __init__(self, broker: Broker = None):
self.broker = broker or get_broker()
# Cluster do not need an init or default broker except for testing,
# The sentinel will create a broker for cluster and utilize ALT_CLUSTERS config in Conf.
self.broker = broker # DON'T USE get_broker() to set a default broker here.
self.sentinel = None
self.stop_event = None
self.start_event = None
self.pid = current_process().pid
self.cluster_id = uuid.uuid4()
self.host = socket.gethostname()
self.timeout = Conf.TIMEOUT
self.timeout = None
signal.signal(signal.SIGTERM, self.sig_handler)
signal.signal(signal.SIGINT, self.sig_handler)
def start(self) -> int:
if setproctitle:
setproctitle.setproctitle(f"qcluster {current_process().name} {self.name}")
# Start Sentinel
self.stop_event = Event()
self.start_event = Event()
@@ -75,7 +78,7 @@ class Cluster:
),
)
self.sentinel.start()
logger.info(_(f"Q Cluster {self.name} starting."))
logger.info(_("Q Cluster %(name)s starting.") % {"name": self.name})
while not self.start_event.is_set():
sleep(0.1)
return self.pid
@@ -83,19 +86,21 @@ class Cluster:
def stop(self) -> bool:
if not self.sentinel.is_alive():
return False
logger.info(_(f"Q Cluster {self.name} stopping."))
logger.info(_("Q Cluster %(name)s stopping.") % {"name": self.name})
self.stop_event.set()
self.sentinel.join()
logger.info(_(f"Q Cluster {self.name} has stopped."))
logger.info(_("Q Cluster %(name)s has stopped.") % {"name": self.name})
self.start_event = None
self.stop_event = None
return True
def sig_handler(self, signum, frame):
logger.debug(
_(
f'{current_process().name} got signal {Conf.SIGNAL_NAMES.get(signum, "UNKNOWN")}'
)
_("%(name)s got signal %(signal)s")
% {
"name": current_process().name,
"signal": Conf.SIGNAL_NAMES.get(signum, "UNKNOWN"),
}
)
self.stop()
@@ -138,7 +143,7 @@ class Sentinel:
start_event,
cluster_id,
broker=None,
timeout=Conf.TIMEOUT,
timeout=None,
start=True,
):
# Make sure we catch signals for the pool
@@ -155,7 +160,7 @@ class Sentinel:
self.start_event = start_event
self.pool_size = Conf.WORKERS
self.pool = []
self.timeout = timeout
self.timeout = timeout or Conf.TIMEOUT
self.task_queue = (
Queue(maxsize=Conf.QUEUE_LIMIT) if Conf.QUEUE_LIMIT else Queue()
)
@@ -166,6 +171,10 @@ class Sentinel:
if start:
self.start()
def queue_name(self):
# multi-queue: cluster name is (broker's) queue_name
return self.broker.list_key if self.broker else '--'
def start(self):
self.broker.ping()
self.spawn_cluster()
@@ -217,21 +226,49 @@ class Sentinel:
db.connections.close_all()
if process == self.monitor:
self.monitor = self.spawn_monitor()
logger.error(_(f"reincarnated monitor {process.name} after sudden death"))
logger.critical(
_("reincarnated monitor %(name)s after sudden death")
% {"name": process.name}
)
elif process == self.pusher:
self.pusher = self.spawn_pusher()
logger.error(_(f"reincarnated pusher {process.name} after sudden death"))
logger.critical(
_("reincarnated pusher %(name)s after sudden death")
% {"name": process.name}
)
else:
self.pool.remove(process)
self.spawn_worker()
if process.timer.value == 0:
# only need to terminate on timeout, otherwise we risk destabilizing the queues
# only need to terminate on timeout, otherwise we risk destabilizing
# the queues
task_name = ""
if psutil:
try:
process_name = psutil.Process(process.pid).name()
name_splits = process_name.split(" ")
task_name = name_splits[3] if len(name_splits) >= 4 and name_splits[2] == "processing" else ""
except psutil.NoSuchProcess:
pass
process.terminate()
logger.warning(_(f"reincarnated worker {process.name} after timeout"))
if task_name:
msg = (
_("reincarnated worker %(name)s after timeout while processing task %(task_name)s")
% {"name": process.name, "task_name": task_name}
)
else:
msg = (
_("reincarnated worker %(name)s after timeout")
% {"name": process.name}
)
logger.critical(msg)
elif int(process.timer.value) == -2:
logger.info(_(f"recycled worker {process.name}"))
logger.info(_("recycled worker %(name)s") % {"name": process.name})
else:
logger.error(_(f"reincarnated worker {process.name} after death"))
logger.critical(
_("reincarnated worker %(name)s after death")
% {"name": process.name}
)
self.reincarnations += 1
@@ -253,13 +290,18 @@ class Sentinel:
def guard(self):
logger.info(
_(
f"{current_process().name} guarding cluster {humanize(self.cluster_id.hex)}"
)
_("%(name)s guarding cluster %(cluster_name)s")
% {
"name": current_process().name,
"cluster_name": humanize(self.cluster_id.hex) + f" [{self.queue_name()}]",
}
)
self.start_event.set()
Stat(self).save()
logger.info(_(f"Q Cluster {humanize(self.cluster_id.hex)} running."))
logger.info(
_("Q Cluster %(cluster_name)s running.")
% {"cluster_name": humanize(self.cluster_id.hex) + f" [{self.queue_name()}]"}
)
counter = 0
cycle = Conf.GUARD_CYCLE # guard loop sleep in seconds
# Guard loop. Runs at least once
@@ -293,7 +335,7 @@ class Sentinel:
def stop(self):
Stat(self).save()
name = current_process().name
logger.info(_(f"{name} stopping cluster processes"))
logger.info(_("%(name)s stopping cluster processes") % {"name": name})
# Stopping pusher
self.event_out.set()
# Wait for it to stop
@@ -318,7 +360,7 @@ class Sentinel:
self.result_queue.close()
# Wait for the result queue to empty
self.result_queue.join_thread()
logger.info(_(f"{name} waiting for the monitor."))
logger.info(_("%(name)s waiting for the monitor.") % {"name": name})
# Wait for everything to close or time out
count = 0
if not self.timeout:
@@ -340,12 +382,18 @@ def pusher(task_queue: Queue, event: Event, broker: Broker = None):
"""
if not broker:
broker = get_broker()
logger.info(_(f"{current_process().name} pushing tasks at {current_process().pid}"))
proc_name = current_process().name
if setproctitle:
setproctitle.setproctitle(f"qcluster {proc_name} pusher")
logger.info(
_("%(name)s pushing tasks at %(id)s")
% {"name": proc_name, "id": current_process().pid}
)
while True:
try:
task_set = broker.dequeue()
except Exception as e:
logger.error(e, traceback.format_exc())
except Exception:
logger.exception("Failed to pull task from broker")
# broker probably crashed. Let the sentinel handle it.
sleep(10)
break
@@ -355,16 +403,19 @@ def pusher(task_queue: Queue, event: Event, broker: Broker = None):
# unpack the task
try:
task = SignedPackage.loads(task[1])
except (TypeError, BadSignature) as e:
logger.error(e, traceback.format_exc())
except (TypeError, BadSignature):
logger.exception("Failed to push task to queue")
broker.fail(ack_id)
continue
task["cluster"] = Conf.CLUSTER_NAME # save actual cluster name to orm task table
task["ack_id"] = ack_id
task_queue.put(task)
logger.debug(_(f"queueing from {broker.list_key}"))
logger.debug(
_("queueing from %(list_key)s") % {"list_key": broker.list_key}
)
if event.is_set():
break
logger.info(_(f"{current_process().name} stopped pushing tasks"))
logger.info(_("%(name)s stopped pushing tasks") % {"name": current_process().name})
def monitor(result_queue: Queue, broker: Broker = None):
@@ -375,8 +426,12 @@ def monitor(result_queue: Queue, broker: Broker = None):
"""
if not broker:
broker = get_broker()
name = current_process().name
logger.info(_(f"{name} monitoring at {current_process().pid}"))
proc_name = current_process().name
if setproctitle:
setproctitle.setproctitle(f"qcluster {proc_name} monitor")
logger.info(
_("%(name)s monitoring at %(id)s") % {"name": proc_name, "id": current_process().pid}
)
for task in iter(result_queue.get, "STOP"):
# save the result
if task.get("cached", False):
@@ -390,28 +445,44 @@ def monitor(result_queue: Queue, broker: Broker = None):
# signal execution done
post_execute.send(sender="django_q", task=task)
# log the result
info_name = get_func_repr(task['func'])
info_name = get_func_repr(task["func"])
if task["success"]:
# log success
logger.info(_(f"Processed '{info_name}' ({task['name']})"))
logger.info(
_("Processed '%(info_name)s' (%(task_name)s)")
% {"info_name": info_name, "task_name": task["name"]}
)
else:
# log failure
logger.error(_(f"Failed '{info_name}' ({task['name']}) - {task['result']}"))
logger.info(_(f"{name} stopped monitoring results"))
logger.error(
_("Failed '%(info_name)s' (%(task_name)s) - %(task_result)s")
% {
"info_name": info_name,
"task_name": task["name"],
"task_result": task["result"],
}
)
logger.info(_("%(name)s stopped monitoring results") % {"name": proc_name})
def worker(
task_queue: Queue, result_queue: Queue, timer: Value, timeout: int = Conf.TIMEOUT
):
"""
Takes a task from the task queue, tries to execute it and puts the result back in the result queue
Takes a task from the task queue, tries to execute it and puts the result back in
the result queue
:param timeout: number of seconds wait for a worker to finish.
:type task_queue: multiprocessing.Queue
:type result_queue: multiprocessing.Queue
:type timer: multiprocessing.Value
"""
proc_name = current_process().name
logger.info(_(f"{proc_name} ready for work at {current_process().pid}"))
logger.info(
_("%(proc_name)s ready for work at %(id)s")
% {"proc_name": proc_name, "id": current_process().pid}
)
if setproctitle:
setproctitle.setproctitle(f"qcluster {proc_name} idle")
task_count = 0
if timeout is None:
timeout = -1
@@ -420,13 +491,33 @@ def worker(
result = None
timer.value = -1 # Idle
task_count += 1
# Get the function from the task
func = task["func"]
func_name = get_func_repr(func)
logger.info(_(f"{proc_name} processing '{func_name}' ({task['name']})"))
f = task["func"]
# Log task creation and set process name
# Get the function from the task
func_name = get_func_repr(f)
task_name = task["name"]
task_desc = (
_("%(proc_name)s processing %(task_name)s '%(func_name)s'")
% {
"proc_name": proc_name,
"func_name": func_name,
"task_name": task_name,
}
)
if "group" in task:
task_desc += f" [{task['group']}]"
logger.info(task_desc)
if setproctitle:
proc_title = f"qcluster {proc_name} processing {task_name} '{func_name}'"
if "group" in task:
proc_title += f" [{task['group']}]"
setproctitle.setproctitle(proc_title)
# if it's not an instance try to get it from the string
if not callable(task["func"]):
if not callable(f):
# locate() returns None if f cannot be loaded
f = pydoc.locate(f)
close_old_django_connections()
timer_value = task.pop("timeout", timeout)
@@ -434,15 +525,19 @@ def worker(
pre_execute.send(sender="django_q", func=f, task=task)
# execute the payload
timer.value = timer_value # Busy
try:
if f is None:
# raise a meaningfull error if task["func"] is not a valid function
raise ValueError(f"Function {task['func']} is not defined")
res = f(*task["args"], **task["kwargs"])
result = (res, True)
except Exception:
result = (f"Could not process '{func_name}'. Check the location of the function and the args/kwargs.", False)
except Exception as e:
result = (f"{e} : {traceback.format_exc()}", False)
if error_reporter:
error_reporter.report()
if task.get("sync", False):
raise Exception(result)
raise
with timer.get_lock():
# Process result
task["result"] = result[0]
@@ -450,23 +545,14 @@ def worker(
task["stopped"] = timezone.now()
result_queue.put(task)
timer.value = -1 # Idle
if setproctitle:
setproctitle.setproctitle(f"qcluster {proc_name} idle")
# Recycle
if task_count == Conf.RECYCLE or rss_check():
timer.value = -2 # Recycled
break
logger.info(_(f"{proc_name} stopped doing work"))
logger.info(_("%(proc_name)s stopped doing work") % {"proc_name": proc_name})
def get_func_repr(func):
# convert func to string
if inspect.isfunction(func):
return f"{func.__module__}.{func.__name__}"
elif inspect.ismethod(func) and hasattr(func.__self__, '__name__'):
return (
f"{func.__self__.__module__}."
f"{func.__self__.__name__}.{func.__name__}"
)
else:
return str(func)
def save_task(task, broker: Broker):
"""
@@ -491,16 +577,22 @@ def save_task(task, broker: Broker):
try:
filters = {}
if Conf.SAVE_LIMIT_PER and Conf.SAVE_LIMIT_PER in {"group", "name", "func"} and Conf.SAVE_LIMIT_PER in task:
if (
Conf.SAVE_LIMIT_PER
and Conf.SAVE_LIMIT_PER in {"group", "name", "func"}
and Conf.SAVE_LIMIT_PER in task
):
value = task[Conf.SAVE_LIMIT_PER]
if Conf.SAVE_LIMIT_PER == "func":
value = get_func_repr(value)
filters[Conf.SAVE_LIMIT_PER] = value
database_to_use = {"using": Conf.ORM if Conf.ORM else Schedule.objects.db} if not Conf.HAS_REPLICA else {}
with db.transaction.atomic(**database_to_use):
with db.transaction.atomic(using=db.router.db_for_write(Success)):
last = Success.objects.filter(**filters).select_for_update().last()
if task["success"] and 0 < Conf.SAVE_LIMIT <= Success.objects.filter(**filters).count():
if (
task["success"]
and 0 < Conf.SAVE_LIMIT <= Success.objects.filter(**filters).count()
):
last.delete()
# check if this task has previous results
@@ -530,6 +622,7 @@ def save_task(task, broker: Broker):
hook=task.get("hook"),
args=task["args"],
kwargs=task["kwargs"],
cluster=task.get("cluster"),
started=task["started"],
stopped=task["stopped"],
result=task["result"],
@@ -537,8 +630,8 @@ def save_task(task, broker: Broker):
success=task["success"],
attempt_count=1,
)
except Exception as e:
logger.error(e)
except Exception:
logger.exception("Could not save task result")
def save_cached(task, broker: Broker):
@@ -589,8 +682,8 @@ def save_cached(task, broker: Broker):
)
# save the task
broker.cache.set(task_key, SignedPackage.dumps(task), timeout)
except Exception as e:
logger.error(e)
except Exception:
logger.exception("Could not save task result")
def scheduler(broker: Broker = None):
@@ -601,14 +694,16 @@ def scheduler(broker: Broker = None):
broker = get_broker()
close_old_django_connections()
try:
database_to_use = {"using": Conf.ORM if Conf.ORM else Schedule.objects.db} if not Conf.HAS_REPLICA else {}
with db.transaction.atomic(**database_to_use):
# Only default cluster will handler schedule with default(null) cluster
Q_default = db.models.Q(cluster__isnull=True) if Conf.CLUSTER_NAME == Conf.PREFIX else db.models.Q(pk__in=[])
with db.transaction.atomic(using=db.router.db_for_write(Schedule)):
for s in (
Schedule.objects.select_for_update()
.exclude(repeats=0)
.filter(next_run__lt=timezone.now())
.filter(
db.models.Q(cluster__isnull=True) | db.models.Q(cluster=Conf.PREFIX)
Q_default | db.models.Q(cluster=Conf.CLUSTER_NAME)
)
):
args = ()
@@ -621,8 +716,13 @@ def scheduler(broker: Broker = None):
except (SyntaxError, ValueError):
# else use the kwargs syntax
try:
parsed_kwargs = ast.parse(f"f({s.kwargs})").body[0].value.keywords
kwargs = {kwarg.arg: ast.literal_eval(kwarg.value) for kwarg in parsed_kwargs}
parsed_kwargs = (
ast.parse(f"f({s.kwargs})").body[0].value.keywords
)
kwargs = {
kwarg.arg: ast.literal_eval(kwarg.value)
for kwarg in parsed_kwargs
}
except (SyntaxError, ValueError):
kwargs = {}
if s.args:
@@ -631,46 +731,25 @@ def scheduler(broker: Broker = None):
if type(args) != tuple:
args = (args,)
q_options = kwargs.get("q_options", {})
if s.intended_date_kwarg:
kwargs[s.intended_date_kwarg] = s.next_run.isoformat()
if s.hook:
q_options["hook"] = s.hook
# set up the next run time
if s.schedule_type != s.ONCE:
next_run = s.next_run
while True:
if s.schedule_type == s.MINUTES:
next_run = next_run + timedelta(minutes=(s.minutes or 1))
elif s.schedule_type == s.HOURLY:
next_run = next_run + timedelta(hours=1)
elif s.schedule_type == s.DAILY:
next_run = next_run + timedelta(days=1)
elif s.schedule_type == s.WEEKLY:
next_run = next_run + timedelta(weeks=1)
elif s.schedule_type == s.MONTHLY:
next_run = add_months(next_run, 1)
elif s.schedule_type == s.QUARTERLY:
next_run = add_months(next_run, 3)
elif s.schedule_type == s.YEARLY:
next_run = add_years(next_run, 1)
elif s.schedule_type == s.CRON:
if not croniter:
raise ImportError(
_(
"Please install croniter to enable cron expressions"
)
)
next_run = croniter(s.cron, localtime()).get_next(datetime)
next_run = s.calculate_next_run(next_run)
if Conf.CATCH_UP or next_run > localtime():
break
s.next_run = next_run
s.repeats += -1
# send it to the cluster
scheduled_broker = broker
try:
scheduled_broker = get_broker(q_options["broker_name"])
except: # invalid broker_name or non existing broker with broker_name
pass
q_options["broker"] = scheduled_broker
# send it to the cluster; any cluster name is allowed in multi-queue scenarios
# because `broker_name` is confusing, using `cluster` name is recommended and takes precedence
q_options["cluster"] = s.cluster or q_options.get("cluster", q_options.pop("broker_name", None))
if q_options['cluster'] is None or q_options['cluster'] == Conf.CLUSTER_NAME:
q_options["broker"] = broker
q_options["group"] = q_options.get("group", s.name or s.id)
kwargs["q_options"] = q_options
s.task = django_q.tasks.async_task(s.func, *args, **kwargs)
@@ -678,14 +757,25 @@ def scheduler(broker: Broker = None):
if not s.task:
logger.error(
_(
f"{current_process().name} failed to create a task from schedule [{s.name or s.id}]"
"%(process_name)s failed to create a task from schedule "
"[%(schedule)s]"
)
% {
"process_name": current_process().name,
"schedule": s.name or s.id,
}
)
else:
logger.info(
_(
f"{current_process().name} created a task from schedule [{s.name or s.id}]"
"%(process_name)s created task %(task_name)s from schedule "
"[%(schedule)s]"
)
% {
"process_name": current_process().name,
"task_name": humanize(s.task),
"schedule": s.name or s.id,
}
)
# default behavior is to delete a ONCE schedule
if s.schedule_type == s.ONCE:
@@ -696,8 +786,8 @@ def scheduler(broker: Broker = None):
s.repeats = 0
# save the schedule
s.save()
except Exception as e:
logger.error(e)
except Exception:
logger.exception("Could not create task from schedule")
def close_old_django_connections():
@@ -724,12 +814,12 @@ def set_cpu_affinity(n: int, process_ids: list, actual: bool = not Conf.TESTING)
"""
# check if we have the psutil module
if not psutil:
logger.warning("Skipping cpu affinity because psutil was not found.")
logger.warning(_("Skipping cpu affinity because psutil was not found."))
return
# check if the platform supports cpu_affinity
if actual and not hasattr(psutil.Process(process_ids[0]), "cpu_affinity"):
logger.warning(
"Faking cpu affinity because it is not supported on this platform"
_("Faking cpu affinity because it is not supported on this platform")
)
actual = False
# get the available processors
@@ -750,7 +840,10 @@ def set_cpu_affinity(n: int, process_ids: list, actual: bool = not Conf.TESTING)
p = psutil.Process(pid)
if actual:
p.cpu_affinity(affinity)
logger.info(_(f"{pid} will use cpu {affinity}"))
logger.info(
_("%(pid)s will use cpu %(affinity)s")
% {"pid": pid, "affinity": affinity}
)
def rss_check():
@@ -760,10 +853,3 @@ def rss_check():
elif psutil:
return psutil.Process().memory_info().rss >= Conf.MAX_RSS * 1024
return False
def localtime() -> datetime:
"""Override for timezone.localtime to deal with naive times and local times"""
if settings.USE_TZ:
return timezone.localtime()
return datetime.now()

View File

@@ -27,6 +27,11 @@ try:
except ModuleNotFoundError:
resource = None
try:
import setproctitle
except ModuleNotFoundError:
setproctitle = None
class Conf:
"""
@@ -34,10 +39,22 @@ class Conf:
"""
try:
conf = settings.Q_CLUSTER
conf = settings.Q_CLUSTER.copy()
except AttributeError:
conf = {}
_Q_CLUSTER_NAME = os.getenv("Q_CLUSTER_NAME")
if _Q_CLUSTER_NAME and _Q_CLUSTER_NAME != conf.get("name") and \
_Q_CLUSTER_NAME != conf.get("cluster_name"):
conf["cluster_name"] = _Q_CLUSTER_NAME
alt_conf = conf.pop("ALT_CLUSTERS")
if isinstance(alt_conf, dict):
alt_conf = alt_conf.get(_Q_CLUSTER_NAME)
if isinstance(alt_conf, dict):
alt_conf.pop('name', None)
alt_conf.pop('cluster_name', None)
conf.update(alt_conf)
# Redis server configuration . Follows standard redis keywords
REDIS = conf.get("redis", {})
@@ -54,9 +71,6 @@ class Conf:
# ORM broker
ORM = conf.get("orm", None)
# ORM support for read/write replicas
HAS_REPLICA = conf.get("has_replica", False)
# Custom broker class
BROKER_CLASS = conf.get("broker_class", None)
@@ -68,12 +82,19 @@ class Conf:
MONGO_DB = conf.get("mongo_db", None)
# Name of the cluster or site. For when you run multiple sites on one redis server
# It's also the `salt` for signing OrmQ, and part of the Redis stats caching key
# For all clusters in one site, PREFIX should be the same value to be able to decrypt payloads
PREFIX = conf.get("name", "default")
# Support alternative cluster name to use multiple queues in one site.
# cluster name and queue name are interchangeable, same thing.
CLUSTER_NAME = conf.get("cluster_name", PREFIX)
# Log output level
LOG_LEVEL = conf.get("log_level", "INFO")
# Maximum number of successful tasks kept in the database. 0 saves everything. -1 saves none
# Maximum number of successful tasks kept in the database. 0 saves everything.
# -1 saves none
# Failures are always saved
SAVE_LIMIT = conf.get("save_limit", 250)
@@ -82,7 +103,13 @@ class Conf:
# Verify SAVE_LIMIT_PER is valid
if SAVE_LIMIT_PER not in ["group", "name", "func", None]:
warn(f"SAVE_LIMIT_PER ({SAVE_LIMIT_PER}) is not a valid option. Options are: 'group', 'name', 'func' and None. Default is None.")
warn(
_(
"SAVE_LIMIT_PER (%(option)s) is not a valid option. Options are: "
"'group', 'name', 'func' and None. Default is None."
)
% {"option": SAVE_LIMIT_PER}
)
# Guard loop sleep in seconds. Should be between 0 and 60 seconds.
GUARD_CYCLE = conf.get("guard_cycle", 0.5)
@@ -113,11 +140,12 @@ class Conf:
# Sets compression of redis packages
COMPRESSED = conf.get("compress", False)
# Number of tasks each worker can handle before it gets recycled. Useful for releasing memory
# Number of tasks each worker can handle before it gets recycled.
# Useful for releasing memory
RECYCLE = conf.get("recycle", 500)
# The maximum resident set size in kilobytes before a worker will recycle. Useful for limiting memory usage
# Not available on all platforms
# The maximum resident set size in kilobytes before a worker will recycle.
# Useful for limiting memory usage. Not available on all platforms
MAX_RSS = conf.get("max_rss", None)
# Number of seconds to wait for a worker to finish.
@@ -135,9 +163,10 @@ class Conf:
# Verify if retry and timeout settings are correct
if not TIMEOUT or (TIMEOUT > RETRY):
warn(
"""Retry and timeout are misconfigured. Set retry larger than timeout,
failure to do so will cause the tasks to be retriggered before completion.
See https://django-q2.readthedocs.io/en/master/configure.html#retry for details."""
"Retry and timeout are misconfigured. Set retry larger than timeout,"
"failure to do so will cause the tasks to be retriggered before completion."
"See https://django-q2.readthedocs.io/en/master/configure.html#retry "
"for details."
)
# Sets the amount of tasks the cluster will try to pop off the broker.
@@ -156,12 +185,14 @@ class Conf:
# The Django cache to use
CACHE = conf.get("cache", "default")
# Use the cache as result backend. Can be 'True' or an integer representing the global cache timeout.
# Use the cache as result backend. Can be 'True' or an integer representing the
# global cache timeout.
# i.e 'cached: 60' , will make all results go the cache and expire in 60 seconds.
CACHED = conf.get("cached", False)
# If set to False the scheduler won't execute tasks in the past.
# Instead it will run once and reschedule the next run in the future. Defaults to True.
# Instead it will run once and reschedule the next run in the future. Defaults to
# True.
CATCH_UP = conf.get("catch_up", True)
# Use the secret key for package signing
@@ -200,6 +231,11 @@ class Conf:
# to manage workarounds during testing
TESTING = conf.get("testing", False)
# Timezone for next_run, overrules Django timezone
TIME_ZONE = None
if settings.USE_TZ:
TIME_ZONE = conf.get("time_zone", settings.TIME_ZONE)
# logger
logger = logging.getLogger("django-q")
@@ -257,5 +293,6 @@ def get_ppid():
return psutil.Process(os.getpid()).ppid()
else:
raise OSError(
"Your OS does not support `os.getppid`. Please install `psutil` as an alternative provider."
"Your OS does not support `os.getppid`. Please install `psutil` as an "
"alternative provider."
)

View File

@@ -6,7 +6,12 @@ from django.core.signing import BadSignature, JSONSerializer, SignatureExpired
from django.core.signing import Signer as Sgnr
from django.core.signing import TimestampSigner as TsS
from django.core.signing import b64_decode, dumps
from django.utils import baseconv
try:
from django.core.signing import base62
except ImportError:
# For django < 4.0
from django.utils.baseconv import base62
from django.utils.crypto import constant_time_compare
from django.utils.encoding import force_bytes, force_str
@@ -32,7 +37,7 @@ def loads(
"""
# TimestampSigner.unsign() returns str but base64 and zlib compression
# operate on bytes.
base64d = force_bytes(TimestampSigner(key, salt=salt).unsign(s, max_age=max_age))
base64d = force_bytes(TimestampSigner(key=key, salt=salt).unsign(s, max_age=max_age))
decompress = False
if base64d[:1] == b".":
# It's compressed; uncompress it first
@@ -69,7 +74,7 @@ class TimestampSigner(Signer, TsS):
"""
result = super(TimestampSigner, self).unsign(value)
value, timestamp = result.rsplit(self.sep, 1)
timestamp = baseconv.base62.decode(timestamp)
timestamp = base62.decode(timestamp)
if max_age is not None:
if isinstance(max_age, datetime.timedelta):
max_age = max_age.total_seconds()

View File

@@ -337,12 +337,18 @@ class HumanHasher:
# Split `bytes` into `target` segments.
seg_size = length // target
segments = [bytes[i * seg_size : (i + 1) * seg_size] for i in range(target)]
# fmt: off
segments = [
bytes[i * seg_size : (i + 1) * seg_size] for i in range(target) # noqa: E203 E501
]
# fmt: on
# Catch any left-over bytes in the last segment.
segments[-1].extend(bytes[target * seg_size :])
segments[-1].extend(bytes[target * seg_size :]) # noqa: E203 E501
# Use a simple XOR checksum-like function for compression.
checksum = lambda bytes: reduce(operator.xor, bytes, 0)
def checksum(bytes):
return reduce(operator.xor, bytes, 0)
checksums = list(map(checksum, segments))
return checksums

View File

@@ -6,7 +6,7 @@ msgid ""
msgstr ""
"Project-Id-Version: \n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2021-01-15 15:53+0100\n"
"POT-Creation-Date: 2023-01-26 01:38+0000\n"
"PO-Revision-Date: 2018-08-05 18:28+0200\n"
"Last-Translator: Jonas Winkler\n"
"Language-Team: \n"
@@ -17,201 +17,202 @@ msgstr ""
"X-Generator: Poedit 2.1.1\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"
#: admin.py:40
#: admin.py:43
msgid "Resubmit selected tasks to queue"
msgstr "Ausgewählte Aufgaben erneut ausführen"
#: brokers/disque.py:59
msgid "No Disque nodes configured"
msgstr "Keine Disque-Knoten konfiguriert"
#: admin.py:107 models.py:293
#, fuzzy
#| msgid "Success"
msgid "success"
msgstr "erfolg"
#: brokers/disque.py:76
msgid "Could not connect to any Disque nodes"
msgstr "Konnte zu keinem Disque-Knoten verbinden"
#: admin.py:119 models.py:295
msgid "last_run"
msgstr ""
#: cluster.py:76
#, fuzzy, python-brace-format
#| msgid "Q Cluster-{} starting."
msgid "Q Cluster {self.name} starting."
msgstr "Q-Cluster {self.name} wird gestartet."
#: cluster.py:84
#, fuzzy, python-brace-format
#| msgid "Q Cluster-{} stopping."
msgid "Q Cluster {self.name} stopping."
msgstr "Q-Cluster {self.name} wird gestoppt."
#: cluster.py:79
#, python-format
msgid "Q Cluster %(name)s starting."
msgstr "Q-Cluster %(name)s wird gestartet."
#: cluster.py:87
#, fuzzy, python-brace-format
#| msgid "Q Cluster-{} has stopped."
msgid "Q Cluster {self.name} has stopped."
msgstr "Q-Cluster {self.name} wurde gestoppt."
#, fuzzy, python-format
#| msgid "Q Cluster-{} stopping."
msgid "Q Cluster %(name)s stopping."
msgstr "Q-Cluster {name} wird gestoppt."
#: cluster.py:95
msgid ""
"{current_process().name} got signal {Conf.SIGNAL_NAMES.get(signum, \"UNKNOWN"
"\")}"
msgstr ""
"{current_process().name} erhielt das Signal {Conf.SIGNAL_NAMES.get(signum, "
"\"UNKNOWN\")}"
#: cluster.py:90
#, python-format
msgid "Q Cluster %(name)s has stopped."
msgstr "Q-Cluster %(name)s wurde gestoppt."
#: cluster.py:216
#, fuzzy, python-brace-format
#| msgid "reincarnated monitor {} after sudden death"
msgid "reincarnated monitor {process.name} after sudden death"
msgstr "Monitor {process.name} wurde nach unerwartetem Absturz neu gestartet"
#: cluster.py:97
#, python-format
msgid "%(name)s got signal %(signal)s"
msgstr "%(name)s erhielt das Signal %(signal)s"
#: cluster.py:219
#, fuzzy, python-brace-format
#| msgid "reincarnated pusher {} after sudden death"
msgid "reincarnated pusher {process.name} after sudden death"
msgstr "Pusher {process.name} wurde nach unerwartetem Absturz neu gestartet"
#: cluster.py:226
#, fuzzy, python-brace-format
#| msgid "reincarnated worker {} after timeout"
msgid "reincarnated worker {process.name} after timeout"
msgstr "Worker {process.name} wurde nach Zeitüberschreitung neu gestartet"
#: cluster.py:228
#, fuzzy, python-brace-format
#| msgid "recycled worker {}"
msgid "recycled worker {process.name}"
msgstr "Worker {process.name} wurde wiederverwendet"
#: cluster.py:224
#, python-format
msgid "reincarnated monitor %(name)s after sudden death"
msgstr "Monitor %(name)s wurde nach unerwartetem Absturz neu gestartet"
#: cluster.py:230
#, fuzzy, python-brace-format
#| msgid "reincarnated worker {} after death"
msgid "reincarnated worker {process.name} after death"
msgstr "Worker {process.name} wurde nach unerwartetem Absturz neu gestartet"
#, python-format
msgid "reincarnated pusher %(name)s after sudden death"
msgstr "Pusher %(name)s wurde nach unerwartetem Absturz neu gestartet"
#: cluster.py:251
#: cluster.py:250
#, fuzzy, python-format
#| msgid "reincarnated worker %(name)s after timeout"
msgid ""
"{current_process().name} guarding cluster {humanize(self.cluster_id.hex)}"
"reincarnated worker %(name)s after timeout while processing task "
"%(task_name)s"
msgstr "Worker %(name)s wurde nach Zeitüberschreitung neu gestartet"
#: cluster.py:255
#, python-format
msgid "reincarnated worker %(name)s after timeout"
msgstr "Worker %(name)s wurde nach Zeitüberschreitung neu gestartet"
#: cluster.py:260
#, python-format
msgid "recycled worker %(name)s"
msgstr "Worker %(name)s wurde wiederverwendet"
#: cluster.py:263
#, python-format
msgid "reincarnated worker %(name)s after death"
msgstr "Worker %(name)s wurde nach unerwartetem Absturz neu gestartet"
#: cluster.py:287
#, python-format
msgid "%(name)s guarding cluster %(cluster_name)s"
msgstr "%(name)s beschützt das Cluster %(cluster_name)s"
#: cluster.py:296
#, python-format
msgid "Q Cluster %(cluster_name)s running."
msgstr "Q-Cluster %(cluster_name)s läuft."
#: cluster.py:332
#, python-format
msgid "%(name)s stopping cluster processes"
msgstr "%(name)s hält Cluster-Prozesse an"
#: cluster.py:357
#, python-format
msgid "%(name)s waiting for the monitor."
msgstr "%(name)s wartet auf den Monitor."
#: cluster.py:383
#, fuzzy, python-format
#| msgid "%(process_name)s pushing tasks at %(id)s"
msgid "%(name)s pushing tasks at %(id)s"
msgstr "%(process_name)s veröffentlicht Aufagaben auf %(id)s"
#: cluster.py:407
#, python-format
msgid "queueing from %(list_key)s"
msgstr "Einreihen von %(list_key)s"
#: cluster.py:411
#, python-format
msgid "%(name)s stopped pushing tasks"
msgstr "%(name)s veröffentlicht keine Aufgaben mehr"
#: cluster.py:426
#, python-format
msgid "%(name)s monitoring at %(id)s"
msgstr "%(name)s beobachtet auf %(id)s"
#: cluster.py:445
#, python-format
msgid "Processed '%(info_name)s' (%(task_name)s)"
msgstr "[%(task_name)s] - '%(info_name)s' wurde verarbeitet"
#: cluster.py:451
#, python-format
msgid "Failed '%(info_name)s' (%(task_name)s) - %(task_result)s"
msgstr "'%(info_name)s' (%(task_name)s) ist fehlgeschlagen - %(task_result)s"
#: cluster.py:458
#, python-format
msgid "%(name)s stopped monitoring results"
msgstr "%(name)s überwacht keine Ergebnisse mehr"
#: cluster.py:474
#, python-format
msgid "%(proc_name)s ready for work at %(id)s"
msgstr "%(proc_name)s ist bereit für Arbeit auf %(id)s"
#: cluster.py:494
#, fuzzy, python-format
#| msgid "%(proc_name)s processing '%(func_name)s' (%(task_name)s)"
msgid "%(proc_name)s processing %(task_name)s '%(func_name)s'"
msgstr "%(proc_name)s verarbeitet '%(func_name)s' (%(task_name)s)"
#: cluster.py:546
#, python-format
msgid "%(proc_name)s stopped doing work"
msgstr "%(proc_name)s hat die Arbeit beendet"
#: cluster.py:751
#, python-format
msgid "%(process_name)s failed to create a task from schedule [%(schedule)s]"
msgstr ""
"{current_process().name} beschützt das Cluster {humanize(self.cluster_id."
"hex)}"
"%(process_name)s konnte keine Aufgabe von Zeitplan [%(schedule)s] erstellen"
#: cluster.py:256
msgid "Q Cluster {humanize(self.cluster_id.hex)} running."
msgstr "Q-Cluster {humanize(self.cluster_id.hex)} ist bereit."
#: cluster.py:290
#, fuzzy, python-brace-format
#| msgid "{} stopping cluster processes"
msgid "{name} stopping cluster processes"
msgstr "{name} hält Cluster-Prozesse an"
#: cluster.py:315
#, fuzzy, python-brace-format
#| msgid "{} waiting for the monitor."
msgid "{name} waiting for the monitor."
msgstr "{name} wartet auf den Monitor."
#: cluster.py:337
msgid "{current_process().name} pushing tasks at {current_process().pid}"
msgstr ""
"{current_process().name} veröffentlicht Aufagaben auf {current_process().pid}"
#: cluster.py:358
#, fuzzy, python-brace-format
#| msgid "queueing from {}"
msgid "queueing from {broker.list_key}"
msgstr "Einreihen von {broker.list_key}"
#: cluster.py:361
#, fuzzy
#| msgid "{} stopped pushing tasks"
msgid "{current_process().name} stopped pushing tasks"
msgstr "{current_process().name} veröffentlicht keine Aufgaben mehr"
#: cluster.py:373
#, fuzzy
#| msgid "{} monitoring at {}"
msgid "{name} monitoring at {current_process().pid}"
msgstr "{name} beobachtet auf {current_process().pid}"
#: cluster.py:387
#, fuzzy
#| msgid "Processed [{}]"
msgid "Processed [{task['name']}]"
msgstr "[{task['name']}] wurde verarbeitet"
#: cluster.py:390
msgid "Failed [{task['name']}] - {task['result']}"
msgstr "[{task['name']}] ist fehlgeschlagen - {task['result']}"
#: cluster.py:391
#, fuzzy, python-brace-format
#| msgid "{} stopped monitoring results"
msgid "{name} stopped monitoring results"
msgstr "{name} überwacht keine Ergebnisse mehr"
#: cluster.py:405
#, fuzzy
#| msgid "{} ready for work at {}"
msgid "{name} ready for work at {current_process().pid}"
msgstr "{name} ist bereit für Arbeit auf {current_process().pid}"
#: cluster.py:415
#, fuzzy
#| msgid "{} processing [{}]"
msgid "{name} processing [{task[\"name\"]}]"
msgstr "{name} verarbeitet [{task[\"name\"]}]"
#: cluster.py:455
#, fuzzy, python-brace-format
#| msgid "{} stopped doing work"
msgid "{name} stopped doing work"
msgstr "{name} hat die Arbeit beendet"
#: cluster.py:619 models.py:143
msgid "Please install croniter to enable cron expressions"
msgstr "Bitte installieren Sie croniter, um Cron-Ausdrücke zu aktivieren"
#: cluster.py:644
#, fuzzy
#| msgid "{} failed to create a task from schedule [{}]"
#: cluster.py:762
#, fuzzy, python-format
#| msgid "%(process_name)s created a task from schedule [%(schedule)s]"
msgid ""
"{current_process().name} failed to create a task from schedule [{s.name or s."
"id}]"
"%(process_name)s created task %(task_name)s from schedule [%(schedule)s]"
msgstr ""
"{current_process().name} konnte keine Aufgabe von Zeitplan [{s.name or s."
"id}] erstellen"
"%(process_name)s hat eine Aufgabe des Zeitplans [%(schedule)s] erstellt"
#: cluster.py:650
#, fuzzy
#| msgid "{} created a task from schedule [{}]"
#: cluster.py:808
msgid "Skipping cpu affinity because psutil was not found."
msgstr "Cpu-Affinität wird übersprungen, da psutil nicht gefunden wurde."
#: cluster.py:813
msgid "Faking cpu affinity because it is not supported on this platform"
msgstr ""
"Vortäuschen von CPU-Affinität, da diese auf dieser Plattform nicht "
"unterstützt wird"
#: cluster.py:835
#, python-format
msgid "%(pid)s will use cpu %(affinity)s"
msgstr "%(pid)s wird CPU %(affinity)s benutzen"
#: conf.py:90
#, python-format
msgid ""
"{current_process().name} created a task from schedule [{s.name or s.id}]"
"SAVE_LIMIT_PER (%(option)s) is not a valid option. Options are: 'group', "
"'name', 'func' and None. Default is None."
msgstr ""
"{current_process().name} hat eine Aufgabe des Zeitplans [{s.name or s.id}] "
"erstellt"
#: cluster.py:716
#, fuzzy, python-brace-format
#| msgid "{} will use cpu {}"
msgid "{pid} will use cpu {affinity}"
msgstr "{pid} wird CPU {affinity} benutzen"
"SAVE_LIMIT_PER (%(option)s) ist keine gültige Option. Optionen sind: "
"'group', 'name', 'func' und None. Standard ist None."
#. Translators: Cluster status descriptions
#: conf.py:184
#: conf.py:207
msgid "Starting"
msgstr "Wird gestartet"
#: conf.py:185
#: conf.py:208
msgid "Working"
msgstr "Arbeitet"
#: conf.py:186
#: conf.py:209
msgid "Idle"
msgstr "Leerlauf"
#: conf.py:187
#: conf.py:210
msgid "Stopped"
msgstr "Gestoppt"
#: conf.py:188
#: conf.py:211
msgid "Stopping"
msgstr "Wird gestoppt"
@@ -225,223 +226,277 @@ msgstr "Startet ein Django-Q-Cluster."
msgid "General information over all clusters."
msgstr "Allgemeine Informationen über alle Cluster"
#. Translators: help text for qmemory management command
#: management/commands/qmemory.py:9
msgid "Monitors Q Cluster memory usage"
msgstr "Überwacht die Speichernutzung von Q Cluster"
#. Translators: help text for qmonitor management command
#: management/commands/qmonitor.py:9
msgid "Monitors Q Cluster activity"
msgstr "Q-Cluster aktiv überwachen"
#: models.py:118
#: models.py:125
msgid "Successful task"
msgstr "Erfolgreiche Aufgabe"
#: models.py:119
#: models.py:126
msgid "Successful tasks"
msgstr "Erfolgreiche Aufgaben"
#: models.py:134
#: models.py:141
msgid "Failed task"
msgstr "Fehlgeschlagene Aufgabe"
#: models.py:135
#: models.py:142
msgid "Failed tasks"
msgstr "Fehlgeschlagene Aufgaben"
#: models.py:159
#: models.py:150 models.py:234
msgid "Please install croniter to enable cron expressions"
msgstr "Bitte installieren Sie croniter, um Cron-Ausdrücke zu aktivieren"
#: models.py:170
msgid "e.g. 1, 2, 'John'"
msgstr "zum Beispiel 1, 2, 'John'"
#: models.py:161
#: models.py:172
msgid "e.g. x=1, y=2, name='John'"
msgstr "zum Beispiel x=1, y=2, name='John'"
#: models.py:173
#: models.py:186
msgid "Once"
msgstr "Einmal"
#: models.py:174
#: models.py:187
msgid "Minutes"
msgstr "Minuten"
#: models.py:175
#: models.py:188
msgid "Hourly"
msgstr "Stündlich"
#: models.py:176
#: models.py:189
msgid "Daily"
msgstr "Täglich"
#: models.py:177
#: models.py:190
msgid "Weekly"
msgstr "Wöchentlich"
#: models.py:178
#: models.py:191
msgid "Biweekly"
msgstr "Zweiwöchentlich"
#: models.py:192
msgid "Monthly"
msgstr "Monatlich"
#: models.py:179
#: models.py:193
msgid "Bimonthly"
msgstr "Zweimonatlich"
#: models.py:194
msgid "Quarterly"
msgstr "Vierteljährlich"
#: models.py:180
#: models.py:195
msgid "Yearly"
msgstr "Jährlich"
#: models.py:181
#: models.py:196
msgid "Cron"
msgstr "Cron"
#: models.py:184
#: models.py:199
msgid "Schedule Type"
msgstr "Zeitplan-Typ"
#: models.py:187
#: models.py:202
msgid "Number of minutes for the Minutes type"
msgstr "Anzahl Minuten für den Typ 'Minuten'"
#: models.py:190
#: models.py:205
msgid "Repeats"
msgstr "Wiederhohlungen"
#: models.py:190
#: models.py:205
msgid "n = n times, -1 = forever"
msgstr "n = n mal, -1 = für immer"
#: models.py:193
#: models.py:208
msgid "Next Run"
msgstr "Nächste Ausführung"
#: models.py:200
#: models.py:215
msgid "Cron expression"
msgstr "Cron-Ausdruck"
#: models.py:226
#: models.py:224
msgid "Name of kwarg to pass intended schedule date"
msgstr "Name des zu passierenden Kwargs vorgesehenes Datum"
#: models.py:299
msgid "Scheduled task"
msgstr "Geplante Aufgabe"
#: models.py:227
#: models.py:300
msgid "Scheduled tasks"
msgstr "Geplante Aufgaben"
#: models.py:250
#: models.py:326
msgid "Queued task"
msgstr "Eingereihte Aufgabe"
#: models.py:251
#: models.py:327
msgid "Queued tasks"
msgstr "Eingereihte Aufgaben"
#: monitor.py:35
#: monitor.py:64 monitor.py:348
msgid "Host"
msgstr "Host"
#: monitor.py:39
#: monitor.py:68 monitor.py:352 monitor.py:459
msgid "Id"
msgstr "Id"
#: monitor.py:43
#: monitor.py:72
msgid "State"
msgstr "Status"
#: monitor.py:47
#: monitor.py:76
msgid "Pool"
msgstr "Pool"
#: monitor.py:51
#: monitor.py:80
msgid "TQ"
msgstr "TQ"
#: monitor.py:55
#: monitor.py:84
msgid "RQ"
msgstr "RQ"
#: monitor.py:59
#: monitor.py:88
msgid "RC"
msgstr "RC"
#: monitor.py:63
#: monitor.py:92
msgid "Up"
msgstr "Up"
#: monitor.py:143 monitor.py:247
#: monitor.py:172 monitor.py:286
msgid "Queued"
msgstr "Eingereiht"
#: monitor.py:151
#: monitor.py:180
msgid "Success"
msgstr "Erfolg"
#: monitor.py:161 monitor.py:255
#: monitor.py:190 monitor.py:294
msgid "Failures"
msgstr "Fehlschläge"
#: monitor.py:172
#: monitor.py:201 monitor.py:498
msgid "[Press q to quit]"
msgstr "[Drücken Sie q zum Beenden]"
#: monitor.py:191
#: monitor.py:227
msgid "day"
msgstr "Tag"
#: monitor.py:212
#: monitor.py:248
msgid "second"
msgstr "Sekunde"
#: monitor.py:215
#: monitor.py:251
msgid "minute"
msgstr "Minute"
#: monitor.py:218
#: monitor.py:254
msgid "hour"
msgstr "Stunde"
#: monitor.py:228
msgid ""
"-- {Conf.PREFIX.capitalize()} { \".\".join(str(v) for v in VERSION)} on "
"{broker.info()} --"
msgstr ""
"-- {Conf.PREFIX.capitalize()} { \".\".join(str(v) for v in VERSION)} auf "
"{broker.info()} --"
#: monitor.py:263
#, python-format
msgid "-- %(prefix)s %(version)s on %(info)s --"
msgstr "-- %(prefix)s %(version)s auf %(info)s --"
#: monitor.py:234
#: monitor.py:273
msgid "Clusters"
msgstr "Cluster"
#: monitor.py:238
#: monitor.py:277
msgid "Workers"
msgstr "Arbeiter"
#: monitor.py:242
#: monitor.py:281
msgid "Restarts"
msgstr "Neustarts"
#: monitor.py:251
#: monitor.py:290
msgid "Successes"
msgstr "Erfolge"
#: monitor.py:260
#: monitor.py:299
msgid "Schedules"
msgstr "Zeitpläne"
#: monitor.py:264
#, fuzzy, python-brace-format
#| msgid "Tasks/{}"
msgid "Tasks/{per}"
msgstr "Aufgaben/{per}"
#: monitor.py:303
#, python-format
msgid "Tasks/%(per)s"
msgstr "Aufgaben/%(per)s"
#: monitor.py:268
#: monitor.py:307
msgid "Avg time"
msgstr "Durchschnittl. Zeit"
#: monitor.py:357
msgid "Available (%)"
msgstr "Verfügbar (%)"
#: monitor.py:363
msgid "Available (MB)"
msgstr "Verfügbar (MB)"
#: monitor.py:368
msgid "Total (MB)"
msgstr "Insgesamt (MB)"
#: monitor.py:373
msgid "Sentinel (MB)"
msgstr "Sentinel (MB)"
#: monitor.py:379
msgid "Monitor (MB)"
msgstr "Monitor (MB)"
#: monitor.py:385
msgid "Workers (MB)"
msgstr "Arbeiter (MB)"
#: monitor.py:487
#, python-format
msgid "Available lowest (): %(memory_percent)s ((at)s)"
msgstr "Niedrigste verfügbar (): %(memory_percent)s ((at)s)"
#: monitor.py:509
msgid "No clusters appear to be running."
msgstr "Es scheinen keine Cluster zu laufen."
#: signals.py:22
#, fuzzy, python-brace-format
#| msgid "malformed return hook '{}' for [{}]"
msgid "malformed return hook '{instance.hook}' for [{instance.name}]"
msgstr "Ungültiger Return-Hook '{instance.hook}' für [{instance.name}]"
#, python-format
msgid "malformed return hook '%(hook)s' for [%(name)s]"
msgstr "Ungültiger Return-Hook '%(hook)s' für [%(name)s]"
#: signals.py:30
#, fuzzy
#| msgid "return hook {} failed on [{}] because {}"
msgid ""
"return hook {instance.hook} failed on [{instance.name}] because {str(e)}"
msgstr ""
"Return-Hook {instance.hook} für [{instance.name}] ist gescheitert: {str(e)}"
#, python-format
msgid "return hook %(hook)s failed on [%(name)s] because %(error)s"
msgstr "Return-Hook %(hook)s für [%(name)s] ist gescheitert: %(error)s"
#, python-format
#~ msgid ""
#~ "Could not process '%(func_name)s'. Check the location of the function and "
#~ "the args/kwargs."
#~ msgstr ""
#~ "Konnte '%(func_name)s' nicht verarbeiten. Überprüfen Sie den Ort der "
#~ "Funktion und die args/kwargs."

View File

@@ -6,349 +6,504 @@ msgid ""
msgstr ""
"Project-Id-Version: \n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2018-08-05 15:48+0200\n"
"POT-Creation-Date: 2023-01-26 01:38+0000\n"
"PO-Revision-Date: 2018-08-05 18:28+0200\n"
"Last-Translator: Thierry BOULOGNE <contact@tng-concepts.com>\n"
"Language-Team: \n"
"Language: fr-FR\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"X-Generator: Poedit 2.1.1\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"
"Last-Translator: Thierry BOULOGNE <contact@tng-concepts.com>\n"
"Language: fr-FR\n"
#: admin.py:48
#: admin.py:43
msgid "Resubmit selected tasks to queue"
msgstr "Resoumettre les tâches sélectionnées à la file d'attente"
#: cluster.py:54
msgid "Q Cluster-{} starting."
msgstr "Démarrage de Q Cluster-{}."
#: admin.py:107 models.py:293
#, fuzzy
#| msgid "Success"
msgid "success"
msgstr "succès"
#: cluster.py:62
msgid "Q Cluster-{} stopping."
msgstr "Arrêt de Q Cluster-{}."
#: admin.py:119 models.py:295
msgid "last_run"
msgstr ""
#: cluster.py:65
msgid "Q Cluster-{} has stopped."
msgstr "Q Cluster-{} a été arrêté."
#: cluster.py:79
#, python-format
msgid "Q Cluster %(name)s starting."
msgstr "Démarrage de Q Cluster-%(name)s."
#: cluster.py:71
msgid "{} got signal {}"
msgstr "{} à reçu le signal {}"
#: cluster.py:87
#, python-format
msgid "Q Cluster %(name)s stopping."
msgstr "Arrêt de Q Cluster-%(name)s."
#: cluster.py:169
msgid "reincarnated monitor {} after sudden death"
msgstr "moniteur réintégré {} après un arrêt intempestif"
#: cluster.py:90
#, python-format
msgid "Q Cluster %(name)s has stopped."
msgstr "Q Cluster-%(name)s a été arrêté."
#: cluster.py:172
msgid "reincarnated pusher {} after sudden death"
msgstr "pousseur réintégré {} après un arrêt intempestif"
#: cluster.py:97
#, python-format
msgid "%(name)s got signal %(signal)s"
msgstr "%(name)s à reçu le signal %(signal)s"
#: cluster.py:179
msgid "reincarnated worker {} after timeout"
msgstr "processus réintégré {} après un arrêt une attente trop longue"
#: cluster.py:224
#, python-format
msgid "reincarnated monitor %(name)s after sudden death"
msgstr "surveillant %(name)s réincarné après un arrêt intempestif"
#: cluster.py:181
msgid "recycled worker {}"
msgstr "processus recyclé"
#: cluster.py:230
#, python-format
msgid "reincarnated pusher %(name)s after sudden death"
msgstr "répartiteur %(name)s réincarné après un arrêt intempestif"
#: cluster.py:183
msgid "reincarnated worker {} after death"
msgstr "processus réintégré {} après arrêt"
#: cluster.py:250
#, python-format
msgid ""
"reincarnated worker %(name)s after timeout while processing task "
"%(task_name)s"
msgstr ""
"processus %(name)s réincarné, délai de traitement dépassé pour la tâche "
"%(task_name)s"
#: cluster.py:202
msgid "{} guarding cluster at {}"
msgstr "{} surveillance du cluster à {}"
#: cluster.py:255
#, python-format
msgid "reincarnated worker %(name)s after timeout"
msgstr "processus %(name)s réincarné, délai de traitement dépassé"
#: cluster.py:205
msgid "Q Cluster-{} running."
msgstr "Q Cluster-{} en cours d'exécution."
#: cluster.py:260
#, python-format
msgid "recycled worker %(name)s"
msgstr "processus recyclé %(name)s"
#: cluster.py:239
msgid "{} stopping cluster processes"
msgstr "{} arrêt des processus de cluster"
#: cluster.py:263
#, python-format
msgid "reincarnated worker %(name)s after death"
msgstr "processus réintégré %(name)s après arrêt"
#: cluster.py:264
msgid "{} waiting for the monitor."
msgstr "{} en attente du moniteur."
#: cluster.py:287
#, python-format
msgid "%(name)s guarding cluster %(cluster_name)s"
msgstr "%(name)s surveillance du cluster à %(cluster_name)s"
#: cluster.py:285
msgid "{} pushing tasks at {}"
msgstr "{} tâche envoyé à {}"
#: cluster.py:296
#, python-format
msgid "Q Cluster %(cluster_name)s running."
msgstr "Démarrage de Q Cluster-%(cluster_name)s."
#: cluster.py:306
msgid "queueing from {}"
msgstr "mise en file d'attente de {}"
#: cluster.py:309
msgid "{} stopped pushing tasks"
msgstr "{} a cessé de pousser les tâches"
#: cluster.py:320
msgid "{} monitoring at {}"
msgstr "{} Surveillance de {}"
#: cluster.py:334
msgid "Processed [{}]"
msgstr "Traitement de [{}]"
#: cluster.py:337
msgid "Failed [{}] - {}"
msgstr "Echec [{}] - {}"
#: cluster.py:338
msgid "{} stopped monitoring results"
msgstr "{} arrêt des résultats de surveillance"
#: cluster.py:349
msgid "{} ready for work at {}"
msgstr "{} prêt pour le travail à {}"
#: cluster.py:332
#, python-format
msgid "%(name)s stopping cluster processes"
msgstr "%(name)s arrêt des processus du cluster"
#: cluster.py:357
msgid "{} processing [{}]"
msgstr "{} en cours de traitement [{}]"
#, python-format
msgid "%(name)s waiting for the monitor."
msgstr "%(name)s en attente du surveillant."
#: cluster.py:398
msgid "{} stopped doing work"
msgstr "{} arrêté de travailler"
#: cluster.py:383
#, python-format
msgid "%(name)s pushing tasks at %(id)s"
msgstr "%(name)s répartit les tâches %(id)s"
#: cluster.py:543
msgid "{} failed to create a task from schedule [{}]"
msgstr "{} Echec de la création d'une tâche à partir de Schedule [{}]"
#: cluster.py:407
#, python-format
msgid "queueing from %(list_key)s"
msgstr "mise en file d'attente de %(list_key)s"
#: cluster.py:547
msgid "{} created a task from schedule [{}]"
msgstr "{} a créé une tâche à partir de Schedule [{}]"
#: cluster.py:411
#, python-format
msgid "%(name)s stopped pushing tasks"
msgstr "%(name)s a cessé de répartir les tâches"
#: cluster.py:595
msgid "{} will use cpu {}"
msgstr "{} utilisera le CPU {}"
#: cluster.py:426
#, python-format
msgid "%(name)s monitoring at %(id)s"
msgstr "%(name)s surveille les résultats %(id)s"
#: conf.py:169
#: cluster.py:445
#, python-format
msgid "Processed '%(info_name)s' (%(task_name)s)"
msgstr "traité '%(info_name)s' (%(task_name)s)"
#: cluster.py:451
#, python-format
msgid "Failed '%(info_name)s' (%(task_name)s) - %(task_result)s"
msgstr "Manqué '%(info_name)s' (%(task_name)s) - %(task_result)s"
#: cluster.py:458
#, python-format
msgid "%(name)s stopped monitoring results"
msgstr "%(name)s a cessé de de surveiller les résultats"
#: cluster.py:474
#, python-format
msgid "%(proc_name)s ready for work at %(id)s"
msgstr "%(proc_name)s prêt pour le travail à %(id)s"
#: cluster.py:494
#, fuzzy, python-format
msgid "%(proc_name)s processing %(task_name)s '%(func_name)s'"
msgstr "%(proc_name)s exécute %(task_name)s '%(func_name)s'"
#: cluster.py:546
#, python-format
msgid "%(proc_name)s stopped doing work"
msgstr "%(proc_name)s a cessé de travailler"
#: cluster.py:751
#, python-format
msgid "%(process_name)s failed to create a task from schedule [%(schedule)s]"
msgstr ""
"%(process_name)s Echec de la création d'une tâche à partir de Schedule "
"[%(schedule)s]"
#: cluster.py:762
#, python-format
msgid ""
"%(process_name)s created task %(task_name)s from schedule [%(schedule)s]"
msgstr ""
"%(process_name)s a créé la tâche %(task_name)s à partir de Schedule "
"[%(schedule)s]"
#: cluster.py:808
msgid "Skipping cpu affinity because psutil was not found."
msgstr "L'affinité cpu ne sera pas définie car psutil n'a pas été trouvé."
#: cluster.py:813
msgid "Faking cpu affinity because it is not supported on this platform"
msgstr ""
"Simulation de l'affinité cpu parce qu'elle n'est pas supportée sur cette "
"plateforme."
#: cluster.py:835
#, python-format
msgid "%(pid)s will use cpu %(affinity)s"
msgstr "%(pid)s utilisera le CPU %(affinity)s"
#: conf.py:90
#, python-format
msgid ""
"SAVE_LIMIT_PER (%(option)s) is not a valid option. Options are: 'group', "
"'name', 'func' and None. Default is None."
msgstr ""
"SAVE_LIMIT_PER (%(option)s) n'est pas une option valide. Les options sont : "
"'group', 'name', 'func' et None. La valeur par défaut est None."
#. Translators: Cluster status descriptions
#: conf.py:207
msgid "Starting"
msgstr "Démarrage"
#: conf.py:170
#: conf.py:208
msgid "Working"
msgstr "Actif"
#: conf.py:171
#: conf.py:209
msgid "Idle"
msgstr "En attente"
#: conf.py:172
#: conf.py:210
msgid "Stopped"
msgstr "Arrêté"
#: conf.py:173
#: conf.py:211
msgid "Stopping"
msgstr "En cours darrêt"
#. Translators: help text for qcluster management command
#: management/commands/qcluster.py:9
msgid "Starts a Django Q Cluster."
msgstr "Démarre un cluster Django Q."
#. Translators: help text for qinfo management command
#: management/commands/qinfo.py:11
msgid "General information over all clusters."
msgstr "Informations générales sur tous les clusters."
#. Translators: help text for qmemory management command
#: management/commands/qmemory.py:9
#, fuzzy
#| msgid "Monitors Q Cluster activity"
msgid "Monitors Q Cluster memory usage"
msgstr "Surveille l'utilisation mémoire du Q cluster"
#. Translators: help text for qmonitor management command
#: management/commands/qmonitor.py:9
msgid "Monitors Q Cluster activity"
msgstr "Activité du cluster Moniteur Q"
msgstr "Surveille l'activité de Q cluster"
#: models.py:104
#: models.py:125
msgid "Successful task"
msgstr "Tâche réussie"
#: models.py:105
#: models.py:126
msgid "Successful tasks"
msgstr "Tâches réussies"
#: models.py:121
#: models.py:141
msgid "Failed task"
msgstr "Tâche échoué"
#: models.py:122
#: models.py:142
msgid "Failed tasks"
msgstr "Tâches échouées"
#: models.py:131
#: models.py:150 models.py:234
msgid "Please install croniter to enable cron expressions"
msgstr "Veuillez installer croniter pour activer les expressions cron."
#: models.py:170
msgid "e.g. 1, 2, 'John'"
msgstr "ex. 1, 2, Jean"
#: models.py:132
#: models.py:172
msgid "e.g. x=1, y=2, name='John'"
msgstr "p. ex. x = 1, y = 2, Nom = Jean"
#: models.py:142
#: models.py:186
msgid "Once"
msgstr "Une fois"
#: models.py:143
#: models.py:187
msgid "Minutes"
msgstr "Minutes"
#: models.py:144
#: models.py:188
msgid "Hourly"
msgstr "Toutes les heures"
#: models.py:145
#: models.py:189
msgid "Daily"
msgstr "Quotidien"
#: models.py:146
#: models.py:190
msgid "Weekly"
msgstr "Hebdomadaire"
#: models.py:147
#: models.py:191
#, fuzzy
#| msgid "Weekly"
msgid "Biweekly"
msgstr "Bihebdomadaire"
#: models.py:192
msgid "Monthly"
msgstr "Mensuel"
#: models.py:148
#: models.py:193
#, fuzzy
#| msgid "Monthly"
msgid "Bimonthly"
msgstr "Bimestriel"
#: models.py:194
msgid "Quarterly"
msgstr "Tous les quart-dheure"
#: models.py:149
#: models.py:195
msgid "Yearly"
msgstr "Annuel"
#: models.py:151
#: models.py:196
msgid "Cron"
msgstr "Cron"
#: models.py:199
msgid "Schedule Type"
msgstr "Type de plannification"
#: models.py:153
#: models.py:202
msgid "Number of minutes for the Minutes type"
msgstr "Nombre de minutes pour le type de minutes"
#: models.py:154
#: models.py:205
msgid "Repeats"
msgstr "Répéter"
#: models.py:154
#: models.py:205
msgid "n = n times, -1 = forever"
msgstr "n = n fois,-1 = Toujours"
#: models.py:155
#: models.py:208
msgid "Next Run"
msgstr "Prochaine exécution"
#: models.py:180
#: models.py:215
msgid "Cron expression"
msgstr "Expression Cron"
#: models.py:224
msgid "Name of kwarg to pass intended schedule date"
msgstr "Nom du kwarg à passer Date prévue de l'horaire"
#: models.py:299
msgid "Scheduled task"
msgstr "Tâche planifiée"
#: models.py:181
#: models.py:300
msgid "Scheduled tasks"
msgstr "Tâches planifiées"
#: models.py:204
#: models.py:326
msgid "Queued task"
msgstr "Tâche en file d'attente"
#: models.py:205
#: models.py:327
msgid "Queued tasks"
msgstr "Tâches en file d'attente"
#: monitor.py:33
#: monitor.py:64 monitor.py:348
msgid "Host"
msgstr "Hôte"
#: monitor.py:34
#: monitor.py:68 monitor.py:352 monitor.py:459
msgid "Id"
msgstr "Id"
#: monitor.py:35
#: monitor.py:72
msgid "State"
msgstr "Statut"
#: monitor.py:36
#: monitor.py:76
msgid "Pool"
msgstr "Piscine"
#: monitor.py:37
#: monitor.py:80
msgid "TQ"
msgstr "TQ"
#: monitor.py:38
#: monitor.py:84
msgid "RQ"
msgstr "RQ"
#: monitor.py:39
#: monitor.py:88
msgid "RC"
msgstr "RC"
#: monitor.py:40
#: monitor.py:92
msgid "Up"
msgstr "Haut"
#: monitor.py:90 monitor.py:165
#: monitor.py:172 monitor.py:286
msgid "Queued"
msgstr "En file d'attente"
#: monitor.py:92
#: monitor.py:180
msgid "Success"
msgstr "Succès"
#: monitor.py:95 monitor.py:173
#: monitor.py:190 monitor.py:294
msgid "Failures"
msgstr "Défaillances"
#: monitor.py:101
#: monitor.py:201 monitor.py:498
msgid "[Press q to quit]"
msgstr "[appuyez sur q pour quitter]"
#: monitor.py:120
#: monitor.py:227
msgid "day"
msgstr "jour"
#: monitor.py:137
#: monitor.py:248
msgid "second"
msgstr "seconde"
#: monitor.py:140
#: monitor.py:251
msgid "minute"
msgstr "minute"
#: monitor.py:143
#: monitor.py:254
msgid "hour"
msgstr "heure"
#: monitor.py:151
msgid "-- {} {} on {} --"
msgstr "--{} {} sur {}--"
#: monitor.py:263
#, python-format
msgid "-- %(prefix)s %(version)s on %(info)s --"
msgstr "--%(prefix)s %(version)s sur %(info)s --"
#: monitor.py:153
#: monitor.py:273
msgid "Clusters"
msgstr "Grappes"
#: monitor.py:157
#: monitor.py:277
msgid "Workers"
msgstr "Processus"
#: monitor.py:161
#: monitor.py:281
msgid "Restarts"
msgstr "Redémarrages"
#: monitor.py:169
#: monitor.py:290
msgid "Successes"
msgstr "Succès"
#: monitor.py:177
#: monitor.py:299
msgid "Schedules"
msgstr "Planifications"
#: monitor.py:181
msgid "Tasks/{}"
msgstr "Tâches/{}"
#: monitor.py:303
#, python-format
msgid "Tasks/%(per)s"
msgstr "Tâches/%(per)s"
#: monitor.py:185
#: monitor.py:307
msgid "Avg time"
msgstr "Temps Moyen"
#: signals.py:21
msgid "malformed return hook '{}' for [{}]"
msgstr "hook de retour mal formé' {} 'pour [{}]"
#: monitor.py:357
msgid "Available (%)"
msgstr ""
#: signals.py:26
msgid "return hook {} failed on [{}] because {}"
msgstr "le crochet de retour {} a échoué sur [{}] parce que {}"
#: monitor.py:363
msgid "Available (MB)"
msgstr "Disponible sur (MB)"
#: monitor.py:368
msgid "Total (MB)"
msgstr "Total (MB)"
#: monitor.py:373
msgid "Sentinel (MB)"
msgstr "Sentinel (MB)"
#: monitor.py:379
msgid "Monitor (MB)"
msgstr "Monitor (MB)"
#: monitor.py:385
#, fuzzy
#| msgid "Workers"
msgid "Workers (MB)"
msgstr "Processus (MB)"
#: monitor.py:487
#, python-format
msgid "Available lowest (): %(memory_percent)s ((at)s)"
msgstr "Disponible le plus bas () : %(memory_percent)s ((at)s)"
#: monitor.py:509
msgid "No clusters appear to be running."
msgstr "Aucun cluster ne semble être en cours d'exécution."
#: signals.py:22
#, python-format
msgid "malformed return hook '%(hook)s' for [%(name)s]"
msgstr "hook de retour '%(hook)s' mal formé pour [%(name)s]"
#: signals.py:30
#, python-format
msgid "return hook %(hook)s failed on [%(name)s] because %(error)s"
msgstr "hook de retour %(hook)s a échoué sur [%(name)s] à cause de %(error)s"
#, python-format
#~ msgid ""
#~ "Could not process '%(func_name)s'. Check the location of the function and "
#~ "the args/kwargs."
#~ msgstr ""
#~ "Impossible de traiter '%(func_name)s'. Vérifiez l'emplacement de la "
#~ "fonction et les args/kwargs."

View File

@@ -8,7 +8,7 @@ msgid ""
msgstr ""
"Project-Id-Version: \n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2022-06-18 23:55+0300\n"
"POT-Creation-Date: 2023-01-26 01:38+0000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: Ethem Güner <ethemguener@gmail.com>\n"
"Language-Team: \n"
@@ -17,422 +17,486 @@ msgstr ""
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"
#: django_q/admin.py:40
#: admin.py:43
msgid "Resubmit selected tasks to queue"
msgstr "Seçili işleri kuyruğa tekrar gönder"
#: django_q/brokers/disque.py:60
msgid "No Disque nodes configured"
msgstr "Disque nodları konfigüre edilmemiş"
#: admin.py:107 models.py:293
#, fuzzy
#| msgid "Success"
msgid "success"
msgstr "başarılı olanlar"
#: django_q/brokers/disque.py:77
msgid "Could not connect to any Disque nodes"
msgstr "Herhangi bir Disque noduna bağlanılamadı"
#: admin.py:119 models.py:295
msgid "last_run"
msgstr ""
#: django_q/cluster.py:79
#, python-brace-format
msgid "Q Cluster {self.name} starting."
msgstr "Q Cluster {self.name} başlatılıyor."
#: cluster.py:79
#, python-format
msgid "Q Cluster %(name)s starting."
msgstr "Q Cluster %(name)s başlatılıyor."
#: django_q/cluster.py:87
#, python-brace-format
msgid "Q Cluster {self.name} stopping."
msgstr "Q Cluster {self.name} durduruluyor."
#: cluster.py:87
#, python-format
msgid "Q Cluster %(name)s stopping."
msgstr "Q Cluster %(name)s durduruluyor."
#: django_q/cluster.py:90
#, python-brace-format
msgid "Q Cluster {self.name} has stopped."
msgstr "Q Cluster {self.name} durduruldu."
#: cluster.py:90
#, python-format
msgid "Q Cluster %(name)s has stopped."
msgstr "Q Cluster %(name)s durduruldu."
#: django_q/cluster.py:98
#: cluster.py:97
#, python-format
msgid "%(name)s got signal %(signal)s"
msgstr "%(name)s, %(signal)s pid'inde izleniyor/monitoring yapılıyor."
#: cluster.py:224
#, python-format
msgid "reincarnated monitor %(name)s after sudden death"
msgstr "Monitor %(name)s ani ölüm sonrası tekrar dirildi"
#: cluster.py:230
#, python-format
msgid "reincarnated pusher %(name)s after sudden death"
msgstr "Pusher %(name)s ani ölüm sonrası tekrar dirildi"
#: cluster.py:250
#, fuzzy, python-format
#| msgid "reincarnated worker %(name)s after timeout"
msgid ""
"{current_process().name} got signal {Conf.SIGNAL_NAMES.get(signum, \"UNKNOWN"
"\")}"
"reincarnated worker %(name)s after timeout while processing task "
"%(task_name)s"
msgstr "Worker %(name)s zaman aşımı sonrası tekrar dirildi"
#: cluster.py:255
#, python-format
msgid "reincarnated worker %(name)s after timeout"
msgstr "Worker %(name)s zaman aşımı sonrası tekrar dirildi"
#: cluster.py:260
#, python-format
msgid "recycled worker %(name)s"
msgstr "Worker %(name)s geri döndürüldü"
#: cluster.py:263
#, python-format
msgid "reincarnated worker %(name)s after death"
msgstr "Worker %(name)s ani ölüm sonrası tekrar dirildi"
#: cluster.py:287
#, python-format
msgid "%(name)s guarding cluster %(cluster_name)s"
msgstr "%(name)s, %(cluster_name)s cluster'ını koruyor"
#: cluster.py:296
#, python-format
msgid "Q Cluster %(cluster_name)s running."
msgstr "Q Cluster %(cluster_name)s başlatılıyor."
#: cluster.py:332
#, python-format
msgid "%(name)s stopping cluster processes"
msgstr "Cluster %(name)s işlemleri durduruluyor."
#: cluster.py:357
#, python-format
msgid "%(name)s waiting for the monitor."
msgstr "%(name)s monitor için bekliyor."
#: cluster.py:383
#, fuzzy, python-format
#| msgid "%(process_name)s pushing tasks at %(id)s"
msgid "%(name)s pushing tasks at %(id)s"
msgstr "%(process_name)s, işleri %(id)s pid'ine gönderiyor."
#: cluster.py:407
#, python-format
msgid "queueing from %(list_key)s"
msgstr ""
"{current_process().name} şu sinyali aldı {Conf.SIGNAL_NAMES.get(signum, \"UNKNOWN"
"\")}"
#: django_q/cluster.py:220
#, python-brace-format
msgid "reincarnated monitor {process.name} after sudden death"
msgstr "Monitor {process.name} ani ölüm sonrası tekrar dirildi"
#: cluster.py:411
#, python-format
msgid "%(name)s stopped pushing tasks"
msgstr "%(name)s işleri göndermeyi durdurdu"
#: django_q/cluster.py:223
#, python-brace-format
msgid "reincarnated pusher {process.name} after sudden death"
msgstr "Pusher {process.name} ani ölüm sonrası tekrar dirildi"
#: cluster.py:426
#, python-format
msgid "%(name)s monitoring at %(id)s"
msgstr "%(name)s, %(id)s pid'inde izleniyor/monitoring yapılıyor."
#: django_q/cluster.py:230
#, python-brace-format
msgid "reincarnated worker {process.name} after timeout"
msgstr "Worker {process.name} zaman aşımı sonrası tekrar dirildi"
#: cluster.py:445
#, python-format
msgid "Processed '%(info_name)s' (%(task_name)s)"
msgstr "[%(task_name)s] - '%(info_name)s işlendi."
#: django_q/cluster.py:232
#, python-brace-format
msgid "recycled worker {process.name}"
msgstr "Worker {process.name} geri döndürüldü"
#: cluster.py:451
#, python-format
msgid "Failed '%(info_name)s' (%(task_name)s) - %(task_result)s"
msgstr "[%(task_name)s] - '%(info_name)s' - %(task_result)s başarısız oldu"
#: django_q/cluster.py:234
#, python-brace-format
msgid "reincarnated worker {process.name} after death"
msgstr "Worker {process.name} ani ölüm sonrası tekrar dirildi"
#: cluster.py:458
#, python-format
msgid "%(name)s stopped monitoring results"
msgstr "%(name)s sonuçları göstermeyi bıraktı"
#: django_q/cluster.py:256
#: cluster.py:474
#, python-format
msgid "%(proc_name)s ready for work at %(id)s"
msgstr "%(proc_name)s, %(id)s pid'inde çalışmaya hazır"
#: cluster.py:494
#, fuzzy, python-format
#| msgid "%(proc_name)s processing '%(func_name)s' (%(task_name)s)"
msgid "%(proc_name)s processing %(task_name)s '%(func_name)s'"
msgstr "%(proc_name)s, '%(func_name)s' [%(task_name)s] işlerini işiyor"
#: cluster.py:546
#, python-format
msgid "%(proc_name)s stopped doing work"
msgstr "%(proc_name)s çalışmayı bıraktı"
#: cluster.py:751
#, python-format
msgid "%(process_name)s failed to create a task from schedule [%(schedule)s]"
msgstr "%(process_name)s programdan bir görev oluşturamadı [%(schedule)s]"
#: cluster.py:762
#, fuzzy, python-format
#| msgid "%(process_name)s created a task from schedule [%(schedule)s]"
msgid ""
"{current_process().name} guarding cluster {humanize(self.cluster_id.hex)}"
msgstr "{current_process().name}, {humanize(self.cluster_id.hex)} cluster'ını koruyor"
"%(process_name)s created task %(task_name)s from schedule [%(schedule)s]"
msgstr "%(process_name)s programdan bir görev oluşturamadı [%(schedule)s]"
#: django_q/cluster.py:261
msgid "Q Cluster {humanize(self.cluster_id.hex)} running."
msgstr "Q Cluster {humanize(self.cluster_id.hex)} çalışıyor."
#: cluster.py:808
msgid "Skipping cpu affinity because psutil was not found."
msgstr "Psutil bulunamadığı için cpu benzeşimi atlanıyor."
#: django_q/cluster.py:295
#, python-brace-format
msgid "{name} stopping cluster processes"
msgstr "Cluster {name} işlemleri durduruluyor."
#: cluster.py:813
msgid "Faking cpu affinity because it is not supported on this platform"
msgstr "Bu platformda desteklenmediği için sahte cpu benzeşimi"
#: django_q/cluster.py:320
#, python-brace-format
msgid "{name} waiting for the monitor."
msgstr "{name} monitor için bekliyor."
#: cluster.py:835
#, python-format
msgid "%(pid)s will use cpu %(affinity)s"
msgstr "%(pid)s cpu %(affinity)s kullanacaktır"
#: django_q/cluster.py:342
msgid "{current_process().name} pushing tasks at {current_process().pid}"
msgstr "{current_process().name, işleri {current_process().pid} pid'ine gönderiyor."
#: django_q/cluster.py:363
#, python-brace-format
msgid "queueing from {broker.list_key}"
msgstr ""
#: django_q/cluster.py:366
msgid "{current_process().name} stopped pushing tasks"
msgstr "{current_process().name} işleri göndermeyi durdurdu"
#: django_q/cluster.py:378
msgid "{name} monitoring at {current_process().pid}"
msgstr "{name}, {current_process().pid} pid'inde izleniyor/monitoring yapılıyor."
#: django_q/cluster.py:394
msgid "Processed [{task['name']}]"
msgstr "[{task['name']}] işlendi."
#: django_q/cluster.py:397
msgid "Failed [{task['name']}] - {task['result']}"
msgstr "[{task['name']}] - {task['result']} başarısız oldu"
#: django_q/cluster.py:398
#, python-brace-format
msgid "{name} stopped monitoring results"
msgstr "{name} sonuçları göstermeyi bıraktı"
#: django_q/cluster.py:412
msgid "{name} ready for work at {current_process().pid}"
msgstr "{name}, {current_process().pid} pid'inde çalışmaya hazır"
#: django_q/cluster.py:422
msgid "{name} processing [{task[\"name\"]}]"
msgstr "{name}, [{task[\"name\"]}] işlerini işiyor"
#: django_q/cluster.py:453
#, python-brace-format
msgid "{name} stopped doing work"
msgstr "{name} çalışmayı bıraktı"
#: django_q/cluster.py:635 django_q/models.py:143
msgid "Please install croniter to enable cron expressions"
msgstr "Cron expressions'ları açmak için croniter yükleyin"
#: django_q/cluster.py:665
#: conf.py:90
#, python-format
msgid ""
"{current_process().name} failed to create a task from schedule [{s.name or s."
"id}]"
msgstr ""
#: django_q/cluster.py:671
msgid ""
"{current_process().name} created a task from schedule [{s.name or s.id}]"
msgstr ""
#: django_q/cluster.py:737
#, python-brace-format
msgid "{pid} will use cpu {affinity}"
"SAVE_LIMIT_PER (%(option)s) is not a valid option. Options are: 'group', "
"'name', 'func' and None. Default is None."
msgstr ""
"SAVE_LIMIT_PER (%(option)s) geçerli bir seçenek değil. Seçenekler şunlardır: "
"'group', 'name', 'func' ve None. Varsayılan değer None'dır."
#. Translators: Cluster status descriptions
#: django_q/conf.py:196
#: conf.py:207
msgid "Starting"
msgstr "Başlıyor"
#: django_q/conf.py:197
#: conf.py:208
msgid "Working"
msgstr "Çalışıyor"
#: django_q/conf.py:198
#: conf.py:209
msgid "Idle"
msgstr "Boşta"
#: django_q/conf.py:199
#: conf.py:210
msgid "Stopped"
msgstr "Durdu"
#: django_q/conf.py:200
#: conf.py:211
msgid "Stopping"
msgstr "Durduruluyor"
#. Translators: help text for qcluster management command
#: django_q/management/commands/qcluster.py:9
#: management/commands/qcluster.py:9
msgid "Starts a Django Q Cluster."
msgstr "Bir Django Q Cluster çalıştırır."
#. Translators: help text for qinfo management command
#: django_q/management/commands/qinfo.py:11
#: management/commands/qinfo.py:11
msgid "General information over all clusters."
msgstr "Tüm cluster'lar için genel bilgiler."
#. Translators: help text for qmemory management command
#: django_q/management/commands/qmemory.py:9
#: management/commands/qmemory.py:9
msgid "Monitors Q Cluster memory usage"
msgstr "Q Cluster'ın bellek kullanımını izler"
#. Translators: help text for qmonitor management command
#: django_q/management/commands/qmonitor.py:9
#: management/commands/qmonitor.py:9
msgid "Monitors Q Cluster activity"
msgstr "Q Cluster'ın aktivitelerini izler"
#: django_q/models.py:118
#: models.py:125
msgid "Successful task"
msgstr "Başarılı iş"
#: django_q/models.py:119
#: models.py:126
msgid "Successful tasks"
msgstr "Başarılı işler"
#: django_q/models.py:134
#: models.py:141
msgid "Failed task"
msgstr "Başarısız iş"
#: django_q/models.py:135
#: models.py:142
msgid "Failed tasks"
msgstr "Başarısız işler"
#: django_q/models.py:159
#: models.py:150 models.py:234
msgid "Please install croniter to enable cron expressions"
msgstr "Cron expressions'ları açmak için croniter yükleyin"
#: models.py:170
msgid "e.g. 1, 2, 'John'"
msgstr "Örneğin: 1, 2, 'Melih'"
#: django_q/models.py:161
#: models.py:172
msgid "e.g. x=1, y=2, name='John'"
msgstr "Örneğin: x=1, y=2, name='Melih'"
#: django_q/models.py:173
#: models.py:186
msgid "Once"
msgstr "Bir kere"
#: django_q/models.py:174
#: models.py:187
msgid "Minutes"
msgstr "Dakika"
#: django_q/models.py:175
#: models.py:188
msgid "Hourly"
msgstr "Saatlik"
#: django_q/models.py:176
#: models.py:189
msgid "Daily"
msgstr "Günlük"
#: django_q/models.py:177
#: models.py:190
msgid "Weekly"
msgstr "Haftalık"
#: django_q/models.py:178
#: models.py:191
#, fuzzy
#| msgid "Weekly"
msgid "Biweekly"
msgstr "İki haftada bir"
#: models.py:192
msgid "Monthly"
msgstr "Aylık"
#: django_q/models.py:179
#: models.py:193
#, fuzzy
#| msgid "Monthly"
msgid "Bimonthly"
msgstr "İki ayda bir"
#: models.py:194
msgid "Quarterly"
msgstr "Bir Çeyrek (3 Ay)"
#: django_q/models.py:180
#: models.py:195
msgid "Yearly"
msgstr "Yıllık"
#: django_q/models.py:181
#: models.py:196
msgid "Cron"
msgstr ""
#: django_q/models.py:184
#: models.py:199
msgid "Schedule Type"
msgstr "Zamanlama Tipi"
#: django_q/models.py:187
#: models.py:202
msgid "Number of minutes for the Minutes type"
msgstr "Dakika tipine göre dakika sayısı"
#: django_q/models.py:190
#: models.py:205
msgid "Repeats"
msgstr "Tekrar eder"
#: django_q/models.py:190
#: models.py:205
msgid "n = n times, -1 = forever"
msgstr "n = n kere, -1 = sonsuza kadar"
#: django_q/models.py:193
#: models.py:208
msgid "Next Run"
msgstr "Bir dahaki çalışma tarihi"
#: django_q/models.py:200
#: models.py:215
msgid "Cron expression"
msgstr ""
#: django_q/models.py:227
#: models.py:224
msgid "Name of kwarg to pass intended schedule date"
msgstr "Geçilecek kwarg'ın adı öngörülen program tarihi"
#: models.py:299
msgid "Scheduled task"
msgstr "Zamanlanmış iş"
#: django_q/models.py:228
#: models.py:300
msgid "Scheduled tasks"
msgstr "Zamanlanmış işler"
#: django_q/models.py:251
#: models.py:326
msgid "Queued task"
msgstr "Sıraya alınmış iş"
#: django_q/models.py:252
#: models.py:327
msgid "Queued tasks"
msgstr "Sıraya alınmış işler"
#: django_q/monitor.py:54 django_q/monitor.py:322
#: monitor.py:64 monitor.py:348
msgid "Host"
msgstr ""
#: django_q/monitor.py:58 django_q/monitor.py:326 django_q/monitor.py:433
#: monitor.py:68 monitor.py:352 monitor.py:459
msgid "Id"
msgstr ""
#: django_q/monitor.py:62
#: monitor.py:72
msgid "State"
msgstr "Durum"
#: django_q/monitor.py:66
#: monitor.py:76
msgid "Pool"
msgstr "Havuz"
#: django_q/monitor.py:70
#: monitor.py:80
msgid "TQ"
msgstr ""
#: django_q/monitor.py:74
#: monitor.py:84
msgid "RQ"
msgstr ""
#: django_q/monitor.py:78
#: monitor.py:88
msgid "RC"
msgstr ""
#: django_q/monitor.py:82
#: monitor.py:92
msgid "Up"
msgstr ""
#: django_q/monitor.py:162 django_q/monitor.py:266
#: monitor.py:172 monitor.py:286
msgid "Queued"
msgstr "Sıraya alınmış"
#: django_q/monitor.py:170
#: monitor.py:180
msgid "Success"
msgstr "Başarılı olanlar"
#: django_q/monitor.py:180 django_q/monitor.py:274
#: monitor.py:190 monitor.py:294
msgid "Failures"
msgstr "Başarısız olanlar"
#: django_q/monitor.py:191
#: monitor.py:201 monitor.py:498
msgid "[Press q to quit]"
msgstr "[Çıkmak için q'ya basın]"
#: django_q/monitor.py:210
#: monitor.py:227
msgid "day"
msgstr "gün"
#: django_q/monitor.py:231
#: monitor.py:248
msgid "second"
msgstr "saniye"
#: django_q/monitor.py:234
#: monitor.py:251
msgid "minute"
msgstr "dakika"
#: django_q/monitor.py:237
#: monitor.py:254
msgid "hour"
msgstr "saat"
#: django_q/monitor.py:247
msgid ""
"-- {Conf.PREFIX.capitalize()} { \".\".join(str(v) for v in VERSION)} on "
"{broker.info()} --"
msgstr ""
#: monitor.py:263
#, python-format
msgid "-- %(prefix)s %(version)s on %(info)s --"
msgstr "-- %(prefix)s %(version)s üzerinde %(info)s --"
#: django_q/monitor.py:253
#: monitor.py:273
msgid "Clusters"
msgstr ""
#: django_q/monitor.py:257
#: monitor.py:277
msgid "Workers"
msgstr ""
#: django_q/monitor.py:261
#: monitor.py:281
msgid "Restarts"
msgstr "Yeniden çalıştırmalar"
#: django_q/monitor.py:270
#: monitor.py:290
msgid "Successes"
msgstr "Başarılı olanlar"
#: django_q/monitor.py:279
#: monitor.py:299
msgid "Schedules"
msgstr "Zamanlanmışlar"
#: django_q/monitor.py:283
#, python-brace-format
msgid "Tasks/{per}"
msgstr "İş/{per}"
#: monitor.py:303
#, python-format
msgid "Tasks/%(per)s"
msgstr "İş/%(per)s"
#: django_q/monitor.py:287
#: monitor.py:307
msgid "Avg time"
msgstr "Ortalama süre"
#: django_q/monitor.py:331
#: monitor.py:357
msgid "Available (%)"
msgstr "Müsait (%) "
#: django_q/monitor.py:337
#: monitor.py:363
msgid "Available (MB)"
msgstr "Müsait (MB)"
#: django_q/monitor.py:342
#: monitor.py:368
msgid "Total (MB)"
msgstr "Toplam (MB)"
#: django_q/monitor.py:347
#: monitor.py:373
msgid "Sentinel (MB)"
msgstr ""
#: django_q/monitor.py:353
#: monitor.py:379
msgid "Monitor (MB)"
msgstr "İzleme (MB)"
#: django_q/monitor.py:359
#: monitor.py:385
msgid "Workers (MB)"
msgstr ""
#: django_q/monitor.py:461
msgid "Available lowest (%): {} ({})"
msgstr "Mevcut en düşük (%): {} ({})"
#: monitor.py:487
#, python-format
msgid "Available lowest (): %(memory_percent)s ((at)s)"
msgstr "Mevcut en düşük (): %(memory_percent)s ((at)s)"
#: django_q/signals.py:22
#, python-brace-format
msgid "malformed return hook '{instance.hook}' for [{instance.name}]"
#: monitor.py:509
msgid "No clusters appear to be running."
msgstr "Hiçbir küme çalışıyor görünmüyor."
#: signals.py:22
#, python-format
msgid "malformed return hook '%(hook)s' for [%(name)s]"
msgstr ""
#: django_q/signals.py:30
msgid ""
"return hook {instance.hook} failed on [{instance.name}] because {str(e)}"
#: signals.py:30
#, python-format
msgid "return hook %(hook)s failed on [%(name)s] because %(error)s"
msgstr ""
#, python-format
#~ msgid ""
#~ "Could not process '%(func_name)s'. Check the location of the function and "
#~ "the args/kwargs."
#~ msgstr ""
#~ "%(func_name)s' işlenemedi. İşlevin konumunu ve args/kwargs öğelerini "
#~ "kontrol edin."

View File

@@ -2,6 +2,7 @@ from django.core.management.base import BaseCommand
from django.utils.translation import gettext as _
from django_q.cluster import Cluster
import os
class Command(BaseCommand):
@@ -16,8 +17,21 @@ class Command(BaseCommand):
default=False,
help="Run once and then stop.",
)
parser.add_argument(
"-n",
"--name",
dest="cluster_name",
default=None,
help="Set alternative cluster name instead of the name in Q_CLUSTER settings (for multi-queue setup). "
"On Linux you should set name through `Q_CLUSTER_NAME=cluster_name python manage.py qcluster` instead."
)
def handle(self, *args, **options):
# Set alternative cluster_name before creating the cluster (cluster_name is broker's queue_name, too)
cluster_name = options.get("cluster_name")
if cluster_name:
os.environ["Q_CLUSTER_NAME"] = cluster_name
q = Cluster()
q.start()
if options.get("run_once", False):

View File

@@ -5,61 +5,142 @@ from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
]
dependencies = []
operations = [
migrations.CreateModel(
name='Schedule',
name="Schedule",
fields=[
('id', models.AutoField(verbose_name='ID', auto_created=True, serialize=False, primary_key=True)),
('func', models.CharField(max_length=256, help_text='e.g. module.tasks.function')),
('hook', models.CharField(null=True, blank=True, max_length=256, help_text='e.g. module.tasks.result_function')),
('args', models.CharField(null=True, blank=True, max_length=256, help_text="e.g. 1, 2, 'John'")),
('kwargs', models.CharField(null=True, blank=True, max_length=256, help_text="e.g. x=1, y=2, name='John'")),
('schedule_type', models.CharField(verbose_name='Schedule Type', choices=[('O', 'Once'), ('H', 'Hourly'), ('D', 'Daily'), ('W', 'Weekly'), ('M', 'Monthly'), ('Q', 'Quarterly'), ('Y', 'Yearly')], default='O', max_length=1)),
('repeats', models.SmallIntegerField(verbose_name='Repeats', default=-1, help_text='n = n times, -1 = forever')),
('next_run', models.DateTimeField(verbose_name='Next Run', default=django.utils.timezone.now, null=True)),
('task', models.CharField(editable=False, null=True, max_length=100)),
(
"id",
models.AutoField(
verbose_name="ID",
auto_created=True,
serialize=False,
primary_key=True,
),
),
(
"func",
models.CharField(
max_length=256, help_text="e.g. module.tasks.function"
),
),
(
"hook",
models.CharField(
null=True,
blank=True,
max_length=256,
help_text="e.g. module.tasks.result_function",
),
),
(
"args",
models.CharField(
null=True,
blank=True,
max_length=256,
help_text="e.g. 1, 2, 'John'",
),
),
(
"kwargs",
models.CharField(
null=True,
blank=True,
max_length=256,
help_text="e.g. x=1, y=2, name='John'",
),
),
(
"schedule_type",
models.CharField(
verbose_name="Schedule Type",
choices=[
("O", "Once"),
("H", "Hourly"),
("D", "Daily"),
("W", "Weekly"),
("M", "Monthly"),
("Q", "Quarterly"),
("Y", "Yearly"),
],
default="O",
max_length=1,
),
),
(
"repeats",
models.SmallIntegerField(
verbose_name="Repeats",
default=-1,
help_text="n = n times, -1 = forever",
),
),
(
"next_run",
models.DateTimeField(
verbose_name="Next Run",
default=django.utils.timezone.now,
null=True,
),
),
("task", models.CharField(editable=False, null=True, max_length=100)),
],
options={
'verbose_name': 'Scheduled task',
'ordering': ['next_run'],
"verbose_name": "Scheduled task",
"ordering": ["next_run"],
},
),
migrations.CreateModel(
name='Task',
name="Task",
fields=[
('id', models.AutoField(verbose_name='ID', auto_created=True, serialize=False, primary_key=True)),
('name', models.CharField(editable=False, max_length=100)),
('func', models.CharField(max_length=256)),
('hook', models.CharField(null=True, max_length=256)),
('args', picklefield.fields.PickledObjectField(editable=False, null=True)),
('kwargs', picklefield.fields.PickledObjectField(editable=False, null=True)),
('result', picklefield.fields.PickledObjectField(editable=False, null=True)),
('started', models.DateTimeField(editable=False)),
('stopped', models.DateTimeField(editable=False)),
('success', models.BooleanField(editable=False, default=True)),
(
"id",
models.AutoField(
verbose_name="ID",
auto_created=True,
serialize=False,
primary_key=True,
),
),
("name", models.CharField(editable=False, max_length=100)),
("func", models.CharField(max_length=256)),
("hook", models.CharField(null=True, max_length=256)),
(
"args",
picklefield.fields.PickledObjectField(editable=False, null=True),
),
(
"kwargs",
picklefield.fields.PickledObjectField(editable=False, null=True),
),
(
"result",
picklefield.fields.PickledObjectField(editable=False, null=True),
),
("started", models.DateTimeField(editable=False)),
("stopped", models.DateTimeField(editable=False)),
("success", models.BooleanField(editable=False, default=True)),
],
),
migrations.CreateModel(
name='Failure',
fields=[
],
name="Failure",
fields=[],
options={
'verbose_name': 'Failed task',
'proxy': True,
"verbose_name": "Failed task",
"proxy": True,
},
bases=('django_q.task',),
bases=("django_q.task",),
),
migrations.CreateModel(
name='Success',
fields=[
],
name="Success",
fields=[],
options={
'verbose_name': 'Successful task',
'proxy': True,
"verbose_name": "Successful task",
"proxy": True,
},
bases=('django_q.task',),
bases=("django_q.task",),
),
]

View File

@@ -4,18 +4,22 @@ from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('django_q', '0001_initial'),
("django_q", "0001_initial"),
]
operations = [
migrations.AlterField(
model_name='schedule',
name='args',
field=models.TextField(help_text="e.g. 1, 2, 'John'", blank=True, null=True),
model_name="schedule",
name="args",
field=models.TextField(
help_text="e.g. 1, 2, 'John'", blank=True, null=True
),
),
migrations.AlterField(
model_name='schedule',
name='kwargs',
field=models.TextField(help_text="e.g. x=1, y=2, name='John'", blank=True, null=True),
model_name="schedule",
name="kwargs",
field=models.TextField(
help_text="e.g. x=1, y=2, name='John'", blank=True, null=True
),
),
]

View File

@@ -4,29 +4,41 @@ from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('django_q', '0002_auto_20150630_1624'),
("django_q", "0002_auto_20150630_1624"),
]
operations = [
migrations.AlterModelOptions(
name='failure',
options={'verbose_name_plural': 'Failed tasks', 'verbose_name': 'Failed task'},
name="failure",
options={
"verbose_name_plural": "Failed tasks",
"verbose_name": "Failed task",
},
),
migrations.AlterModelOptions(
name='schedule',
options={'verbose_name_plural': 'Scheduled tasks', 'ordering': ['next_run'], 'verbose_name': 'Scheduled task'},
name="schedule",
options={
"verbose_name_plural": "Scheduled tasks",
"ordering": ["next_run"],
"verbose_name": "Scheduled task",
},
),
migrations.AlterModelOptions(
name='success',
options={'verbose_name_plural': 'Successful tasks', 'verbose_name': 'Successful task'},
name="success",
options={
"verbose_name_plural": "Successful tasks",
"verbose_name": "Successful task",
},
),
migrations.RemoveField(
model_name='task',
name='id',
model_name="task",
name="id",
),
migrations.AddField(
model_name='task',
name='id',
field=models.CharField(max_length=32, primary_key=True, editable=False, serialize=False),
model_name="task",
name="id",
field=models.CharField(
max_length=32, primary_key=True, editable=False, serialize=False
),
),
]

View File

@@ -1,23 +1,31 @@
from django.db import migrations, models
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('django_q', '0003_auto_20150708_1326'),
("django_q", "0003_auto_20150708_1326"),
]
operations = [
migrations.AlterModelOptions(
name='failure',
options={'verbose_name_plural': 'Failed tasks', 'verbose_name': 'Failed task', 'ordering': ['-stopped']},
name="failure",
options={
"verbose_name_plural": "Failed tasks",
"verbose_name": "Failed task",
"ordering": ["-stopped"],
},
),
migrations.AlterModelOptions(
name='success',
options={'verbose_name_plural': 'Successful tasks', 'verbose_name': 'Successful task', 'ordering': ['-stopped']},
name="success",
options={
"verbose_name_plural": "Successful tasks",
"verbose_name": "Successful task",
"ordering": ["-stopped"],
},
),
migrations.AlterModelOptions(
name='task',
options={'ordering': ['-stopped']},
name="task",
options={"ordering": ["-stopped"]},
),
]

View File

@@ -4,18 +4,18 @@ from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('django_q', '0004_auto_20150710_1043'),
("django_q", "0004_auto_20150710_1043"),
]
operations = [
migrations.AddField(
model_name='schedule',
name='name',
model_name="schedule",
name="name",
field=models.CharField(max_length=100, null=True),
),
migrations.AddField(
model_name='task',
name='group',
model_name="task",
name="group",
field=models.CharField(max_length=100, null=True, editable=False),
),
]

View File

@@ -4,18 +4,36 @@ from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('django_q', '0005_auto_20150718_1506'),
("django_q", "0005_auto_20150718_1506"),
]
operations = [
migrations.AddField(
model_name='schedule',
name='minutes',
field=models.PositiveSmallIntegerField(help_text='Number of minutes for the Minutes type', blank=True, null=True),
model_name="schedule",
name="minutes",
field=models.PositiveSmallIntegerField(
help_text="Number of minutes for the Minutes type",
blank=True,
null=True,
),
),
migrations.AlterField(
model_name='schedule',
name='schedule_type',
field=models.CharField(max_length=1, choices=[('O', 'Once'), ('I', 'Minutes'), ('H', 'Hourly'), ('D', 'Daily'), ('W', 'Weekly'), ('M', 'Monthly'), ('Q', 'Quarterly'), ('Y', 'Yearly')], default='O', verbose_name='Schedule Type'),
model_name="schedule",
name="schedule_type",
field=models.CharField(
max_length=1,
choices=[
("O", "Once"),
("I", "Minutes"),
("H", "Hourly"),
("D", "Daily"),
("W", "Weekly"),
("M", "Monthly"),
("Q", "Quarterly"),
("Y", "Yearly"),
],
default="O",
verbose_name="Schedule Type",
),
),
]

View File

@@ -4,21 +4,29 @@ from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('django_q', '0006_auto_20150805_1817'),
("django_q", "0006_auto_20150805_1817"),
]
operations = [
migrations.CreateModel(
name='OrmQ',
name="OrmQ",
fields=[
('id', models.AutoField(primary_key=True, auto_created=True, verbose_name='ID', serialize=False)),
('key', models.CharField(max_length=100)),
('payload', models.TextField()),
('lock', models.DateTimeField(null=True)),
(
"id",
models.AutoField(
primary_key=True,
auto_created=True,
verbose_name="ID",
serialize=False,
),
),
("key", models.CharField(max_length=100)),
("payload", models.TextField()),
("lock", models.DateTimeField(null=True)),
],
options={
'verbose_name_plural': 'Queued tasks',
'verbose_name': 'Queued task',
"verbose_name_plural": "Queued tasks",
"verbose_name": "Queued task",
},
),
]

View File

@@ -4,13 +4,13 @@ from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('django_q', '0007_ormq'),
("django_q", "0007_ormq"),
]
operations = [
migrations.AlterField(
model_name='schedule',
name='name',
model_name="schedule",
name="name",
field=models.CharField(blank=True, max_length=100, null=True),
),
]

View File

@@ -4,13 +4,17 @@ from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('django_q', '0008_auto_20160224_1026'),
("django_q", "0008_auto_20160224_1026"),
]
operations = [
migrations.AlterField(
model_name='schedule',
name='repeats',
field=models.IntegerField(default=-1, help_text='n = n times, -1 = forever', verbose_name='Repeats'),
model_name="schedule",
name="repeats",
field=models.IntegerField(
default=-1,
help_text="n = n times, -1 = forever",
verbose_name="Repeats",
),
),
]

View File

@@ -5,23 +5,29 @@ from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('django_q', '0009_auto_20171009_0915'),
("django_q", "0009_auto_20171009_0915"),
]
operations = [
migrations.AlterField(
model_name='task',
name='args',
field=picklefield.fields.PickledObjectField(editable=False, null=True, protocol=-1),
model_name="task",
name="args",
field=picklefield.fields.PickledObjectField(
editable=False, null=True, protocol=-1
),
),
migrations.AlterField(
model_name='task',
name='kwargs',
field=picklefield.fields.PickledObjectField(editable=False, null=True, protocol=-1),
model_name="task",
name="kwargs",
field=picklefield.fields.PickledObjectField(
editable=False, null=True, protocol=-1
),
),
migrations.AlterField(
model_name='task',
name='result',
field=picklefield.fields.PickledObjectField(editable=False, null=True, protocol=-1),
model_name="task",
name="result",
field=picklefield.fields.PickledObjectField(
editable=False, null=True, protocol=-1
),
),
]

View File

@@ -6,18 +6,35 @@ from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('django_q', '0010_auto_20200610_0856'),
("django_q", "0010_auto_20200610_0856"),
]
operations = [
migrations.AddField(
model_name='schedule',
name='cron',
field=models.CharField(blank=True, help_text='Cron expression', max_length=100, null=True),
model_name="schedule",
name="cron",
field=models.CharField(
blank=True, help_text="Cron expression", max_length=100, null=True
),
),
migrations.AlterField(
model_name='schedule',
name='schedule_type',
field=models.CharField(choices=[('O', 'Once'), ('I', 'Minutes'), ('H', 'Hourly'), ('D', 'Daily'), ('W', 'Weekly'), ('M', 'Monthly'), ('Q', 'Quarterly'), ('Y', 'Yearly'), ('C', 'Cron')], default='O', max_length=1, verbose_name='Schedule Type'),
model_name="schedule",
name="schedule_type",
field=models.CharField(
choices=[
("O", "Once"),
("I", "Minutes"),
("H", "Hourly"),
("D", "Daily"),
("W", "Weekly"),
("M", "Monthly"),
("Q", "Quarterly"),
("Y", "Yearly"),
("C", "Cron"),
],
default="O",
max_length=1,
verbose_name="Schedule Type",
),
),
]

View File

@@ -8,13 +8,19 @@ import django_q.models
class Migration(migrations.Migration):
dependencies = [
('django_q', '0011_auto_20200628_1055'),
("django_q", "0011_auto_20200628_1055"),
]
operations = [
migrations.AlterField(
model_name='schedule',
name='cron',
field=models.CharField(blank=True, help_text='Cron expression', max_length=100, null=True, validators=[django_q.models.validate_cron]),
model_name="schedule",
name="cron",
field=models.CharField(
blank=True,
help_text="Cron expression",
max_length=100,
null=True,
validators=[django_q.models.validate_cron],
),
),
]

View File

@@ -6,13 +6,13 @@ from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('django_q', '0012_auto_20200702_1608'),
("django_q", "0012_auto_20200702_1608"),
]
operations = [
migrations.AddField(
model_name='task',
name='attempt_count',
model_name="task",
name="attempt_count",
field=models.IntegerField(default=0),
),
]

View File

@@ -6,13 +6,13 @@ from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('django_q', '0013_task_attempt_count'),
("django_q", "0013_task_attempt_count"),
]
operations = [
migrations.AddField(
model_name='schedule',
name='cluster',
model_name="schedule",
name="cluster",
field=models.CharField(blank=True, default=None, max_length=100, null=True),
),
]

View File

@@ -0,0 +1,35 @@
# Generated by Django 4.1.2 on 2022-11-10 01:35
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("django_q", "0014_schedule_cluster"),
]
operations = [
migrations.AlterField(
model_name="schedule",
name="schedule_type",
field=models.CharField(
choices=[
("O", "Once"),
("I", "Minutes"),
("H", "Hourly"),
("D", "Daily"),
("W", "Weekly"),
("BW", "Biweekly"),
("M", "Monthly"),
("BM", "Bimonthly"),
("Q", "Quarterly"),
("Y", "Yearly"),
("C", "Cron"),
],
default="O",
max_length=2,
verbose_name="Schedule Type",
),
),
]

View File

@@ -0,0 +1,25 @@
# Generated by Django 4.1.2 on 2023-01-15 22:34
from django.db import migrations, models
import django_q.models
class Migration(migrations.Migration):
dependencies = [
("django_q", "0015_alter_schedule_schedule_type"),
]
operations = [
migrations.AddField(
model_name="schedule",
name="intended_date_kwarg",
field=models.CharField(
blank=True,
help_text="Name of kwarg to pass intended schedule date",
max_length=100,
null=True,
validators=[django_q.models.validate_kwarg],
),
),
]

View File

@@ -0,0 +1,43 @@
# Generated by Django 4.1.5 on 2023-03-07 12:18
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("django_q", "0016_schedule_intended_date_kwarg"),
]
operations = [
migrations.AddField(
model_name="task",
name="cluster",
field=models.CharField(blank=True, default=None, max_length=100, null=True),
),
migrations.AlterField(
model_name="ormq",
name="key",
field=models.CharField(
help_text="Name of the target cluster", max_length=100
),
),
migrations.AlterField(
model_name="ormq",
name="lock",
field=models.DateTimeField(
help_text="Prevent any cluster from pulling until", null=True
),
),
migrations.AlterField(
model_name="schedule",
name="cluster",
field=models.CharField(
blank=True,
default=None,
help_text="Name of the target cluster",
max_length=100,
null=True,
),
),
]

View File

@@ -1,3 +1,6 @@
from datetime import datetime, timedelta
from keyword import iskeyword
# Django
from django import get_version
from django.core.exceptions import ValidationError
@@ -5,16 +8,21 @@ from django.db import models
from django.template.defaultfilters import truncatechars
from django.urls import reverse
from django.utils import timezone
from django.utils.timezone import is_aware
from django.utils.html import format_html
from django.utils.translation import gettext_lazy as _
from django.utils.functional import cached_property
# External
from picklefield import PickledObjectField
from picklefield.fields import dbsafe_decode
# Local
from django_q.conf import croniter
from django_q.conf import croniter, Conf
from django_q.signing import SignedPackage
from django_q.utils import localtime, add_months, add_years
from .utils import get_func_repr
class Task(models.Model):
@@ -26,6 +34,7 @@ class Task(models.Model):
kwargs = PickledObjectField(null=True, protocol=-1)
result = PickledObjectField(null=True, protocol=-1)
group = models.CharField(max_length=100, editable=False, null=True)
cluster = models.CharField(max_length=100, default=None, null=True, blank=True)
started = models.DateTimeField(editable=False)
stopped = models.DateTimeField(editable=False)
success = models.BooleanField(default=True, editable=False)
@@ -147,6 +156,10 @@ def validate_cron(value):
raise ValidationError(e)
def validate_kwarg(value):
return value.isidentifier() and not iskeyword(value)
class Schedule(models.Model):
name = models.CharField(max_length=100, null=True, blank=True)
func = models.CharField(max_length=256, help_text="e.g. module.tasks.function")
@@ -165,7 +178,9 @@ class Schedule(models.Model):
HOURLY = "H"
DAILY = "D"
WEEKLY = "W"
BIWEEKLY = "BW"
MONTHLY = "M"
BIMONTHLY = "BM"
QUARTERLY = "Q"
YEARLY = "Y"
CRON = "C"
@@ -175,13 +190,15 @@ class Schedule(models.Model):
(HOURLY, _("Hourly")),
(DAILY, _("Daily")),
(WEEKLY, _("Weekly")),
(BIWEEKLY, _("Biweekly")),
(MONTHLY, _("Monthly")),
(BIMONTHLY, _("Bimonthly")),
(QUARTERLY, _("Quarterly")),
(YEARLY, _("Yearly")),
(CRON, _("Cron")),
)
schedule_type = models.CharField(
max_length=1, choices=TYPE, default=TYPE[0][0], verbose_name=_("Schedule Type")
max_length=2, choices=TYPE, default=TYPE[0][0], verbose_name=_("Schedule Type")
)
minutes = models.PositiveSmallIntegerField(
null=True, blank=True, help_text=_("Number of minutes for the Minutes type")
@@ -200,7 +217,65 @@ class Schedule(models.Model):
help_text=_("Cron expression"),
)
task = models.CharField(max_length=100, null=True, editable=False)
cluster = models.CharField(max_length=100, default=None, null=True, blank=True)
cluster = models.CharField(
max_length=100, default=None, null=True, blank=True,
help_text=_("Name of the target cluster")
)
intended_date_kwarg = models.CharField(
max_length=100,
null=True,
blank=True,
validators=[validate_kwarg],
help_text=_("Name of kwarg to pass intended schedule date"),
)
def calculate_next_run(self, next_run=None):
# next run is always in UTC
next_run = next_run or self.next_run
if self.schedule_type == self.CRON:
if not croniter:
raise ImportError(
_("Please install croniter to enable cron expressions")
)
return croniter(self.cron, localtime()).get_next(datetime)
if self.schedule_type == self.MINUTES:
add = timedelta(minutes=(self.minutes or 1))
elif self.schedule_type == self.HOURLY:
add = timedelta(hours=1)
elif self.schedule_type == self.DAILY:
add = timedelta(days=1)
elif self.schedule_type == self.WEEKLY:
add = timedelta(weeks=1)
elif self.schedule_type == self.BIWEEKLY:
add = timedelta(weeks=2)
elif self.schedule_type == self.MONTHLY:
add = timedelta(days=(add_months(next_run, 1) - next_run).days)
elif self.schedule_type == self.BIMONTHLY:
add = timedelta(days=(add_months(next_run, 2) - next_run).days)
elif self.schedule_type == self.QUARTERLY:
add = timedelta(days=(add_months(next_run, 3) - next_run).days)
elif self.schedule_type == self.YEARLY:
add = timedelta(days=(add_years(next_run, 1) - next_run).days)
# add normal timedelta, we will correct this later based on timezone
next_run += add
# DST differencers don't matter with minutes, hourly or yearly, so skip those
if self.schedule_type not in [self.MINUTES, self.HOURLY, self.YEARLY]:
# Get localtimes and then remove the tzinfo, so we can get the actual difference
current_next_run = localtime(next_run - add).replace(tzinfo=None)
new_next_run = localtime(next_run).replace(tzinfo=None)
# get the difference between them, this should be (-)1 or (-)0.5 hour
# based on DST active or not
extra_diff = (new_next_run - current_next_run) - add
# subtract difference
next_run -= extra_diff
return next_run
def success(self):
if self.task and Task.objects.filter(id=self.task):
@@ -224,7 +299,6 @@ class Schedule(models.Model):
last_run.allow_tags = True
last_run.short_description = _("last_run")
class Meta:
app_label = "django_q"
verbose_name = _("Scheduled task")
@@ -233,24 +307,38 @@ class Schedule(models.Model):
class OrmQ(models.Model):
key = models.CharField(max_length=100)
key = models.CharField(max_length=100, help_text=_("Name of the target cluster"))
payload = models.TextField()
lock = models.DateTimeField(null=True)
lock = models.DateTimeField(null=True, help_text=_("Prevent any cluster from pulling until"))
@cached_property
def task(self):
return SignedPackage.loads(self.payload)
try:
return SignedPackage.loads(self.payload)
except Exception as e:
return {"id": "*" + e.__class__.__name__}
def func(self):
return self.task()["func"]
return get_func_repr(self.task.get("func"))
def task_id(self):
return self.task()["id"]
return self.task.get("id")
def name(self):
return self.task()["name"]
return self.task.get("name")
def group(self):
return self.task().get("group")
return self.task.get("group")
def args(self):
return self.task.get("args")
def kwargs(self):
return self.task.get("kwargs")
def q_options(self):
exclude = {"id", "name", "group", "func", "args", "kwargs"}
return {k: v for k, v in self.task.items() if k not in exclude}
class Meta:
app_label = "django_q"

View File

@@ -19,28 +19,30 @@ try:
except ImportError:
psutil = None
# optional
try:
from blessed import Terminal
except ImportError:
pass
def get_process_mb(pid):
try:
process = psutil.Process(pid)
mb_used = round(process.memory_info().rss / 1024 ** 2, 2)
mb_used = round(process.memory_info().rss / 1024**2, 2)
except psutil.NoSuchProcess:
mb_used = "NO_PROCESS_FOUND"
return mb_used
BLESSED_INSTALL_MESSAGE = "Blessed is not installed. Please install blessed to use this: https://pypi.org/project/blessed/"
BLESSED_INSTALL_MESSAGE = (
"Blessed is not installed. Please install blessed to use this: "
"https://pypi.org/project/blessed/"
)
def monitor(run_once=False, broker=None):
if not broker:
broker = get_broker()
try:
from blessed import Terminal
term = Terminal()
except:
except ImportError:
print(BLESSED_INSTALL_MESSAGE)
return
@@ -204,8 +206,10 @@ def info(broker=None):
if not broker:
broker = get_broker()
try:
from blessed import Terminal
term = Terminal()
except:
except ImportError:
print(BLESSED_INSTALL_MESSAGE)
return
@@ -256,9 +260,12 @@ def info(broker=None):
print(
term.black_on_green(
term.center(
_(
f'-- {Conf.PREFIX.capitalize()} { ".".join(str(v) for v in VERSION)} on {broker.info()} --'
)
_("-- %(prefix)s %(version)s on %(info)s --")
% {
"prefix": Conf.PREFIX.capitalize(),
"version": ".".join(str(v) for v in VERSION),
"info": broker.info(),
}
)
)
)
@@ -293,7 +300,7 @@ def info(broker=None):
+ term.move_x(1 * col_width)
+ term.white(str(models.Schedule.objects.count()))
+ term.move_x(2 * col_width)
+ term.cyan(_(f"Tasks/{per}"))
+ term.cyan(_("Tasks/%(per)s") % {"per": per})
+ term.move_x(3 * col_width)
+ term.white(f"{tasks_per:.2f}")
+ term.move_x(4 * col_width)
@@ -308,8 +315,10 @@ def memory(run_once=False, workers=False, broker=None):
if not broker:
broker = get_broker()
try:
from blessed import Terminal
term = Terminal()
except:
except ImportError:
print(BLESSED_INSTALL_MESSAGE)
return
broker.ping()
@@ -389,7 +398,7 @@ def memory(run_once=False, workers=False, broker=None):
)
# memory available (MB)
memory_available = round(
psutil.virtual_memory().available / 1024 ** 2, 2
psutil.virtual_memory().available / 1024**2, 2
)
if memory_available_percentage < MEMORY_AVAILABLE_LOWEST_PERCENTAGE:
MEMORY_AVAILABLE_LOWEST_PERCENTAGE = memory_available_percentage
@@ -413,7 +422,7 @@ def memory(run_once=False, workers=False, broker=None):
print(
term.move(row, 4 * col_width)
+ term.center(
round(psutil.virtual_memory().total / 1024 ** 2, 2),
round(psutil.virtual_memory().total / 1024**2, 2),
width=col_width - 1,
)
)
@@ -475,17 +484,18 @@ def memory(run_once=False, workers=False, broker=None):
row += 1
print(
term.move(row, 0)
+ _("Available lowest (%): {} ({})").format(
str(MEMORY_AVAILABLE_LOWEST_PERCENTAGE),
MEMORY_AVAILABLE_LOWEST_PERCENTAGE_AT.strftime(
+ _("Available lowest (): %(memory_percent)s ((at)s)")
% {
"memory_percent": str(MEMORY_AVAILABLE_LOWEST_PERCENTAGE),
"at": MEMORY_AVAILABLE_LOWEST_PERCENTAGE_AT.strftime(
"%Y-%m-%d %H:%M:%S+00:00"
),
)
}
)
# for testing
if run_once:
return Stat.get_all(broker=broker)
print(term.move(row + 2, 0) + term.center("[Press q to quit]"))
print(term.move(row + 2, 0) + term.center(_("[Press q to quit]")))
val = term.inkey(timeout=1)
@@ -496,5 +506,5 @@ def get_ids():
for s in stat:
print(s.cluster_id)
else:
print("No clusters appear to be running.")
print(_("No clusters appear to be running."))
return True

View File

@@ -1,5 +1,6 @@
"""
The code is derived from https://github.com/althonos/pronto/commit/3384010dfb4fc7c66a219f59276adef3288a886b
The code is derived from
https://github.com/althonos/pronto/commit/3384010dfb4fc7c66a219f59276adef3288a886b
"""
import multiprocessing
import multiprocessing.queues

View File

@@ -19,16 +19,16 @@ def call_hook(sender, instance, **kwargs):
f = getattr(m, func)
except (ValueError, ImportError, AttributeError):
logger.error(
_(f"malformed return hook '{instance.hook}' for [{instance.name}]")
_("malformed return hook '%(hook)s' for [%(name)s]")
% {"hook": instance.hook, "name": instance.name}
)
return
try:
f(instance)
except Exception as e:
logger.error(
_(
f"return hook {instance.hook} failed on [{instance.name}] because {str(e)}"
)
_("return hook %(hook)s failed on [%(name)s] because %(error)s")
% {"hook": instance.hook, "name": instance.name, "error": str(e)}
)

View File

@@ -31,6 +31,7 @@ def async_task(func, *args, **kwargs):
"iter_cached",
"chain",
"broker",
"cluster",
"timeout",
)
q_options = keywords.pop("q_options", {})
@@ -52,7 +53,7 @@ def async_task(func, *args, **kwargs):
elif key in keywords:
task[key] = keywords.pop(key)
# don't serialize the broker
broker = task.pop("broker", get_broker())
broker = task.pop("broker", None) or get_broker(task.get("cluster"))
# overrides
if "cached" not in task and Conf.CACHED:
task["cached"] = Conf.CACHED
@@ -71,7 +72,7 @@ def async_task(func, *args, **kwargs):
return _sync(pack)
# push it
enqueue_id = broker.enqueue(pack)
logger.info(f"Enqueued {enqueue_id}")
logger.info(f"Enqueued [{broker.list_key}] {enqueue_id}")
logger.debug(f"Pushed {tag}")
return task["id"]
@@ -90,6 +91,7 @@ def schedule(func, *args, **kwargs):
:type next_run: datetime.datetime
:param cluster: optional cluster name.
:param cron: optional cron expression
:param intended_date_kwarg: optional identifier to pass intended schedule date.
:param kwargs: function keyword arguments.
:return: the schedule object.
:rtype: Schedule
@@ -102,6 +104,7 @@ def schedule(func, *args, **kwargs):
next_run = kwargs.pop("next_run", timezone.now())
cron = kwargs.pop("cron", None)
cluster = kwargs.pop("cluster", None)
intended_date_kwarg = kwargs.pop("intended_date_kwarg", None)
# check for name duplicates instead of am unique constraint
if name and Schedule.objects.filter(name=name).exists():
@@ -120,6 +123,7 @@ def schedule(func, *args, **kwargs):
next_run=next_run,
cron=cron,
cluster=cluster,
intended_date_kwarg=intended_date_kwarg,
)
# make sure we trigger validation
s.full_clean()
@@ -270,6 +274,7 @@ def fetch_cached(task_id, wait=0, broker=None):
hook=task.get("hook"),
args=task["args"],
kwargs=task["kwargs"],
cluster=task.get("cluster"),
started=task["started"],
stopped=task["stopped"],
result=task["result"],
@@ -340,6 +345,7 @@ def fetch_group_cached(group_id, failures=True, wait=0, count=None, broker=None)
hook=task.get("hook"),
args=task["args"],
kwargs=task["kwargs"],
cluster=task.get("cluster"),
started=task["started"],
stopped=task["stopped"],
result=task["result"],
@@ -600,7 +606,8 @@ class Chain:
def result(self, wait=0):
"""
return the full list of results from the chain when it finishes. blocks until timeout.
return the full list of results from the chain when it finishes. blocks until
timeout.
:param int wait: how many milliseconds to wait for a result
:return: an unsorted list of results
"""
@@ -611,7 +618,8 @@ class Chain:
def fetch(self, failures=True, wait=0):
"""
get the task result objects from the chain when it finishes. blocks until timeout.
get the task result objects from the chain when it finishes. blocks until
timeout.
:param failures: include failed tasks
:param int wait: how many milliseconds to wait for a result
:return: an unsorted list of task objects

View File

@@ -1,7 +1,5 @@
import os
import django
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -77,7 +75,7 @@ DATABASES = {
LANGUAGE_CODE = "en-us"
TIME_ZONE = "UTC"
TIME_ZONE = "Europe/Amsterdam"
USE_I18N = True
@@ -130,5 +128,5 @@ Q_CLUSTER = {
"testing": True,
"log_level": "DEBUG",
"django_redis": "default",
"redis": f"redis://{REDIS_HOST}:6379/0"
"redis": f"redis://{REDIS_HOST}:6379/0",
}

View File

@@ -2,12 +2,11 @@ import os
from time import sleep
import pytest
import redis
from django_q.brokers import Broker, get_broker
from django_q.conf import Conf
from django_q.humanhash import uuid
from django_q.tests.settings import REDIS_HOST, MONGO_HOST
from django_q.tests.settings import MONGO_HOST, REDIS_HOST
def test_broker(monkeypatch):

View File

@@ -1,8 +1,8 @@
from datetime import datetime
import os
import sys
import threading
import uuid as uuidlib
from datetime import datetime
from math import copysign
from multiprocessing import Event, Value
from time import sleep
@@ -11,9 +11,6 @@ from typing import Optional
import pytest
from django.utils import timezone
myPath = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, myPath + "/../")
from django_q.brokers import Broker, get_broker
from django_q.cluster import Cluster, Sentinel, monitor, pusher, save_task, worker
from django_q.conf import Conf
@@ -32,9 +29,12 @@ from django_q.tasks import (
result,
result_group,
)
from django_q.tests.tasks import TaskError, multiply
from django_q.tests.tasks import multiply, TaskError
from django_q.utils import add_months, add_years
myPath = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, myPath + "/../")
class WordClass:
def __init__(self):
@@ -64,7 +64,7 @@ def test_sync(broker):
@pytest.mark.django_db
def test_sync_raise_exception(broker):
with pytest.raises(Exception):
with pytest.raises(TaskError):
async_task("django_q.tests.tasks.raise_exception", broker=broker, sync=True)
@@ -409,6 +409,7 @@ def test_recycle(broker, monkeypatch):
assert Success.objects.count() == Conf.SAVE_LIMIT
broker.delete_queue()
@pytest.mark.django_db
def test_save_limit_per_func(broker, monkeypatch):
# set up the Sentinel
@@ -442,15 +443,14 @@ def test_save_limit_per_func(broker, monkeypatch):
# run monitor
monitor(result_queue)
assert Success.objects.count() == 3
assert set(Success.objects.filter().values_list('func', flat=True)) == {
'django_q.tests.tasks.countdown',
'django_q.tests.tasks.hello',
'django_q.tests.tasks.multiply',
assert set(Success.objects.filter().values_list("func", flat=True)) == {
"django_q.tests.tasks.countdown",
"django_q.tests.tasks.hello",
"django_q.tests.tasks.multiply",
}
broker.delete_queue()
@pytest.mark.django_db
def test_max_rss(broker, monkeypatch):
# set up the Sentinel
@@ -538,7 +538,6 @@ def test_attempt_count(broker, monkeypatch):
assert saved_task.attempt_count == 1
sleep(0.5)
# second save
old_stopped = task["stopped"]
task["stopped"] = timezone.now()
save_task(task, broker)
saved_task = Task.objects.get(id=task["id"])
@@ -770,6 +769,7 @@ def test_add_months():
assert new_date.month == 2
assert new_date.day == 29
@pytest.mark.django_db
def test_add_years():
# add some months

View File

@@ -1,9 +1,10 @@
import os
from datetime import timedelta
from datetime import datetime, timedelta
from multiprocessing import Event, Value
from unittest import mock
import pytest
import django
from django.core.exceptions import ValidationError
from django.db import IntegrityError
from django.test import override_settings
@@ -11,7 +12,7 @@ from django.utils import timezone
from django.utils.timezone import is_naive
from django_q.brokers import Broker, get_broker
from django_q.cluster import monitor, pusher, scheduler, worker, localtime
from django_q.cluster import localtime, monitor, pusher, scheduler, worker
from django_q.conf import Conf
from django_q.queues import Queue
from django_q.tasks import Schedule, fetch
@@ -21,11 +22,24 @@ from django_q.tests.testing_utilities.multiple_database_routers import (
TestingMultipleAppsDatabaseRouter,
TestingReplicaDatabaseRouter,
)
from django_q.utils import add_months
if django.VERSION < (4, 0):
# pytz is the default in django 3.2. Remove when no support for 3.2
from pytz import timezone as ZoneInfo
else:
try:
from zoneinfo import ZoneInfo
except ImportError:
from backports.zoneinfo import ZoneInfo
@pytest.fixture
def broker(monkeypatch) -> Broker:
"""Patches the Conf object setting the DJANGO_REDIS attribute allowing a default redis configuration."""
"""
Patches the Conf object setting the DJANGO_REDIS attribute allowing a default
redis configuration.
"""
monkeypatch.setattr(Conf, "DJANGO_REDIS", "default")
return get_broker()
@@ -36,25 +50,11 @@ def orm_broker(monkeypatch) -> None:
monkeypatch.setattr(Conf, "ORM", "default")
@pytest.fixture
def orm_no_replica_broker(orm_broker, monkeypatch) -> Broker:
"""Generates a Broker with a disabled read replica database configuration."""
monkeypatch.setattr(Conf, "HAS_REPLICA", False)
return get_broker(list_key="scheduler_test:q")
@pytest.fixture
def orm_replica_broker(orm_broker, monkeypatch) -> Broker:
"""Generates a Broker with read replica database configuration."""
monkeypatch.setattr(Conf, "HAS_REPLICA", True)
return get_broker(list_key="scheduler_test:q")
REPLICA_DATABASE_ROUTERS = [
f"{TestingReplicaDatabaseRouter.__module__}.{TestingReplicaDatabaseRouter.__name__}"
]
REPLICA_DATABASES = {
"default": {
"writable": {
"ENGINE": "django.db.backends.sqlite3",
"NAME": os.path.join(BASE_DIR, "db.sqlite3"),
},
@@ -65,7 +65,7 @@ REPLICA_DATABASES = {
}
MULTIPLE_APPS_DATABASE_ROUTERS = [
f"{TestingMultipleAppsDatabaseRouter.__module__}.{TestingMultipleAppsDatabaseRouter.__name__}"
f"{TestingMultipleAppsDatabaseRouter.__module__}.{TestingMultipleAppsDatabaseRouter.__name__}" # noqa: E501
]
MULTIPLE_APPS_DATABASES = {
"default": {
@@ -79,6 +79,109 @@ MULTIPLE_APPS_DATABASES = {
}
@pytest.mark.django_db
def test_scheduler_daylight_saving_time_daily(broker, monkeypatch):
# Set up a startdate in the Amsterdam timezone (without dst 1 hour ahead). The
# 28th of March 2021 is the day when sunlight saving starts (at 2 am)
monkeypatch.setattr(Conf, "TIME_ZONE", "Europe/Amsterdam")
tz = ZoneInfo('Europe/Amsterdam')
broker.list_key = "scheduler_test:q"
# Let's start a schedule at 1 am on the 27th of March. This is in AMS timezone.
# So, 2021-03-27 00:00:00 when saved (due to TZ being Amsterdam and saved in UTC)
start_date = datetime(2021, 3, 27, 1, 0, 0)
# Create schedule with the next run date on the start date. It will move one day
# forward when we run the scheduler
schedule = create_schedule(
"math.copysign",
1,
-1,
name="test math",
schedule_type=Schedule.DAILY,
next_run=start_date,
)
# Run scheduler so we get the next run date
scheduler(broker=broker)
schedule.refresh_from_db()
# It's now the day after exactly at midnight UTC
next_run = schedule.next_run
assert str(next_run) == "2021-03-28 00:00:00+00:00"
# In the Amsterdam timezone, it's 1 hour over midnight (+01)
next_run = next_run.astimezone(tz)
assert str(next_run) == "2021-03-28 01:00:00+01:00"
# Run scheduler so we get the next run date
scheduler(broker=broker)
schedule.refresh_from_db()
next_run = schedule.next_run
assert str(next_run) == "2021-03-28 23:00:00+00:00"
next_run = next_run.astimezone(tz)
# In the Amsterdam timezone, it's 1 hour over midnight (+02)
assert str(next_run) == "2021-03-29 01:00:00+02:00"
# Run scheduler so we get the next run date
scheduler(broker=broker)
schedule.refresh_from_db()
next_run = schedule.next_run
assert str(next_run) == "2021-03-29 23:00:00+00:00"
next_run = next_run.astimezone(tz)
assert str(next_run) == "2021-03-30 01:00:00+02:00"
# Create second schedule with the next run date on the start date. It will move
# one day forward when we run the scheduler
start_date = datetime(2021, 10, 29, 1, 0, 0)
schedule = create_schedule(
"django_q.tests.tasks.word_multiply",
2,
name="multiply",
schedule_type=Schedule.DAILY,
next_run=start_date,
)
# Run scheduler so we get the next run date
scheduler(broker=broker)
schedule.refresh_from_db()
next_run = schedule.next_run
assert str(next_run) == "2021-10-29 23:00:00+00:00"
# In the Amsterdam timezone, it's 1 hour over midnight (+02)
next_run = next_run.astimezone(tz)
assert str(next_run) == "2021-10-30 01:00:00+02:00"
# Run scheduler so we get the next run date
scheduler(broker=broker)
schedule.refresh_from_db()
next_run = schedule.next_run
assert str(next_run) == "2021-10-30 23:00:00+00:00"
# In the Amsterdam timezone, it's 1 hour over midnight (+02)
next_run = next_run.astimezone(tz)
assert str(next_run) == "2021-10-31 01:00:00+02:00"
# Run scheduler so we get the next run date
scheduler(broker=broker)
schedule.refresh_from_db()
next_run = schedule.next_run
assert str(next_run) == "2021-11-01 00:00:00+00:00"
# In the Amsterdam timezone, it's 1 hour over midnight (+01)
# Switch of DST
next_run = next_run.astimezone(tz)
assert str(next_run) == "2021-11-01 01:00:00+01:00"
@pytest.mark.django_db
def test_scheduler(broker, monkeypatch):
broker.list_key = "scheduler_test:q"
@@ -228,7 +331,30 @@ def test_scheduler(broker, monkeypatch):
# Done
broker.delete_queue()
monkeypatch.setattr(Conf, "PREFIX", "some_cluster_name")
# test bimonthly
schedule = create_schedule(
"django_q.tests.tasks.word_multiply",
2,
word="catch_up",
schedule_type=Schedule.BIMONTHLY,
)
scheduler(broker=broker)
schedule = Schedule.objects.get(pk=schedule.pk)
assert schedule.next_run.date() == add_months(timezone.now(), 2).date()
# test biweekly
schedule = create_schedule(
"django_q.tests.tasks.word_multiply",
2,
word="catch_up",
schedule_type=Schedule.BIWEEKLY,
)
scheduler(broker=broker)
schedule = Schedule.objects.get(pk=schedule.pk)
assert schedule.next_run.date() == (timezone.now() + timedelta(weeks=2)).date()
broker.delete_queue()
monkeypatch.setattr(Conf, "CLUSTER_NAME", "some_cluster_name")
# create a schedule on another cluster
schedule = create_schedule(
"math.copysign",
@@ -252,7 +378,7 @@ def test_scheduler(broker, monkeypatch):
# queue must be empty
assert task_queue.qsize() == 0
monkeypatch.setattr(Conf, "PREFIX", "default")
monkeypatch.setattr(Conf, "CLUSTER_NAME", "default")
# create a schedule on the same cluster
schedule = create_schedule(
"math.copysign",
@@ -277,61 +403,72 @@ def test_scheduler(broker, monkeypatch):
assert task_queue.qsize() == 1
@override_settings(
DATABASE_ROUTERS=REPLICA_DATABASE_ROUTERS, DATABASES=REPLICA_DATABASES
)
@pytest.mark.django_db
def test_scheduler_atomic_transaction_must_specify_a_database_when_no_replicas_are_used(
orm_no_replica_broker: Broker,
):
"""
GIVEN a environment without a read replica database
WHEN the scheduler is called
THEN the transaction atomic must be called using the configured database in the Conf.ORM settings.
"""
broker = orm_no_replica_broker
with mock.patch("django_q.cluster.db") as mocked_db:
scheduler(broker=broker)
# The router should correctly set the database to use!
mocked_db.transaction.atomic.assert_called_with(using=broker.connection.db)
def test_intended_schedule_kwarg(broker, monkeypatch):
broker.list_key = "scheduler_test:q"
broker.delete_queue()
run_date = timezone.now()-timedelta(hours=1)
schedule = create_schedule(
"math.copysign",
1,
-1,
name="test math",
hook="django_q.tests.tasks.result",
schedule_type=Schedule.HOURLY,
repeats=1,
next_run=run_date,
intended_date_kwarg='intended_date',
)
assert schedule.last_run() is None
assert schedule.intended_date_kwarg == 'intended_date'
# run scheduler
scheduler(broker=broker)
# set up the workflow
task_queue = Queue()
stop_event = Event()
stop_event.set()
# push it
pusher(task_queue, stop_event, broker=broker)
assert task_queue.qsize() == 1
task = task_queue.get()
assert 'intended_date' in task['kwargs']
assert task['kwargs']['intended_date'] == run_date.isoformat()
@override_settings(
DATABASE_ROUTERS=REPLICA_DATABASE_ROUTERS, DATABASES=REPLICA_DATABASES
)
@pytest.mark.django_db
def test_scheduler_atomic_transaction_must_specify_no_database_when_read_write_replicas_are_used(
orm_replica_broker: Broker,
def test_scheduler_atomic_must_specify_the_write_db(
orm_broker: Broker,
):
"""
GIVEN a environment with a read/write configured replica database
WHEN the scheduler is called
THEN the transaction must be called without a specific database, thus letting the database router pick.
THEN the transaction must be called with the write database.
"""
with mock.patch("django_q.cluster.db") as mocked_db:
scheduler(broker=orm_replica_broker)
# No specific databases should be set here, this is the job of the router!
mocked_db.transaction.atomic.assert_called_with()
broker = get_broker(list_key="scheduler_test:q")
with mock.patch("django_q.cluster.db.transaction") as mocked_db:
scheduler(broker=broker)
mocked_db.atomic.assert_called_with(using="writable")
@override_settings(
DATABASE_ROUTERS=MULTIPLE_APPS_DATABASE_ROUTERS, DATABASES=MULTIPLE_APPS_DATABASES
)
@pytest.mark.django_db
def test_scheduler_atomic_transaction_must_specify_the_database_based_on_router_redirection(
orm_no_replica_broker: Broker,
def test_scheduler_atomic_must_specify_the_database_based_on_router_redirection(
orm_broker: Broker,
):
"""
GIVEN a environment without a read replica database
WHEN the scheduler is called
THEN the transaction atomic must be called using the configured database in the Conf.ORM settings.
THEN the transaction atomic must be called using the default connection.
"""
broker = orm_no_replica_broker
with mock.patch("django_q.cluster.db") as mocked_db:
broker = get_broker(list_key="scheduler_test:q")
with mock.patch("django_q.cluster.db.transaction") as mocked_db:
scheduler(broker=broker)
# The router should correctly set the database to use!
assert broker.connection.db == "default"
mocked_db.transaction.atomic.assert_called_with(using=broker.connection.db)
mocked_db.atomic.assert_called_with(using="default")
def test_localtime():

View File

@@ -12,9 +12,9 @@ class TestingReplicaDatabaseRouter:
def db_for_write(self, model, **hints):
"""
Always write to DEFAULT database
Always write to WRITABLE database
"""
return "default"
return "writable"
class TestingMultipleAppsDatabaseRouter:

View File

@@ -1,7 +1,23 @@
import datetime
from datetime import date
from django.utils.timezone import make_aware
from datetime import datetime
import calendar
import inspect
from datetime import date
import django
from django.utils import timezone
from django.conf import settings
from django_q.conf import Conf
if django.VERSION < (4, 0):
# pytz is the default in django 3.2. Remove when no support for 3.2
from pytz import timezone as ZoneInfo
else:
try:
from zoneinfo import ZoneInfo
except ImportError:
from backports.zoneinfo import ZoneInfo
# credits: https://stackoverflow.com/a/4131114
# Made them aware of timezone
@@ -9,22 +25,50 @@ def add_months(d, months):
month = d.month - 1 + months
year = d.year + month // 12
month = month % 12 + 1
day = min(d.day, calendar.monthrange(year,month)[1])
day = min(d.day, calendar.monthrange(year, month)[1])
return d.replace(year=year, month=month, day=day)
# credits: https://stackoverflow.com/a/15743908
# Changed the last line to make it a little easier to read and changed it to move February 29 to 28 next year
# Also made them aware of timezone
# Changed the last line to make it a little easier to read and changed it to move
# February 29 to 28 next year.
def add_years(d, years):
"""Return a date that's `years` years after the date (or datetime)
object `d`. Return the same calendar date (month and day) in the
destination year, if it exists, otherwise use the previous day
(thus changing February 29 to February 28).
"""
try:
return d.replace(year = d.year + years)
return d.replace(year=d.year + years)
except ValueError:
new_date = d + (date(d.year + years, 3, 1) - date(d.year, 3, 1))
return d.replace(year=new_date.year, month=new_date.month, day=new_date.day)
def get_func_repr(func):
# convert func to string
if inspect.isfunction(func):
return f"{func.__module__}.{func.__name__}"
elif inspect.ismethod(func) and hasattr(func.__self__, "__name__"):
return (
f"{func.__self__.__module__}." f"{func.__self__.__name__}.{func.__name__}"
)
else:
return str(func) if func else None
def localtime(value=None) -> datetime:
"""Override for timezone.localtime to deal with naive times and local times"""
if settings.USE_TZ:
if django.VERSION >= (4, 0) and settings.USE_DEPRECATED_PYTZ:
import pytz
convert_to_tz = pytz.timezone(Conf.TIME_ZONE)
else:
convert_to_tz = ZoneInfo(Conf.TIME_ZONE)
return timezone.localtime(value=value, timezone=convert_to_tz)
if value is None:
return datetime.now()
else:
return value

View File

@@ -52,6 +52,30 @@ You can have multiple clusters on multiple machines, working on the same queue a
- They use the same cluster name. See :doc:`configure`
- They share the same ``SECRET_KEY`` for Django.
.. _multiple-queues
Multiple Queues
-----------------
You can have multiple queues in one Django site, and use multiple cluster to work on each queue.
Different queues are identified by different queue names which are also cluster names.
To run an alternate cluster, e.g. to work on the 'long' queue, start your cluster with command::
# On Linux
$ Q_CLUSTER_NAME=long python manage.py qcluster
# On Windows
$ python manage.py qcluster --name long
You can set different Q_CLUSTER options for alternative clusters, such as 'timeout', 'queue_limit'
and any other options which are valid in :doc:`configure`. See :ref:`alt-clusters`.
.. note::
To use multiple queue, use the keyword argument `cluster` in async_task() and schedule():
* if `cluster` is not set (the default), async_task() and schedule() will be handled by the default cluster;
* if `cluster` is set, only clusters with matching cluster name will run the task or do the schedule.
Using a Procfile
----------------
If you host on `Heroku <https://heroku.com>`__ or you are using `Honcho <https://github.com/nickstenning/honcho>`__ you can start the cluster from a :file:`Procfile` with an entry like this::

View File

@@ -16,12 +16,10 @@
import os
import sys
import sphinx_rtd_theme
myPath = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, myPath + '/../')
os.environ['DJANGO_SETTINGS_MODULE'] = 'django_q.tests.settings'
nitpick_ignore = [('py:class', 'datetime')]
sys.path.insert(0, myPath + "/../")
os.environ["DJANGO_SETTINGS_MODULE"] = "django_q.tests.settings"
nitpick_ignore = [("py:class", "datetime")]
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
@@ -37,50 +35,54 @@ nitpick_ignore = [('py:class', 'datetime')]
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx_rtd_theme',
'sphinx.ext.todo',
'sphinx.ext.intersphinx',
"sphinx_rtd_theme",
"sphinx.ext.todo",
"sphinx.ext.intersphinx",
# 'sphinx.ext.autodoc'
]
intersphinx_mapping = {'python': ('https://docs.python.org/3.8', None),
'django': ('https://docs.djangoproject.com/en/2.2/',
'https://docs.djangoproject.com/en/2.2/_objects/')}
intersphinx_mapping = {
"python": ("https://docs.python.org/3.8", None),
"django": (
"https://docs.djangoproject.com/en/2.2/",
"https://docs.djangoproject.com/en/2.2/_objects/",
),
}
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
templates_path = ["_templates"]
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
source_suffix = ".rst"
# The encoding of source files.
# source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
master_doc = "index"
# General information about the project.
project = 'Django Q2'
copyright = '2015-2021, Ilan Steemers - 2022, Stan Triepels'
author = 'Ilan Steemers, Stan Triepels'
project = "Django Q2"
copyright = "2015-2021, Ilan Steemers - 2022, Stan Triepels"
author = "Ilan Steemers, Stan Triepels"
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = '1.4'
version = "1.5"
# The full version, including alpha/beta/rc tags.
release = '1.4.2'
release = "1.5.2"
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = 'en'
language = "en"
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
@@ -90,7 +92,7 @@ language = 'en'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build']
exclude_patterns = ["_build"]
# The reST default role (used for this markup: `text`) to use for all
# documents.
@@ -108,7 +110,7 @@ add_module_names = False
# show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
pygments_style = "sphinx"
# A list of ignored prefixes for module index sorting.
# modindex_common_prefix = []
@@ -124,7 +126,7 @@ todo_include_todos = True
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'sphinx_rtd_theme'
html_theme = "sphinx_rtd_theme"
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
@@ -136,11 +138,11 @@ html_theme_options = {
# 'github_banner': True,
}
html_sidebars = {
'**': [
'about.html',
'navigation.html',
'relations.html',
'searchbox.html',
"**": [
"about.html",
"navigation.html",
"relations.html",
"searchbox.html",
]
}
# Add any paths that contain custom themes here, relative to this directory.
@@ -161,12 +163,12 @@ html_sidebars = {
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
html_favicon = '_static/favicon.ico'
html_favicon = "_static/favicon.ico"
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
html_static_path = ["_static"]
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
@@ -229,20 +231,17 @@ html_static_path = ['_static']
# html_search_scorer = 'scorer.js'
# Output file base name for HTML help builder.
htmlhelp_basename = 'DjangoQ2doc'
htmlhelp_basename = "DjangoQ2doc"
# -- Options for LaTeX output ---------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
# 'preamble': '',
# Latex figure (float) alignment
# 'figure_align': 'htbp',
}
@@ -251,8 +250,7 @@ latex_elements = {
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'DjangoQ2.tex', 'Django Q2 Documentation',
'Ilan Steemers', 'manual'),
(master_doc, "DjangoQ2.tex", "Django Q2 Documentation", "Ilan Steemers", "manual"),
]
# The name of an image file (relative to this directory) to place at the top of
@@ -280,10 +278,7 @@ latex_documents = [
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'djangoq2', 'Django Q2 Documentation',
[author], 1)
]
man_pages = [(master_doc, "djangoq2", "Django Q2 Documentation", [author], 1)]
# If true, show URL addresses after external links.
@@ -296,9 +291,15 @@ man_pages = [
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'DjangoQ2', 'Django Q2 Documentation',
author, 'DjangoQ2', 'A multiprocessing distributed task queue for Django.',
'Miscellaneous'),
(
master_doc,
"DjangoQ2",
"Django Q2 Documentation",
author,
"DjangoQ2",
"A multiprocessing distributed task queue for Django.",
"Miscellaneous",
),
]
# Documents to append as an appendix to all manuals.

View File

@@ -20,7 +20,18 @@ Configuration is handled via the ``Q_CLUSTER`` dictionary in your :file:`setting
'redis': {
'host': '127.0.0.1',
'port': 6379,
'db': 0, }
'db': 0, },
'ALT_CLUSTERS': {
'long': {
'timeout': 3000,
'retry': 3600,
'max_attempts': 2,
},
'short': {
'timeout': 10,
'max_attempts': 1,
},
}
}
All configuration settings are optional:
@@ -70,6 +81,13 @@ Set this to something that makes sense for your project. Can be overridden for i
See :ref:`retry` for details how to set values for timeout and retry.
.. _time_zone:
time_zone
~~~~~~~
The timezone that is used for task scheduling. Use this if you are having issue with DST. The scheduler uses UTC to calculate the next date and will therefore ignore any DST changes. This will cause 1 hour or 0.5 hour changes in the schedule when time is moved one hour ahead or back. Defaults to `settings.TIME_ZONE` if `USE_TZ` is enabled.
.. _ack_failures:
ack_failures
@@ -310,16 +328,6 @@ Using the Django ORM backend will also enable the Queued Tasks table in the Admi
If you need better performance , you should consider using a different database backend than the main project.
Set ``orm`` to the name of that database connection and make sure you run migrations on it using the ``--database`` option.
When using the Django database as a message broker, you can set the ``has_replica`` boolean keyword to ensure Django-Q2 works properly letting a `Database Router <https://docs.djangoproject.com/en/3.2/topics/db/multi-db/>`__. ::
# example ORM broker connection with replica database
Q_CLUSTER = {
...
'orm': 'default',
'has_replica': True
}
.. _mongo_configuration:
mongo
@@ -462,6 +470,24 @@ As a rule of thumb; cpu_affinity 1 favors repetitive short running tasks, while
*Psutil does not support cpu affinity on OS X at this time.*
.. _alt-clusters:
ALT_CLUSTERS
~~~~~~~~~~~~
For multiple clusters working on multiple queues to run in one Django site.
ALT_CLUSTERS should be a dict with cluster_name as its key, and the value is the configuration for the cluster
with the key as its name. The configuration items are consistent with Q_CLUSTER,
except for a few items such as name/cluster_name/ALT_CLUSTER, which are not available of course.
See :ref:`multiple-queues`.
.. note::
For a cluster, if its name is in ALT_CLUSTERS, the config item in ALT_CLUSTER will override
the same config item in the Q_CLUSTER root. Other config items in Q_CLUSTER root remain in effect for this cluster.
.. py:module:: django_q
.. rubric:: Footnotes

View File

@@ -27,7 +27,7 @@ Features
- Rollbar and Sentry support
Django Q2 is tested with: Python 3.7, 3.8, 3.9 and 3.10, Django 2.2.x and 3.2.x
Django Q2 is tested with: Python 3.8, 3.9, 3.10 and 3.11. Works with Django 3.2.x, 4.1.x and 4.2.x
Currently available in English, German and French.

View File

@@ -4,7 +4,7 @@ Installation
- Install the latest version with pip::
$ pip install django-q
$ pip install django-q2
- Add :mod:`django_q` to ``INSTALLED_APPS`` in your projects :file:`settings.py`::
@@ -27,12 +27,12 @@ Installation
Requirements
------------
Django Q2 is tested for Python 3.7, 3.8, 3.9 and 3.10
Django Q2 is tested for Python 3.8, 3.9, 3.10 and 3.11
- `Django <https://www.djangoproject.com>`__
Django Q2 aims to use as much of Django's standard offerings as possible.
The code is tested against Django versions `3.2.x` and `4.1.x`.
The code is tested against Django versions `3.2.x`, `4.1.x` and`4.2.x`.
- `Django-picklefield <https://github.com/gintas/django-picklefield>`__
@@ -55,6 +55,10 @@ Optional
$ pip install psutil
- `setproctitle <https://github.com/dvarrazzo/py-setproctitle>`__ python module to customize the process title by Daniele Varrazzo', is an optional requirement used to set informative process titles::
$ pip install setproctitle
- `Hiredis <https://github.com/redis/hiredis>`__ parser. This C library maintained by the core Redis team is faster than the standard PythonParser during high loads::
$ pip install hiredis
@@ -125,7 +129,7 @@ Other known issues are:
Python
~~~~~~
Current tests are performed with 3.7, 3.8, 3.9 and 3.10
Current tests are performed with 3.8, 3.9, 3.10 and 3.11
If you do encounter any regressions with earlier versions, please submit an issue on `github <https://github.com/GDay/django-q2>`__
Open-source packages
@@ -136,7 +140,7 @@ You can reference the `requirements <https://github.com/GDay/django-q2/blob/mast
Django
~~~~~~
We strive to be compatible with last two major version of Django.
At the moment this means we support the 3.2.x and 4.1.x releases.
At the moment this means we support the 3.2.x, 4.1.x and 4.2.x releases.
Since we are now no longer supporting Python 2, we can also not support older versions of Django that do not support Python >= 3.6
For this you can always use older releases, but they are no longer maintained.

View File

@@ -69,6 +69,10 @@ You can change this by setting the :ref:`catch_up` configuration setting to ``Fa
The scheduler will then skip execution of scheduled events in the past.
Instead those tasks will run once when the cluster starts again and the scheduler will find the next available slot in the future according to original schedule parameters.
When :ref:`catch_up` is to ``True`` it may be useful for the task to know what was the date and time it was originally intended to run at.
To achieve this, pass an identifier name to parameter `intended_date_kwarg` when creating the schedule. The intended datetime will then be passed - in isoformat string - as
a kwarg with that identifier name to the task that has been created.
Management Commands
-------------------
@@ -123,9 +127,10 @@ Reference
:param int repeats: Number of times to repeat schedule. -1=Always, 0=Never, n =n.
:param datetime next_run: Next or first scheduled execution datetime.
:param str cluster: optional cluster name. Task will be executed only on a cluster with a matching :ref:`name`.
:param str intended_date_kwarg: optional identifier to pass intended schedule date.
:param dict q_options: options passed to async_task for this schedule
:param kwargs: optional keyword arguments for the scheduled function.
.. note::
q_options does not accept the 'broker' key with a broker instance but accepts a 'broker_name' key instead. This can be used to specify the broker connection name to assign the task. If a broker with the specified name does not exist or is not running at the moment of placing the task in queue it fallbacks to the random broker/queue that handled the schedule.
@@ -165,7 +170,7 @@ Reference
.. py:attribute:: TYPE
:attr:`ONCE`, :attr:`MINUTES`, :attr:`HOURLY`, :attr:`DAILY`, :attr:`WEEKLY`, :attr:`MONTHLY`, :attr:`QUARTERLY`, :attr:`YEARLY`, :attr:`CRON`
:attr:`ONCE`, :attr:`MINUTES`, :attr:`HOURLY`, :attr:`DAILY`, :attr:`WEEKLY`, :attr:`BIWEEKLY`, :attr:`MONTHLY`, :attr:`BIMONTHLY`, :attr:`QUARTERLY`, :attr:`YEARLY`, :attr:`CRON`
.. py:attribute:: minutes
@@ -183,9 +188,13 @@ Reference
When set to -1, this will keep counting down.
.. py:attribute:: cluster
Task will be executed only on a cluster with a matching :ref:`name`.
.. py:attribute:: intended_date_kwarg
Name of kwarg to pass intended schedule date.
.. py:attribute:: next_run
Datetime of the next scheduled execution.
@@ -224,10 +233,23 @@ Reference
`'W'` the task will run every week on they day and time of the first run.
.. py:attribute:: BIWEEKLY
`'BW'` the task will run once every two weeks on they day and time of the first run.
.. py:attribute:: MONTHLY
`'M'` the tasks runs every month on they day and time of the last run.
.. note::
Months are tricky. If you schedule something on the 31st of the month and the next month has only 30 days or less, the task will run on the last day of the next month.
It will however continue to run on that day, e.g. the 28th, in subsequent months.
.. py:attribute:: BIMONTHLY
`'BM'` the tasks runs once every two months on they day and time of the last run.
.. note::
Months are tricky. If you schedule something on the 31st of the month and the next month has only 30 days or less, the task will run on the last day of the next month.

1635
poetry.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,6 +1,6 @@
[tool.poetry]
name = "django-q2"
version = "1.4.2"
version = "1.5.2"
packages = [
{ include = "django_q" },
]
@@ -26,10 +26,10 @@ classifiers = [
'Operating System :: MacOS',
'Programming Language :: Python',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'Topic :: Internet :: WWW/HTTP',
'Topic :: System :: Distributed Computing',
'Topic :: Software Development :: Libraries :: Python Modules',
@@ -44,7 +44,7 @@ include = ['CHANGELOG.md']
[tool.poetry.dependencies]
python = ">=3.8.14, <4"
python = ">=3.8, <4"
django = ">=3.2"
django-picklefield = "^3.1"
@@ -59,6 +59,7 @@ croniter = { version = "^1.3.7", optional = true }
django-q-rollbar = {version = ">=0.1", optional = true}
django-q-sentry = {version = ">=0.1", optional = true}
redis = {version = "^4.3.4", optional = true}
setproctitle = {version = "^1.3.2", optional = true}
[tool.poetry.dev-dependencies]
@@ -70,9 +71,9 @@ black = "^22.10.0"
isort = {extras = ["requirements_deprecated_finder"], version = "^5.10.1"}
[tool.poetry.extras]
requires = ["poetry_core>=1.0.0"]
requires = ["poetry_core"]
build-backend = ["poetry.core.masonry.api"]
testing = ["django-redis", "croniter", "hiredis", "psutil", "iron-mq", "boto3", "pymongo", "blessed", "redis"]
testing = ["django-redis", "croniter", "hiredis", "psutil", "iron-mq", "boto3", "pymongo", "blessed", "redis", "setproctitle"]
rollbar = ["django-q-rollbar"]
sentry = ["django-q-sentry"]

2
tox.ini Normal file
View File

@@ -0,0 +1,2 @@
[flake8]
max-line-length=88