mirror of
https://github.com/django-q2/django-q2.git
synced 2026-10-07 16:58:13 +08:00
Updated architecture docs
This commit is contained in:
+13
-15
@@ -101,17 +101,15 @@ Pusher
|
|||||||
""""""
|
""""""
|
||||||
|
|
||||||
The pusher process continuously checks the Redis list for new task
|
The pusher process continuously checks the Redis list for new task
|
||||||
packages and pushes them on the Task Queue.
|
packages. It checks the signing and unpacks the task to the Task Queue.
|
||||||
|
|
||||||
Worker
|
Worker
|
||||||
""""""
|
""""""
|
||||||
|
|
||||||
A worker process pulls a package of the Task Queue and checks the signing and unpacks the task.
|
A worker process pulls a task of the Task Queue and it sets a shared countdown timer with :ref:`sentinel` indicating it is about to start work.
|
||||||
Before executing the task it set a timer on the :ref:`sentinel` indicating its about to start work.
|
The worker then tries to execute the task and afterwards the timer is reset and any results (including errors) are saved to the package.
|
||||||
Afterwards it the timer is reset and any results (including errors) are saved to the package.
|
|
||||||
Irrespective of the failure or success of any of these steps, the package is then pushed onto the Result Queue.
|
Irrespective of the failure or success of any of these steps, the package is then pushed onto the Result Queue.
|
||||||
|
|
||||||
|
|
||||||
Monitor
|
Monitor
|
||||||
"""""""
|
"""""""
|
||||||
|
|
||||||
@@ -125,38 +123,38 @@ Sentinel
|
|||||||
|
|
||||||
The sentinel spawns all process and then checks the health of all
|
The sentinel spawns all process and then checks the health of all
|
||||||
workers, including the pusher and the monitor. This includes checking timers on each worker for timeouts.
|
workers, including the pusher and the monitor. This includes checking timers on each worker for timeouts.
|
||||||
In case of a sudden death or timeout, it will reincarnate the failing processes. When a stop signal, the sentinel will halt the
|
In case of a sudden death or timeout, it will reincarnate the failing processes. When a stop signal is received, the sentinel will halt the
|
||||||
pusher and instruct the workers and monitor to finish the remaining items. See :ref:`stop_procedure`
|
pusher and instruct the workers and monitor to finish the remaining items. See :ref:`stop_procedure`
|
||||||
|
|
||||||
Timeouts
|
Timeouts
|
||||||
""""""""
|
""""""""
|
||||||
Before each task execution the worker resets a timer on the sentinel and resets it again after execution.
|
Before each task execution the worker sets a countdown timer on the sentinel and resets it again after execution.
|
||||||
Meanwhile the the sentinel checks if the timers don't exceed the timeout amount, in which case it will terminate the worker and reincarnate a new one.
|
Meanwhile the sentinel checks if the timers don't reach zero, in which case it will terminate the worker and reincarnate a new one.
|
||||||
|
|
||||||
Scheduler
|
Scheduler
|
||||||
"""""""""
|
"""""""""
|
||||||
Once a minute the scheduler checks for any scheduled tasks that should be starting.
|
Twice a minute the scheduler checks for any scheduled tasks that should be starting.
|
||||||
|
|
||||||
- Creates a task from the schedule
|
- Creates a task from the schedule
|
||||||
- Subtracts 1 from :attr:`django_q.Schedule.repeats`
|
- Subtracts 1 from :attr:`django_q.Schedule.repeats`
|
||||||
- Sets the next run time if there are repeats left or if its negative.
|
- Sets the next run time if there are repeats left or if it has a negative value.
|
||||||
|
|
||||||
.. _stop_procedure:
|
.. _stop_procedure:
|
||||||
|
|
||||||
Stop procedure
|
Stop procedure
|
||||||
""""""""""""""
|
""""""""""""""
|
||||||
|
|
||||||
When a stop signal is given, the sentinel exits the guard loop and instructs the pusher to stop pushing.
|
When a stop signal is received, the sentinel exits the guard loop and instructs the pusher to stop pushing.
|
||||||
Once this is confirmed, the sentinel pushes poison pills onto the task queue and will wait for all the workers to die.
|
Once this is confirmed, the sentinel pushes poison pills onto the task queue and will wait for all the workers to exit.
|
||||||
This ensures that the queue is emptied before the workers exit.
|
This ensures that the task queue is emptied before the workers exit.
|
||||||
Afterwards the sentinel waits for the monitor to empty the result and then the stop procedure is complete.
|
Afterwards the sentinel waits for the monitor to empty the result queue and the stop procedure is complete.
|
||||||
|
|
||||||
- Send stop event to pusher
|
- Send stop event to pusher
|
||||||
- Wait for pusher to exit
|
- Wait for pusher to exit
|
||||||
- Put poison pills in the Task Queue
|
- Put poison pills in the Task Queue
|
||||||
- Wait for all the workers to clear the queue and stop
|
- Wait for all the workers to clear the queue and stop
|
||||||
- Put a poison pill on the Result Queue
|
- Put a poison pill on the Result Queue
|
||||||
- Wait for monitor to process remaining results
|
- Wait for monitor to process remaining results and exit
|
||||||
- Signal that we have stopped
|
- Signal that we have stopped
|
||||||
|
|
||||||
.. warning::
|
.. warning::
|
||||||
|
|||||||
Reference in New Issue
Block a user