mirror of
https://github.com/django-q2/django-q2.git
synced 2026-09-22 17:28:12 +08:00
Andy Galasso
480b77dfe4
* AttributeError when start_event is None, and guard process faster stop This commit has two related fixes that come into play when frequently shutting down and restarting the cluster. If a SIGINT or SIGTERM signal is received while the main process is waiting for the sentinel process to set start_event, the signal handler sets start_event to None, and the loop that polls start_event will raise an unhandled AttributeError attempting to check start_event.is_set(). The second issue is that the guard process sleeps for the guard cycle interval in each iteration before checking the stop event, preventing the guard cycle from terminating until the sleep completes. The cluster can be made more responsive to a shutdown request by using the event's wait(cycle) method rather than time.sleep(cycle) so the process wakes immediately when the event is set. The commit also fixes the case where the guard process sleeps for an extra cycle when the cycle counter happens to have been reset to zero on the loop iterations when the scheduler is called. This fix helps in our environment where we are frequently stopping and starting the cluster and where we have increased the guard cycle setting to several seconds. * Add sentinel premature death check while waiting for sentinel to start. Suggested by github copilot [here](https://github.com/django-q2/django-q2/pull/305/changes#r3143964420) * Add test case test_cluster_early_stop Add a test case to validate stopping the cluster before the sentinel has set the cluster's start_event. The test deterministically triggers the AttributeError when start_event is None (without the PR's fix in cluster.py). As requested by copilot: https://github.com/django-q2/django-q2/pull/305#discussion_r3143964399 * Add test test_cluster_stop_responsive Ensure that stopping the cluster is responsive and does not wait for a full GUARD_CYCLE to stop the cluster. As requested by copilot: https://github.com/django-q2/django-q2/pull/305/changes#r3143964414 * Address copilot review comments. 1. Do not raise RuntimeError if sentinel early exit was caused by SIGINT/SIGTERM stopping the cluster while waiting for the sentinel to start. Addresses https://github.com/django-q2/django-q2/pull/305#discussion_r3298618046 2. Ensure that the monkeypatched broker instance remains pickleable so the test_cluster_early_stop test will also work on platforms that use the spawn Process start method. Addresses https://github.com/django-q2/django-q2/pull/305#discussion_r3298618086 3. Do not allow the test_cluster_early_stop test to hang if the sentinel process or main test process terminates unexpectedly without setting the sentinel_event or test_event. Addresses https://github.com/django-q2/django-q2/pull/305#discussion_r3298618102