Acquire: a released resource stays unusable for as long as the releasing goroutine keeps running (~1 ms per contended acquire)

[mikluko]

What happens

Under contention, a resource that Release hands to a waiter is not used by that waiter until the releasing goroutine blocks, is preempted, or another P steals the waiter. If the releaser goes on doing CPU work after Release (decoding, publishing, the rest of a request handler), the resource is held by nobody for that long. With every P busy that is the whole of the releaser's remaining run: ~1 ms per contended acquire in our service, ~1.85 ms in the benchmark below at 2 ms of post-release work, and throughput down by a third.

Why

releaseAcquiredResource releases acquireSem (x/sync/semaphore), which wakes the first waiter by closing its channel. The runtime readies a goroutine unblocked by a channel operation into the waker's own P as runnext, and runnext is the last thing a stealing P tries (stealWork, on its final pass, after runqgrab's 3 µs pause). So the waiter runs when its waker parks, unless a P is idle. The semaphore is strictly FIFO, so a goroutine that is already running and calls Acquire queues behind the sleeping waiter rather than taking the free token; that is what keeps the token idle. sync.Mutex avoids the same trap by letting running goroutines barge, switching to FIFO handoff only after a waiter has starved for 1 ms.

Reproduction

https://gist.github.com/mikluko/57dbb8a46e2d38b9b8b56cb4d7675f77 — one _test.go, public API only: a pool of 4, ~100 waiters, a 1 ms parked hold, and per arm the CPU work the releaser does after Release. residue-µs is 4 × ns/op − hold, the time per acquire the resource is held by nobody. go test -bench . -cpu 4, linux/arm64, Go 1.26.7, puddle v2.2.2:

BenchmarkPuddle/work=0-4            341320 ns/op      5 residue-µs
BenchmarkPuddle/work=1ms-4          381999 ns/op    335 residue-µs
BenchmarkPuddle/work=1ms/yield-4    256290 ns/op     17 residue-µs
BenchmarkPuddle/work=2ms-4          831440 ns/op   1853 residue-µs
BenchmarkPuddle/work=2ms/yield-4    507890 ns/op     43 residue-µs

The yield arms add runtime.Gosched() right after Release. BenchmarkSemaphore in the gist runs the same load on the bare x/sync semaphore and matches arm for arm, so the pool's mutex and idle stack are not involved. darwin/arm64 shows the same shape.

We first met it in pgxpool (v5.10.0): 4 connections shared by 200 workers, Acquire at 98% of a hot span, and a flat ~1.1 ms of token time per acquire that no tracer could see inside, found by subtracting the traced connection hold from MaxConns/λ. The benchmark reproduces that subtraction.

Possible fixes

1. runtime.Gosched() after p.acquireSem.Release(1) in releaseAcquiredResource (after the mutex is unlocked): one line, and in our rig it turns the residue into tens of µs and raises throughput 15–80% depending on post-release work. Its cost is that the releaser goes to the global run queue, and a parked hold nearby lengthened by 100–300 µs in the same runs.
2. A barging fast path in Acquire: try the semaphore before queueing even when waiters exist, with a starvation guard in the style of sync.Mutex. That removes the idle-token window without a yield, at the price of strict FIFO, which x/sync users rely on (x/sync/semaphore: document ordering of Weighted.Acquire golang/go#56910), so it would have to be puddle's own semaphore.
3. Keep the behaviour and document it. We have the yield in our own code, after every release, and it works; the argument for a fix inside the pool is that every pgxpool user under contention pays this and nothing in the pool's stats points at it (AcquireDuration includes it as ordinary queueing).

#24 replaced the previous sync.Cond design for a scheduler reason too, and the semaphore fixed that pathology; this one is the FIFO handoff it brought in, seen only when the pool is saturated by releasers that keep running.