[PATCH for-next] io_uring/futex: use GFP_KERNEL_ACCOUNT for futex data allocation

[PATCH for-next] io_uring/futex: use GFP_KERNEL_ACCOUNT for futex data allocation

[Jens Axboe]

Be a bit nicer and ensure it plays nice with memcg accounting.

Signed-off-by: Jens Axboe <axboe@kernel.dk>

---

diff --git a/io_uring/futex.c b/io_uring/futex.c
index 11bfff5a80df..1dabcfd503b8 100644
--- a/io_uring/futex.c
+++ b/io_uring/futex.c
@@ -186,7 +186,7 @@ int io_futexv_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 		return -EINVAL;
 
 	ifd = kzalloc(struct_size_t(struct io_futexv_data, futexv, iof->futex_nr),
-			GFP_KERNEL);
+			GFP_KERNEL_ACCOUNT);
 	if (!ifd)
 		return -ENOMEM;
 
-- 
Jens Axboe

Re: [PATCH for-next] io_uring/futex: use GFP_KERNEL_ACCOUNT for futex data allocation

[Chengfeng Lin]

Hi Jens,

I tested 6e0d71c288fd against its direct parent on bare metal. In a narrow
io_uring futex WAITV -> WAKE workload, the child was 8.09% slower. A
separate 424-line standalone reproducer showed a 6.49% slowdown. A matched
scalar WAIT -> WAKE control changed by -0.55%. All compared kernels actually
ran with preempt=full.

The test system was a Core i7-12700KF with 32 GiB RAM. The workload was pinned
to P-core CPU 2, with the governor and EPP set to performance, Turbo disabled,
and GCC 15.2.0.

#regzbot introduced: 6e0d71c288fdcf5866f5d0c6cde850a091cc3c55
#regzbot title: io_uring futex WAITV accounted-allocation slowdown

This is a focused synthetic microbenchmark, not an application benchmark. It
uses one raw-UAPI ring and eight independent wait vectors. Each vector has
eight cacheline-separated private futex words. A timed cycle submits eight
IORING_OP_FUTEX_WAITV requests, wakes element 3 in every vector, and validates
all 16 CQEs. An untimed wake then verifies that the seven remaining waiters in
each vector were removed.

I used a fresh boot for each point:

  816095894c0f parent A -> 6e0d71c288fd child -> 816095894c0f parent B

Each point had 3 warm-up rounds and 15 measured rounds. Every measured round
ran 512 cycles, or 4,096 WAITV/wake pairs. Results in ns/pair were:

  implementation   parent A     child   parent B   child vs midpoint
  formal             1045.687  1128.704   1042.755        +8.091%
  standalone         1040.270  1110.916   1046.198        +6.488%

The formal drop-first result was +8.080%, parent drift was -0.280%, and the
maximum CV was 0.151%. The standalone drop-first result was +6.511%, with
+0.570% parent drift. All 90 WAITV timing rows passed the CQE, returned-value,
overflow, residual-waiter, and CPU checks. Untimed child traces also hit
io_futexv_prep(), io_futexv_wait(), io_futexv_complete(), and the wake path
with the expected request counts.

The exact source change is only GFP_KERNEL -> GFP_KERNEL_ACCOUNT for the
per-WAITV data allocation. I understand the memcg-accounting purpose and am
not suggesting a revert.

As a separate current-baseline diagnostic, I compared unmodified v7.2 against
a direct child changing only this allocation back to GFP_KERNEL. The
no-account child was 8.12% and 9.00% faster in the formal and standalone WAITV
tests, while the scalar control changed by +0.09%. These values use a different
baseline and are not combined with the direct-parent results above.

Is this per-WAITV cost an expected accounting trade-off, or could the same
memcg accounting be retained with lower per-request overhead?

Evidence bundle:

  https://github.com/lcf0399/linux-regression-evidence/tree/25ed417f566fc83b942e1787a0b53d555ccec291/io-uring-futex-waitv-accounted-allocation

Standalone reproducer:

  https://github.com/lcf0399/linux-regression-evidence/tree/25ed417f566fc83b942e1787a0b53d555ccec291/io-uring-futex-waitv-accounted-allocation/reproducer

Thanks,
Chengfeng