Fwd: [axboe/liburing] Improper memory ordering when reading wakeup flag with SQPOLL (Issue #541)

Fwd: [axboe/liburing] Improper memory ordering when reading wakeup flag with SQPOLL (Issue #541)

[Ammar Faizi]

This is just a test Forward Message, ignore...
-------- Forwarded Message --------
Subject: 	[axboe/liburing] Improper memory ordering when reading wakeup flag with SQPOLL (Issue #541)
Date: 	Sat, 26 Feb 2022 06:56:19 -0800
From: 	Almog Khaikin <notifications@github.com>
To: 	axboe/liburing <liburing@noreply.github.com>
CC: 	Subscribed <subscribed@noreply.github.com>

This is a followup to #219 <https://github.com/axboe/liburing/issues/219>.
In that discussion it was decided that a relaxed load is sufficient even though the documentation states that an acquire load is needed but it's actually the opposite. Between updating the sq tail and reading the flags a full memory barrier (|smp_mb|) is required. This is even documented in the kernel source code <https://github.com/torvalds/linux/blob/master/fs/io_uring.c#L25-L28>.

This is a case of a read-after-write which neither acquire nor release semantics deal with. Even on x86 which has a strong memory model a read-after-write is the only situation where memory operations can be reordered due to the store buffer mechanism that the processor uses.

This same bug is also present in the kernel code <https://github.com/torvalds/linux/blob/master/fs/io_uring.c#L7609-L7616>. The kernel sets the NEED_WAKEUP flag and then checks if there are sqes one last time before going to sleep. This is again a read-after-write which means a full memory barrier is required but it's missing in the kernel code.

With the current implementation the following sequence of events is possible:

|Kernel sleep timer expires Kernel sets NEED_WAKEUP flag Application submits new work to the sq (updates the sq tail) Kernel checks sq for new entries and doesn't see the update to the sq tail because it's in the store buffer Application checks NEED_WAKEUP flag and doesn't see it because the store to the flag is in the store buffer Kernel thread goes to sleep Application moves on without invoking io_uring_enter |

The result is there is now an IO operation sitting in the sq and not being processed while the application thinks the operation was submitted. I used x86 specific terminology and referenced the store buffer just to simplify the explanation but this applies to all architectures. In the C11 memory model a WriteRead can be reordered if there is no full barrier between the two operations.