Re: [PATCH RFC] io_uring/rsrc: add last-lookup cache hit to io_rsrc_node_lookup()

[Jann Horn <jannh@google.com>]

On Wed, Oct 30, 2024 at 9:25 PM Jens Axboe <axboe@kernel.dk> wrote:
> On 10/30/24 11:20 AM, Jann Horn wrote:
> > On Wed, Oct 30, 2024 at 5:58?PM Jens Axboe <axboe@kernel.dk> wrote:
> >> This avoids array_index_nospec() for repeated lookups on the same node,
> >> which can be quite common (and costly). If a cached node is removed from
> >
> > You're saying array_index_nospec() can be quite costly - which
> > architecture is this on? Is this the cost of the compare+subtract+and
> > making the critical path longer?
>
> Tested this on arm64, in a vm to be specific. Let me try and generate
> some numbers/profiles on x86-64 as well. It's noticeable there as well,
> though not quite as bad as the below example. For arm64, with the patch,
> we get roughly 8.7% of the time spent getting a resource - without it's
> 66% of the time. This is just doing a microbenchmark, but it clearly
> shows that anything following the barrier on arm64 is very costly:
>
>   0.98 │       ldr   x21, [x0, #96]
>        │     ↓ tbnz  w2, #1, b8
>   1.04 │       ldr   w1, [x21, #144]
>        │       cmp   w1, w19
>        │     ↓ b.ls  a0
>        │ 30:   mov   w1, w1
>        │       sxtw  x0, w19
>        │       cmp   x0, x1
>        │       ngc   x0, xzr
>        │       csdb
>        │       ldr   x1, [x21, #160]
>        │       and   w19, w19, w0
>  93.98 │       ldr   x19, [x1, w19, sxtw #3]
>
> and accounts for most of that 66% of the total cost of the micro bench,
> even though it's doing a ton more stuff than simple getting this node
> via a lookup.

Ah, actually... a difference between x86 and arm64 is that arm64 does
an extra Speculative Data Barrier here, while x86 just does some
arithmetic. Which I think is to work around "data value predictions",
in which case my idea of using bitwise AND probably isn't valid.

https://developer.arm.com/documentation/102816/0205/ section "Software
Mitigations" says "Such code sequences are based around specific data
processing operations (for example conditional select or conditional
move) and a new barrier instruction (CSDB). The combination of both a
conditional select/conditional move and the new barrier are sufficient
to address this problem on ALL Arm implementations, both current and
future".