Re: io_uring failure on parisc (32-bit userspace and 64-bit kernel)

[John David Anglin <dave.anglin@bell.net>]

On 2023-02-13 5:05 p.m., Helge Deller wrote:
> On 2/13/23 22:05, Jens Axboe wrote:
>> On 2/13/23 1:59?PM, Helge Deller wrote:
>>>> Yep sounds like it. What's the caching architecture of parisc?
>>>
>>> parisc is Virtually Indexed, Physically Tagged (VIPT).
>>
>> That's what I assumed, so virtual aliasing is what we're dealing with
>> here.
>>
>>> Thanks for the patch!
>>> Sadly it doesn't fix the problem, as the kernel still sees
>>> ctx->rings->sq.tail as being 0.
>>> Interestingly it worked once (not reproduceable) directly after bootup,
>>> which indicates that we at least look at the right address from kernel side.
>>>
>>> So, still needs more debugging/testing.
>>
>> It's not like this is untested stuff, so yeah it'll generally be
>> correct, it just seems that parisc is a bit odd in that the virtual
>> aliasing occurs between the kernel and userspace addresses too. At least
>> that's what it seems like.
>
> True.
>
>> But I wonder if what needs flushing is the user side, not the kernel
>> side? Either that, or my patch is not flushing the right thing on the
>> kernel side.
>>
>> Is it possible to flush it from the userspace side? Presumable that's
>> what we'd need on the sqe side, and then the kernel side for the cqe
>> filling. So probably the patch is half-way correct :-)
>
> I hacked up in __io_uring_flush_sq() in liburing/src/queue.c this code
> (which I hope is correct):
>                 if (!(ring->flags & IORING_SETUP_SQPOLL))
>                         IO_URING_WRITE_ONCE(*sq->ktail, tail);
>                 else
>                         io_uring_smp_store_release(sq->ktail, tail);
>         } /* ADDED: */
>         { int i;  unsigned long p = (unsigned long)sq->ktail & ~(4096-1);
>           fprintf(stderr, "FLUSH CACHE OF PAGE %lx\n", p);
>           for (i=0; i < 4096; i += 8)
>                 asm volatile("fdc 0(%0)" : : "r" (p+i));
>         }
>
> The kernel sometimes sees the tail value now (it fails afterwards, but that's ok for now).
> But I'm not sure yet if this is really the effect of the fdc (flush data cache instruction),
> or pure luck because the aliasing of the userspace address and kernel address matches in
> a sucessful run.
If the user and kernel aliases are not equivalent, the kernel must also flush the page to
invalidate any lines that may be present in the cache before trying to access the data in the page.
> For me it seems as it's the aliasing which makes it work sometimes.
>
> In this regard I wonder why we don't provide the cacheflush syscall on parisc....
The kernel knows the cache stride and can optimize the flush.  But it needs to handle non access TLB
faults on userspace.  Userspace can also do flushes.

Dave

-- 
John David Anglin  dave.anglin@bell.net