Re: memory access op ideas

[Avi Kivity <avi@scylladb.com>]

On 22/04/2022 17.50, Jens Axboe wrote:
> On 4/13/22 4:33 AM, Avi Kivity wrote:
>> Unfortunately, only ideas, no patches. But at least the first seems very easy.
>>
>>
>> - IORING_OP_MEMCPY_IMMEDIATE - copy some payload included in the op
>> itself (1-8 bytes) to a user memory location specified by the op.
>>
>>
>> Linked to another op, this can generate an in-memory notification
>> useful for busy-waiters or the UMWAIT instruction
>>
>> This would be useful for Seastar, which looks at a timer-managed
>> memory location to check when to break computation loops.
> This one would indeed be trivial to do. If we limit the max size
> supported to eg 8 bytes like suggested, then it could be in the sqe
> itself and just copied to the user address specified.
>
> Eg have sqe->len be the length (1..8 bytes), sqe->addr the destination
> address, and sqe->off the data to copy.
>
> If you'll commit to testing this, I can hack it up pretty quickly...


I can certainly commit to test it in a VM (my workstation has a 
hate-hate relationship with custom kernels).


>> - IORING_OP_MEMCPY - asynchronously copy memory
>>
>>
>> Some CPUs include a DMA engine, and io_uring is a perfect interface to
>> exercise it. It may be difficult to find space for two iovecs though.
> I've considered this one in the past too, and it is indeed an ideal fit
> in terms of API. Outside of the DMA engines, it can also be used to
> offload memcpy to a GPU, for example.
>
> The io_uring side would not be hard to wire up, basically just have the
> sqe specfy source, destination, length. Add some well defined flags
> depending on what the copy engine offers, for example.
>
> But probably some work required here in exposing an API and testing
> etc...


Perhaps the interface should be kept separate from io_uring. e.g. use a 
pidfd to represent the address space, and then issue 
IORING_OP_PREADV/IORING_OP_PWRITEV to initiate dma. Then one can copy 
across process boundaries.


A different angle is to use expose the dma device as a separate fd. This 
can be useful as dma engine can often do other operations, like xor or 
crc or encryption or compression. In any case I'd argue for the 
interface to be useful outside io_uring, although that considerably 
increases the scope. I also don't have a direct use case for it, though 
I'm sure others will.


The kernel itself should find the DMA engine useful for things like 
memory compaction.