[RFC] support memory recycle for ring-mapped provided buffer

[RFC] support memory recycle for ring-mapped provided buffer

[Hao Xu]

Hi all,

I've actually done most code of this, but I think it's necessary to
first ask community for comments on the design. what I do is when
consuming a buffer, don't increment the head, but check the length
in real use. Then update the buffer info like
buff->addr += len, buff->len -= len;
(off course if a req consumes the whole buffer, just increment head)
and since we now changed the addr of buffer, a simple buffer id is
useless for userspace to get the data. We have to deliver the original
addr back to userspace through cqe->extra1, which means this feature
needs CQE32 to be on.
This way a provided buffer may be splited to many pieces, and userspace
should track each piece, when all the pieces are spare again, they can
re-provide the buffer.(they can surely re-provide each piece separately
but that causes more and more memory fragments, anyway, it's users'
choice.)

How do you think of this? Actually I'm not a fun of big cqe, it's not
perfect to have the limitation of having CQE32 on, but seems no other
option?

Thanks,
Hao

Re: [RFC] support memory recycle for ring-mapped provided buffer

[Hao Xu]

On 6/10/22 13:55, Hao Xu wrote:
> Hi all,
> 
> I've actually done most code of this, but I think it's necessary to
> first ask community for comments on the design. what I do is when
> consuming a buffer, don't increment the head, but check the length
> in real use. Then update the buffer info like
> buff->addr += len, buff->len -= len;
> (off course if a req consumes the whole buffer, just increment head)
> and since we now changed the addr of buffer, a simple buffer id is
> useless for userspace to get the data. We have to deliver the original
> addr back to userspace through cqe->extra1, which means this feature
> needs CQE32 to be on.
> This way a provided buffer may be splited to many pieces, and userspace
> should track each piece, when all the pieces are spare again, they can
> re-provide the buffer.(they can surely re-provide each piece separately
> but that causes more and more memory fragments, anyway, it's users'
> choice.)
> 
> How do you think of this? Actually I'm not a fun of big cqe, it's not
> perfect to have the limitation of having CQE32 on, but seems no other
> option?
> 
> Thanks,
> Hao

To implement this, CQE32 have to be introduced to almost everywhere.
For example for io_issue_sqe:

def->issue();
if (unlikely(CQE32))
     __io_req_complete32();
else
     __io_req_complete();

which will cerntainly have some overhead for main path. Any comments?

Regards,
Hao