public inbox for [email protected]
 help / color / mirror / Atom feed
* Possible bug for ring-mapped provided buffer
@ 2022-06-09  7:53 Hao Xu
  2022-06-09  9:33 ` Hao Xu
  2022-06-09 10:06 ` Jens Axboe
  0 siblings, 2 replies; 9+ messages in thread
From: Hao Xu @ 2022-06-09  7:53 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, Pavel Begunkov

Hi all,
I haven't done tests to demonstrate it. It is for partial io case, we
don't consume/release the buffer before arm_poll in ring-mapped mode.
But seems we should? Otherwise ring head isn't moved and other requests
may take that buffer. What do I miss?

Regards,
Hao

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Possible bug for ring-mapped provided buffer
  2022-06-09  7:53 Possible bug for ring-mapped provided buffer Hao Xu
@ 2022-06-09  9:33 ` Hao Xu
  2022-06-09  9:54   ` Hao Xu
  2022-06-09 10:06 ` Jens Axboe
  1 sibling, 1 reply; 9+ messages in thread
From: Hao Xu @ 2022-06-09  9:33 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, Pavel Begunkov

On 6/9/22 15:53, Hao Xu wrote:
> Hi all,
> I haven't done tests to demonstrate it. It is for partial io case, we
> don't consume/release the buffer before arm_poll in ring-mapped mode.
> But seems we should? Otherwise ring head isn't moved and other requests
> may take that buffer. What do I miss?
> 
> Regards,
> Hao

something like this:


diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c
index d2b2b4728381..ae4c69ad0f86 100644
--- a/io_uring/kbuf.c
+++ b/io_uring/kbuf.c
@@ -48,7 +48,10 @@ void __io_kbuf_recycle(struct io_kiocb *req, unsigned 
issue_flags)
          * If the tail has already been incremented, hang on to it.
          */
         if (req->flags & REQ_F_BUFFER_RING) {
-               if (req->buf_list) {
+               if (req->flags & REQ_F_PARTIAL_IO) {
+                       req->buf_list->head++;
+                       req->buf_list = NULL;
+               } else if (req->buf_list) {
                         req->buf_index = req->buf_list->bgid;
                         req->flags &= ~REQ_F_BUFFER_RING;
                 }
diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h
index b58d9d20c97e..9ecb175e60a9 100644
--- a/io_uring/kbuf.h
+++ b/io_uring/kbuf.h
@@ -58,8 +58,14 @@ static inline void io_kbuf_recycle(struct io_kiocb 
*req, unsigned issue_flags)
  {
         if (!(req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)))
                 return;
-       /* don't recycle if we already did IO to this buffer */
-       if (req->flags & REQ_F_PARTIAL_IO)
+       /*
+        * For legacy provided buffer mode, don't recycle if we already did
+        * IO to this buffer. For ring-mapped provided buffer mode, we 
should
+        * increment ring->head to explicitly monopolize the buffer to avoid
+        * multiple use.
+        */
+       if ((req->flags & REQ_F_BUFFER_SELECTED) &&
+           (req->flags & REQ_F_PARTIAL_IO))
                 return;
         __io_kbuf_recycle(req, issue_flags);
  }

^ permalink raw reply related	[flat|nested] 9+ messages in thread

* Re: Possible bug for ring-mapped provided buffer
  2022-06-09  9:33 ` Hao Xu
@ 2022-06-09  9:54   ` Hao Xu
  0 siblings, 0 replies; 9+ messages in thread
From: Hao Xu @ 2022-06-09  9:54 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, Pavel Begunkov

On 6/9/22 17:33, Hao Xu wrote:
> On 6/9/22 15:53, Hao Xu wrote:
>> Hi all,
>> I haven't done tests to demonstrate it. It is for partial io case, we
>> don't consume/release the buffer before arm_poll in ring-mapped mode.
>> But seems we should? Otherwise ring head isn't moved and other requests
>> may take that buffer. What do I miss?
>>
>> Regards,
>> Hao
> 
> something like this:
> 

forgot something in previous diff

diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c
index d2b2b4728381..9ff8d14277ff 100644
--- a/io_uring/kbuf.c
+++ b/io_uring/kbuf.c
@@ -49,8 +49,15 @@ void __io_kbuf_recycle(struct io_kiocb *req, unsigned 
issue_flags)
          */
         if (req->flags & REQ_F_BUFFER_RING) {
                 if (req->buf_list) {
-                       req->buf_index = req->buf_list->bgid;
-                       req->flags &= ~REQ_F_BUFFER_RING;
+                       if (req->flags & REQ_F_PARTIAL_IO) {
+                               io_ring_submit_lock(ctx, issue_flags);
+                               req->buf_list->head++;
+                               io_ring_submit_unlock(ctx, issue_flags);
+                               req->buf_list = NULL;
+                       } else {
+                               req->buf_index = req->buf_list->bgid;
+                               req->flags &= ~REQ_F_BUFFER_RING;
+                       }
                 }
                 return;
         }
diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h
index b58d9d20c97e..9ecb175e60a9 100644
--- a/io_uring/kbuf.h
+++ b/io_uring/kbuf.h
@@ -58,8 +58,14 @@ static inline void io_kbuf_recycle(struct io_kiocb 
*req, unsigned issue_flags)
  {
         if (!(req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)))
                 return;
-       /* don't recycle if we already did IO to this buffer */
-       if (req->flags & REQ_F_PARTIAL_IO)
+       /*
+        * For legacy provided buffer mode, don't recycle if we already did
+        * IO to this buffer. For ring-mapped provided buffer mode, we 
should
+        * increment ring->head to explicitly monopolize the buffer to avoid
+        * multiple use.
+        */
+       if ((req->flags & REQ_F_BUFFER_SELECTED) &&
+           (req->flags & REQ_F_PARTIAL_IO))
                 return;
         __io_kbuf_recycle(req, issue_flags);
  }



^ permalink raw reply related	[flat|nested] 9+ messages in thread

* Re: Possible bug for ring-mapped provided buffer
  2022-06-09  7:53 Possible bug for ring-mapped provided buffer Hao Xu
  2022-06-09  9:33 ` Hao Xu
@ 2022-06-09 10:06 ` Jens Axboe
  2022-06-09 10:14   ` Hao Xu
  1 sibling, 1 reply; 9+ messages in thread
From: Jens Axboe @ 2022-06-09 10:06 UTC (permalink / raw)
  To: Hao Xu, io-uring; +Cc: Pavel Begunkov

On 6/9/22 1:53 AM, Hao Xu wrote:
> Hi all,
> I haven't done tests to demonstrate it. It is for partial io case, we
> don't consume/release the buffer before arm_poll in ring-mapped mode.
> But seems we should? Otherwise ring head isn't moved and other requests
> may take that buffer. What do I miss?

On vacation this week, so can't take a look at the code. But the
principle is precisely not to consume the buffer if we arm poll, because
then the next one can grab it instead. We don't want to consume a buffer
over poll, as that defeats the purpose of a provided buffer. It should
be grabbed and consumed only if we can use it right now.

Hence the way it should work is that we DON'T consume the buffer in this
case, and that someone else can just use it. At the same time, we should
ensure that we grab a NEW buffer for this case, whenever the poll
triggers and we can retry the IO. As mentioned I can't check the code
right now, but perhaps you can take a look.

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Possible bug for ring-mapped provided buffer
  2022-06-09 10:06 ` Jens Axboe
@ 2022-06-09 10:14   ` Hao Xu
  2022-06-09 10:19     ` Jens Axboe
  0 siblings, 1 reply; 9+ messages in thread
From: Hao Xu @ 2022-06-09 10:14 UTC (permalink / raw)
  To: Jens Axboe, io-uring; +Cc: Pavel Begunkov

On 6/9/22 18:06, Jens Axboe wrote:
> On 6/9/22 1:53 AM, Hao Xu wrote:
>> Hi all,
>> I haven't done tests to demonstrate it. It is for partial io case, we
>> don't consume/release the buffer before arm_poll in ring-mapped mode.
>> But seems we should? Otherwise ring head isn't moved and other requests
>> may take that buffer. What do I miss?
> 
> On vacation this week, so can't take a look at the code. But the
> principle is precisely not to consume the buffer if we arm poll, because
> then the next one can grab it instead. We don't want to consume a buffer
> over poll, as that defeats the purpose of a provided buffer. It should
> be grabbed and consumed only if we can use it right now.
> 
> Hence the way it should work is that we DON'T consume the buffer in this
> case, and that someone else can just use it. At the same time, we should
> ensure that we grab a NEW buffer for this case, whenever the poll

If we grab a new buffer for it, then we have to copy the data since we
have done partial io...this also defeats the purpose of this feature.
What the legacy provided buffer mode do in this case is just
keep/consume that buffer. So I'd think we should keep the consistency.
But yes, there may be a better way.

> triggers and we can retry the IO. As mentioned I can't check the code
> right now, but perhaps you can take a look.
> 


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Possible bug for ring-mapped provided buffer
  2022-06-09 10:14   ` Hao Xu
@ 2022-06-09 10:19     ` Jens Axboe
  2022-06-09 10:32       ` Hao Xu
  0 siblings, 1 reply; 9+ messages in thread
From: Jens Axboe @ 2022-06-09 10:19 UTC (permalink / raw)
  To: Hao Xu, io-uring; +Cc: Pavel Begunkov

On 6/9/22 4:14 AM, Hao Xu wrote:
> On 6/9/22 18:06, Jens Axboe wrote:
>> On 6/9/22 1:53 AM, Hao Xu wrote:
>>> Hi all,
>>> I haven't done tests to demonstrate it. It is for partial io case, we
>>> don't consume/release the buffer before arm_poll in ring-mapped mode.
>>> But seems we should? Otherwise ring head isn't moved and other requests
>>> may take that buffer. What do I miss?
>>
>> On vacation this week, so can't take a look at the code. But the
>> principle is precisely not to consume the buffer if we arm poll, because
>> then the next one can grab it instead. We don't want to consume a buffer
>> over poll, as that defeats the purpose of a provided buffer. It should
>> be grabbed and consumed only if we can use it right now.
>>
>> Hence the way it should work is that we DON'T consume the buffer in this
>> case, and that someone else can just use it. At the same time, we should
>> ensure that we grab a NEW buffer for this case, whenever the poll
> 
> If we grab a new buffer for it, then we have to copy the data since we
> have done partial io...this also defeats the purpose of this feature.

For partial IO, we never drop the buffer. See the logic in
io_kbuf_recycle(). It should be as follows:

- If PARTIAL_IO is set, then hang on to the buffer. You can't consume a
  partial buffer anyway.

- If no IO has been done and it's a ring provided buffer, just hang on
  to the bgid and clear the fact that we grabbed a buffer. That's all
  you need to do in this case, someone else may grab it and we'll grab a
  new one for this request whenever it's time to do so.

> What the legacy provided buffer mode do in this case is just
> keep/consume that buffer. So I'd think we should keep the consistency.
> But yes, there may be a better way.

The legacy mode has to do that, as it always has to grab the buffer. We
don't need to do that in the ring case, it's an efficiency thing as
well. If we do need to arm poll, we don't have to do anything but ensure
that we grab a new one next time. Consuming it would be the wrong thing
to do in that case, as it defeats the purpose of a provided buffer. You
may as well just pass in a buffer at that point.

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Possible bug for ring-mapped provided buffer
  2022-06-09 10:19     ` Jens Axboe
@ 2022-06-09 10:32       ` Hao Xu
  2022-06-09 15:06         ` Jens Axboe
  0 siblings, 1 reply; 9+ messages in thread
From: Hao Xu @ 2022-06-09 10:32 UTC (permalink / raw)
  To: Jens Axboe, io-uring; +Cc: Pavel Begunkov

On 6/9/22 18:19, Jens Axboe wrote:
> On 6/9/22 4:14 AM, Hao Xu wrote:
>> On 6/9/22 18:06, Jens Axboe wrote:
>>> On 6/9/22 1:53 AM, Hao Xu wrote:
>>>> Hi all,
>>>> I haven't done tests to demonstrate it. It is for partial io case, we
>>>> don't consume/release the buffer before arm_poll in ring-mapped mode.
>>>> But seems we should? Otherwise ring head isn't moved and other requests
>>>> may take that buffer. What do I miss?
>>>
>>> On vacation this week, so can't take a look at the code. But the
>>> principle is precisely not to consume the buffer if we arm poll, because
>>> then the next one can grab it instead. We don't want to consume a buffer
>>> over poll, as that defeats the purpose of a provided buffer. It should
>>> be grabbed and consumed only if we can use it right now.
>>>
>>> Hence the way it should work is that we DON'T consume the buffer in this
>>> case, and that someone else can just use it. At the same time, we should
>>> ensure that we grab a NEW buffer for this case, whenever the poll
>>
>> If we grab a new buffer for it, then we have to copy the data since we
>> have done partial io...this also defeats the purpose of this feature.
> 
> For partial IO, we never drop the buffer. See the logic in
> io_kbuf_recycle(). It should be as follows:

Yea, in io_kbuf_recycle(), if it's partial io, we just return. For
legacy mode, this means we keep the buffer. For ring-mapped mode, this
means we then release the uring_lock without moving the ring->head,
and then other requests may take that buffer which is in use..
And next time we do (for example) recv(), we lost the data which we got
at the previous time.
Do I miss something?

> 
> - If PARTIAL_IO is set, then hang on to the buffer. You can't consume a
>    partial buffer anyway.
> 
> - If no IO has been done and it's a ring provided buffer, just hang on
>    to the bgid and clear the fact that we grabbed a buffer. That's all
>    you need to do in this case, someone else may grab it and we'll grab a
>    new one for this request whenever it's time to do so.
> 
>> What the legacy provided buffer mode do in this case is just
>> keep/consume that buffer. So I'd think we should keep the consistency.
>> But yes, there may be a better way.
> 
> The legacy mode has to do that, as it always has to grab the buffer. We
> don't need to do that in the ring case, it's an efficiency thing as
> well. If we do need to arm poll, we don't have to do anything but ensure
> that we grab a new one next time. Consuming it would be the wrong thing
> to do in that case, as it defeats the purpose of a provided buffer. You
> may as well just pass in a buffer at that point.
> 


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Possible bug for ring-mapped provided buffer
  2022-06-09 10:32       ` Hao Xu
@ 2022-06-09 15:06         ` Jens Axboe
  2022-06-09 16:08           ` Hao Xu
  0 siblings, 1 reply; 9+ messages in thread
From: Jens Axboe @ 2022-06-09 15:06 UTC (permalink / raw)
  To: Hao Xu, io-uring; +Cc: Pavel Begunkov

On 6/9/22 4:32 AM, Hao Xu wrote:
> On 6/9/22 18:19, Jens Axboe wrote:
>> On 6/9/22 4:14 AM, Hao Xu wrote:
>>> On 6/9/22 18:06, Jens Axboe wrote:
>>>> On 6/9/22 1:53 AM, Hao Xu wrote:
>>>>> Hi all,
>>>>> I haven't done tests to demonstrate it. It is for partial io case, we
>>>>> don't consume/release the buffer before arm_poll in ring-mapped mode.
>>>>> But seems we should? Otherwise ring head isn't moved and other requests
>>>>> may take that buffer. What do I miss?
>>>>
>>>> On vacation this week, so can't take a look at the code. But the
>>>> principle is precisely not to consume the buffer if we arm poll, because
>>>> then the next one can grab it instead. We don't want to consume a buffer
>>>> over poll, as that defeats the purpose of a provided buffer. It should
>>>> be grabbed and consumed only if we can use it right now.
>>>>
>>>> Hence the way it should work is that we DON'T consume the buffer in this
>>>> case, and that someone else can just use it. At the same time, we should
>>>> ensure that we grab a NEW buffer for this case, whenever the poll
>>>
>>> If we grab a new buffer for it, then we have to copy the data since we
>>> have done partial io...this also defeats the purpose of this feature.
>>
>> For partial IO, we never drop the buffer. See the logic in
>> io_kbuf_recycle(). It should be as follows:
> 
> Yea, in io_kbuf_recycle(), if it's partial io, we just return. For
> legacy mode, this means we keep the buffer. For ring-mapped mode, this
> means we then release the uring_lock without moving the ring->head,
> and then other requests may take that buffer which is in use..
> And next time we do (for example) recv(), we lost the data which we got
> at the previous time.
> Do I miss something?

If we don't commit for ring mapped buffers, then yeah that's definitely
a bug. Please send a fix :-)

Pavel can take care of it this week.

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Possible bug for ring-mapped provided buffer
  2022-06-09 15:06         ` Jens Axboe
@ 2022-06-09 16:08           ` Hao Xu
  0 siblings, 0 replies; 9+ messages in thread
From: Hao Xu @ 2022-06-09 16:08 UTC (permalink / raw)
  To: Jens Axboe, io-uring; +Cc: Pavel Begunkov

On 6/9/22 23:06, Jens Axboe wrote:
> On 6/9/22 4:32 AM, Hao Xu wrote:
>> On 6/9/22 18:19, Jens Axboe wrote:
>>> On 6/9/22 4:14 AM, Hao Xu wrote:
>>>> On 6/9/22 18:06, Jens Axboe wrote:
>>>>> On 6/9/22 1:53 AM, Hao Xu wrote:
>>>>>> Hi all,
>>>>>> I haven't done tests to demonstrate it. It is for partial io case, we
>>>>>> don't consume/release the buffer before arm_poll in ring-mapped mode.
>>>>>> But seems we should? Otherwise ring head isn't moved and other requests
>>>>>> may take that buffer. What do I miss?
>>>>>
>>>>> On vacation this week, so can't take a look at the code. But the
>>>>> principle is precisely not to consume the buffer if we arm poll, because
>>>>> then the next one can grab it instead. We don't want to consume a buffer
>>>>> over poll, as that defeats the purpose of a provided buffer. It should
>>>>> be grabbed and consumed only if we can use it right now.
>>>>>
>>>>> Hence the way it should work is that we DON'T consume the buffer in this
>>>>> case, and that someone else can just use it. At the same time, we should
>>>>> ensure that we grab a NEW buffer for this case, whenever the poll
>>>>
>>>> If we grab a new buffer for it, then we have to copy the data since we
>>>> have done partial io...this also defeats the purpose of this feature.
>>>
>>> For partial IO, we never drop the buffer. See the logic in
>>> io_kbuf_recycle(). It should be as follows:
>>
>> Yea, in io_kbuf_recycle(), if it's partial io, we just return. For
>> legacy mode, this means we keep the buffer. For ring-mapped mode, this
>> means we then release the uring_lock without moving the ring->head,
>> and then other requests may take that buffer which is in use..
>> And next time we do (for example) recv(), we lost the data which we got
>> at the previous time.
>> Do I miss something?
> 
> If we don't commit for ring mapped buffers, then yeah that's definitely
> a bug. Please send a fix :-)
> 
> Pavel can take care of it this week.
> 

I'll send a patch tomorrow.

Thanks,
Hao

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2022-06-09 16:08 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-06-09  7:53 Possible bug for ring-mapped provided buffer Hao Xu
2022-06-09  9:33 ` Hao Xu
2022-06-09  9:54   ` Hao Xu
2022-06-09 10:06 ` Jens Axboe
2022-06-09 10:14   ` Hao Xu
2022-06-09 10:19     ` Jens Axboe
2022-06-09 10:32       ` Hao Xu
2022-06-09 15:06         ` Jens Axboe
2022-06-09 16:08           ` Hao Xu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox