io_uring_setup spuriously returning ENOMEM for one user

io_uring_setup spuriously returning ENOMEM for one user

[Andres Freund]

Hi,

While testing the error handling of my uring using postgres branch I
just encountered the situation that io_uring_setup() always fails with
ENOMEN.

It only does so for the user I did the testing on and not for other
users. During the testing a few io_uring using processes were kill -9'd
and a few core-dumped after abort(). No io_uring using processes are
still alive.

As the issue only happens to the one uid I suspect that
current_user()->locked_mem got corrupted, perhaps after hitting the
limit for real.

Unfortunately I do not see any way to debug this without restarting. It
seems the user wide limit isn't exported anywhere? I found
uids_sysfs_init() while grepping around, but it seems that's just a
leftover.

This happened on 07a56bb875afbe39dabbf6ba7b83783d166863db / 5.8rc5 +
16. I left the machine running for now, in case there's something that
can be debugged while running. But I've to restart it in a bit. It took
a while to hit this issue, unfortunately.

Greetings,

Andres Freund

Re: io_uring_setup spuriously returning ENOMEM for one user

[Pavel Begunkov]

On 16/07/2020 23:05, Andres Freund wrote:
> Hi,
> 
> While testing the error handling of my uring using postgres branch I
> just encountered the situation that io_uring_setup() always fails with
> ENOMEN.
> 
> It only does so for the user I did the testing on and not for other
> users. During the testing a few io_uring using processes were kill -9'd
> and a few core-dumped after abort(). No io_uring using processes are
> still alive.
> 
> As the issue only happens to the one uid I suspect that
> current_user()->locked_mem got corrupted, perhaps after hitting the
> limit for real.

Any chance it's using SQPOLL mode?

> 
> Unfortunately I do not see any way to debug this without restarting. It
> seems the user wide limit isn't exported anywhere? I found
> uids_sysfs_init() while grepping around, but it seems that's just a
> leftover.
> 
> This happened on 07a56bb875afbe39dabbf6ba7b83783d166863db / 5.8rc5 +
> 16. I left the machine running for now, in case there's something that
> can be debugged while running. But I've to restart it in a bit. It took
> a while to hit this issue, unfortunately.
> 

-- 
Pavel Begunkov

Re: io_uring_setup spuriously returning ENOMEM for one user

[Andres Freund]

Hi,

On 2020-07-16 23:12:41 +0300, Pavel Begunkov wrote:
> On 16/07/2020 23:05, Andres Freund wrote:
> > Hi,
> > 
> > While testing the error handling of my uring using postgres branch I
> > just encountered the situation that io_uring_setup() always fails with
> > ENOMEN.
> > 
> > It only does so for the user I did the testing on and not for other
> > users. During the testing a few io_uring using processes were kill -9'd
> > and a few core-dumped after abort(). No io_uring using processes are
> > still alive.
> > 
> > As the issue only happens to the one uid I suspect that
> > current_user()->locked_mem got corrupted, perhaps after hitting the
> > limit for real.
> 
> Any chance it's using SQPOLL mode?

No. It's a "plain" uring. The only thing that could be considered
special is that one of the rings is shared between processes (which all
run as the same user).

Greetings,

Andres Freund

Re: io_uring_setup spuriously returning ENOMEM for one user

[Jens Axboe]

On 7/16/20 2:20 PM, Andres Freund wrote:
> Hi,
> 
> On 2020-07-16 23:12:41 +0300, Pavel Begunkov wrote:
>> On 16/07/2020 23:05, Andres Freund wrote:
>>> Hi,
>>>
>>> While testing the error handling of my uring using postgres branch I
>>> just encountered the situation that io_uring_setup() always fails with
>>> ENOMEN.
>>>
>>> It only does so for the user I did the testing on and not for other
>>> users. During the testing a few io_uring using processes were kill -9'd
>>> and a few core-dumped after abort(). No io_uring using processes are
>>> still alive.
>>>
>>> As the issue only happens to the one uid I suspect that
>>> current_user()->locked_mem got corrupted, perhaps after hitting the
>>> limit for real.
>>
>> Any chance it's using SQPOLL mode?
> 
> No. It's a "plain" uring. The only thing that could be considered
> special is that one of the rings is shared between processes (which all
> run as the same user).

Do you have this one:

ommit 309fc03a3284af62eb6082fb60327045a1dabf57 (tag: io_uring-5.8-2020-07-10)
Author: Jens Axboe <axboe@kernel.dk>
Date:   Fri Jul 10 09:13:34 2020 -0600

    io_uring: account user memory freed when exit has been queued

-- 
Jens Axboe