public inbox for [email protected]
 help / color / mirror / Atom feed
* "Cannot allocate memory" on ring creation (not RLIMIT_MEMLOCK)
@ 2020-12-17  8:19 Dmitry Kadashev
  2020-12-17  8:26 ` Norman Maurer
  2020-12-18 15:26 ` Jens Axboe
  0 siblings, 2 replies; 52+ messages in thread
From: Dmitry Kadashev @ 2020-12-17  8:19 UTC (permalink / raw)
  To: io-uring, Jens Axboe

Hi,

We've ran into something that looks like a memory accounting problem in the
kernel / io_uring code. We use multiple rings per process, and generally it
works fine. Until it does not - new ring creation just fails with ENOMEM. And at
that point it fails consistently until the box is rebooted.

More details: we use multiple rings per process, typically they are initialized
on the process start (not necessarily, but that is not important here, let's
just assume all are initialized on the process start). On a freshly booted box
everything works fine. But after a while - and some process restarts -
io_uring_queue_init() starts to fail with ENOMEM. Sometimes we see it fail, but
then subsequent ones succeed (in the same process), but over time it gets worse,
and eventually no ring can be initialized. And once that happens the only way to
fix the problem is to restart the box.  Most of the mentioned restarts are
graceful: a new process is started and then the old one is killed, possibly with
the KILL signal if it does not shut down in time.  Things work fine for some
time, but eventually we start getting those errors.

Originally we've used 5.6.6 kernel, but given the fact quite a few accounting
issues were fixed in io_uring in 5.8, we've tried 5.9.5 as well, but the issue
is not gone.

Just in case, everything else seems to be working fine, it just falls back to
the thread pool instead of io_uring, and then everything continues to work just
fine.

I was not able to spot anything suspicious in the /proc/meminfo. We have
RLIMIT_MEMLOCK set to infinity. And on a box that currently experiences the
problem /proc/meminfo shows just 24MB as locked.

Any pointers to how can we debug this?

Thanks,
Dmitry

^ permalink raw reply	[flat|nested] 52+ messages in thread

end of thread, other threads:[~2020-12-23 12:31 UTC | newest]

Thread overview: 52+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-12-17  8:19 "Cannot allocate memory" on ring creation (not RLIMIT_MEMLOCK) Dmitry Kadashev
2020-12-17  8:26 ` Norman Maurer
2020-12-17  8:36   ` Dmitry Kadashev
2020-12-17  8:40     ` Dmitry Kadashev
2020-12-17 10:38       ` Josef
2020-12-17 11:10         ` Dmitry Kadashev
2020-12-17 13:43           ` Victor Stewart
2020-12-18  9:20             ` Dmitry Kadashev
2020-12-18 17:22               ` Jens Axboe
2020-12-18 15:26 ` Jens Axboe
2020-12-18 17:21   ` Josef
2020-12-18 17:23     ` Jens Axboe
2020-12-19  2:49       ` Josef
2020-12-19 16:13         ` Jens Axboe
2020-12-19 16:29           ` Jens Axboe
2020-12-19 17:11             ` Jens Axboe
2020-12-19 17:34               ` Norman Maurer
2020-12-19 17:38                 ` Jens Axboe
2020-12-19 20:51                   ` Josef
2020-12-19 21:54                     ` Jens Axboe
2020-12-19 23:13                       ` Jens Axboe
2020-12-19 23:42                         ` Josef
2020-12-19 23:42                         ` Pavel Begunkov
2020-12-20  0:25                           ` Jens Axboe
2020-12-20  0:55                             ` Pavel Begunkov
2020-12-21 10:35                               ` Dmitry Kadashev
2020-12-21 10:49                                 ` Dmitry Kadashev
2020-12-21 11:00                                 ` Dmitry Kadashev
2020-12-21 15:36                                   ` Pavel Begunkov
2020-12-22  3:35                                   ` Pavel Begunkov
2020-12-22  4:07                                     ` Pavel Begunkov
2020-12-22 11:04                                       ` Dmitry Kadashev
2020-12-22 11:06                                         ` Dmitry Kadashev
2020-12-22 13:13                                           ` Dmitry Kadashev
2020-12-22 16:33                                         ` Pavel Begunkov
2020-12-23  8:39                                           ` Dmitry Kadashev
2020-12-23  9:38                                             ` Dmitry Kadashev
2020-12-23 11:48                                               ` Dmitry Kadashev
2020-12-23 12:27                                                 ` Pavel Begunkov
2020-12-20  1:57                             ` Pavel Begunkov
2020-12-20  7:13                               ` Josef
2020-12-20 13:00                                 ` Pavel Begunkov
2020-12-20 14:19                                   ` Pavel Begunkov
2020-12-20 15:56                                     ` Josef
2020-12-20 15:58                                       ` Pavel Begunkov
2020-12-20 16:14                                   ` Jens Axboe
2020-12-20 16:59                                     ` Josef
2020-12-20 18:23                                       ` Josef
2020-12-20 18:41                                         ` Pavel Begunkov
2020-12-21  8:22                                           ` Josef
2020-12-21 15:30                                             ` Pavel Begunkov
2020-12-21 10:31               ` Dmitry Kadashev

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox