RE: [EXT] Re: FYI, fsnotify contention with aio and io_uring.

[Pierre Labat <plabat@micron.com>]

Hi Jeff,

Indeed, by default, in my configuration, pipewire is running.
When I can re-test, I'll disabled it and see if that remove the problem.
Thanks for the hint!

Pierre

> -----Original Message-----
> From: Jeff Moyer <jmoyer@redhat.com>
> Sent: Wednesday, August 9, 2023 10:15 AM
> To: Pierre Labat <plabat@micron.com>
> Cc: Jens Axboe <axboe@kernel.dk>; 'io-uring@vger.kernel.org' <io-
> uring@vger.kernel.org>
> Subject: Re: [EXT] Re: FYI, fsnotify contention with aio and io_uring.
> 
> CAUTION: EXTERNAL EMAIL. Do not click links or open attachments unless you
> recognize the sender and were expecting this message.
> 
> 
> Pierre Labat <plabat@micron.com> writes:
> 
> > Micron Confidential
> >
> > Hi Jeff and Jens,
> >
> > About "FAN_MODIFY fsnotify watch set on /dev".
> >
> > Was using Fedora34 distro (with 6.3.9 kernel), and fio. Without any
> particular/specific setting.
> > I tried to see what could watch /dev but failed at that.
> > I used the inotify-info tool, but that display watchers using the
> > inotify interface. And nothing was watching /dev via inotify.
> > Need to figure out how to do the same but for the fanotify interface.
> > I'll look at it again and let you know.
> 
> You wouldn't happen to be running pipewire, would you?
> 
> https://urldefense.com/v3/__https://gitlab.freedesktop.org/pipewire/pipewir
> e/-
> /commit/88f0dbd6fcd0a412fc4bece22afdc3ba0151e4cf__;!!KZTdOCjhgt4hgw!6E063jj
> -_XK1NceWzms7DaYacILy4cKmeNVA3xalNwkd0zrYTX-IouUnvJ8bZs-RG3YSdk5XpFoo$
> 
> -Jeff
> 
> >
> > Regards,
> >
> > Pierre
> >
> >
> >
> > Micron Confidential
> >> -----Original Message-----
> >> From: Jens Axboe <axboe@kernel.dk>
> >> Sent: Tuesday, August 8, 2023 2:41 PM
> >> To: Jeff Moyer <jmoyer@redhat.com>; Pierre Labat <plabat@micron.com>
> >> Cc: 'io-uring@vger.kernel.org' <io-uring@vger.kernel.org>
> >> Subject: [EXT] Re: FYI, fsnotify contention with aio and io_uring.
> >>
> >> CAUTION: EXTERNAL EMAIL. Do not click links or open attachments
> >> unless you recognize the sender and were expecting this message.
> >>
> >>
> >> On 8/7/23 2:11?PM, Jeff Moyer wrote:
> >> > Hi, Pierre,
> >> >
> >> > Pierre Labat <plabat@micron.com> writes:
> >> >
> >> >> Hi,
> >> >>
> >> >> This is FYI, may be you already knows about that, but in case you
> >> don't....
> >> >>
> >> >> I was pushing the limit of the number of nvme read IOPS, the FIO +
> >> >> the Linux OS can handle. For that, I have something special under
> >> >> the Linux nvme driver. As a consequence I am not limited by
> >> >> whatever the NVME SSD max IOPS or IO latency would be.
> >> >>
> >> >> As I cranked the number of system cores and FIO jobs doing direct
> >> >> 4k random read on /dev/nvme0n1, I hit a wall. The IOPS scaling
> >> >> slows (less than linear) and around 15 FIO jobs on 15 core
> >> >> threads, the overall IOPS, in fact, goes down as I add more FIO
> >> >> jobs. For example on a system with 24 cores/48 threads, when I
> >> >> goes beyond 15 FIO jobs, the overall IOPS starts to go down.
> >> >>
> >> >> This happens the same for io_uring and aio. Was using kernel
> >> >> version
> >> 6.3.9. Using one namespace (/dev/nvme0n1).
> >> >
> >> > [snip]
> >> >
> >> >> As you can see 76% of the cpu on the box is sucked up by
> >> >> lockref_get_not_zero() and lockref_put_return().  Looking at the
> >> >> code, there is contention when IO_uring call fsnotify_access().
> >> >
> >> > Is there a FAN_MODIFY fsnotify watch set on /dev?  If so, it might
> >> > be a good idea to find out what set it and why.
> >>
> >> This would be my guess too, some distros do seem to do that. The
> >> notification bits scale horribly, nobody should use it for anything
> >> high performance...
> >>
> >> --
> >> Jens Axboe