Re: Fwd: btrfs / io-uring corrupting reads

[Filipe Manana <fdmanana@kernel.org>]

On Wed, Mar 02, 2022 at 08:51:53AM +1100, Daniel Black wrote:
> Filipe,
> 
> DId you find anything? This is starting to be noticed by our mutual users.

Yes, I've sent a patch to address it:

https://lore.kernel.org/linux-btrfs/39c96b5608ed99b7d666d4d159f8d135e86b9606.1646219178.git.fdmanana@suse.com/T/#u

The details are in the changelog.

Basically, MariaDB is getting a short read - it asks to read 16K of data
but it only gets the first 4K of data (or the first 8K or 12K). It treats
such partial read as a corruption, but in fact it could check that it's a
short read and then try to read the remaining data.

I understand it's an unexpected result, since it knows its read requests
don't cross the EOF boundary and knows exactly the length and file offsets
of its pages/blocks, so it expects a read requesting 16K of data to return
exactly 16K of data, and no less than that. I've worked in the database
industry before, and the same assumptions existed on the engines I worked on.
Either way it was a behaviour change in btrfs and it can, and should be,
addressed in btrfs.

And yes, before your report, there was at least another on the btrfs mailing
list from someone getting MariaDB corruptions on btrfs only. However yours had
a useful reproducer that made it easier to dig into.

Thanks.

> 
> https://jira.mariadb.org/browse/MDEV-27900
> https://mariadb.zulipchat.com/#narrow/stream/118759-general/topic/Corrupt.20database.20page.20when.20updating.20from.2010.2E5
> 
> On Tue, Feb 22, 2022 at 11:55 PM Filipe Manana <fdmanana@kernel.org> wrote:
> >
> > On Tue, Feb 22, 2022 at 12:46 PM Filipe Manana <fdmanana@kernel.org> wrote:
> > >
> > > On Tue, Feb 22, 2022 at 12:22 PM Daniel Black <daniel@mariadb.org> wrote:
> > > >
> > > > On Tue, Feb 22, 2022 at 10:42 PM Filipe Manana <fdmanana@kernel.org> wrote:
> > > >
> > > > > I gave it a try, but it fails setting up io_uring:
> > > > >
> > > > > 2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
> > > > > 2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1
> > > > > 2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
> > > > > 2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
> > > > > 2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool
> > > > >
> > > > > So that's why it doesn't fail here, as it fallbacks to no aio mode.
> > > >
> > > > error 1 is EPERM. Seems it needs --privileged on the container startup
> > > > as a podman argument (before the image name). Sorry I missed that
> > > >
> > > > > Any idea why it's failing to setup io_uring?
> > > > >
> > > > > I have the liburing2 and liburing-dev packages installed on debian, and
> > > > > tried with a 5.17-rc4 kernel.
> > > >
> > > > Taking https://packages.debian.org/bookworm/mariadb-server-core-10.6 package:
> > > >
> > > > mariadb-install-db --no-defaults --datadir=/empty/btrfs/path
> > > > --innodb-use-native-aio=0
> > > >
> > > > mariadbd --no-defaults --datadir=/empty/btrfs/path --innodb-use-native-aio=1
> > > >
> > > > should achieve the same thing.
> > >
> > > Sorry, I have no experience with mariadb and podman. How am I supposed
> > > to run that?
> > > Is that supposed to run inside the container, on the host? Do I need
> > > to change the podman command lines?
> > >
> > > What I did before was:
> > >
> > > DEV=/dev/sdh
> > > MNT=/mnt/sdh
> > >
> > > mkfs.btrfs -f $DEV
> > > mount $DEV $MNT
> > >
> > > mkdir $MNT/noaio
> > > chown fdmanana: $MNT/noaio
> > >
> > > podman run --name mdbinit --rm -v $MNT/noaio/:/var/lib/mysql:Z -e
> > > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
> > > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
> > > --innodb_use_native_aio=0
> > >
> > >
> > > Then in another shell:
> > >
> > > podman kill --all
> > >
> > > podman run --rm -v $MNT/noaio/:/var/lib/mysql:Z -e
> > > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
> > > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
> > > --innodb_use_native_aio=1
> > >
> > >
> > > What should I change or add in there?
> >
> > Ok, just passing  --privileged to both podman commands triggered the
> > bug as in your report.
> > I'll see if I can figure out what's causing the read corruption.
> >
> >
> > >
> > > Thanks.