btrfs / io-uring corrupting reads

btrfs / io-uring corrupting reads

[Daniel Black]

Per references at the bottom btfs + iouring manage to corrupt the
reading of a file.

Using podman here however docker or another container runtime will
probably work. As will any MariaDB-10.6 on a distro with a native
liburing userspace. Apologies for the older and bloated container
image.

Reproduction of bug:

using a btrfs:

$ dd if=/dev/zero  of=../btrfs.blk bs=1M count=2K
$ sudo losetup --direct-io=on  -f ../btrfs.blk
$ sudo mkfs.btrfs /dev/loop6
$ sudo mount /dev/loop/6 /mnt/btrfstest
$ sudo mkdir /mnt/btrfstest/noaio
$ sudo chown dan: /mnt/btrfstest/noaio

Initialize database on directory:

$ podman run --name mdbinit --rm -v
/mnt/btrfstest/noaio/:/var/lib/mysql:Z -e
MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
--innodb_use_native_aio=0

$ podman kill mdbinit

Switch to using uring to read:

$ podman run --rm -v /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e
MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
--innodb_use_native_aio=1

Failure observed on startup:

2022-02-21 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk
or a failed read of file './ibdata1' page [page id: space=0, page
number=9]. You may have to recover from a backup.

2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):

 len 16384; hex 00000000000000092022-02-21 14:43:31 0 [ERROR] InnoDB:
Database page corruption on disk or a failed read of file './ibdata1'
page [page id: space=0, page number=243]. You may have to recover from
a backup.

ffffffffffffff2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii
and hex (16384 bytes):

 len 16384; hex
00000000000000f3ffffff0000000000009c2045bf00ffffff0000000000ffffff00000000000000020171000900000000015500020006000700000000000000090000000000000000000000000500000000000000020572000000000000000204b208010000030085690000006e66696d756d000908a0d3000300000803000073757072656d756d000000000013080000100500a000000000000000000000000000000b5359535f464f524549474e18080000180500c0000000000000000c5359ffffff535f464fff0000524549474e5f434f4c531308ffffff0000200500ff0000db000000000000000d5359535f56495254550000000000000056414c00000000000020080000280501030000ffffffff0000ffffffff0000000000000000000e6d7973000000716c2f696e6e6f64625f7461626c655f7300ffffffff0000ffffffff0000000000746174732000ffff080000300501ffff0000ffffffff000005d6692b000000000000000f6d7973716c2f696e6e6f64625f696e6465785fd2007374617473220800003805015500000000000000106d7973716c2f7472616e73616374696f6e5f72656769737472791c0800004005000000f4ffffff74000000000000ffffff00116d79ffffffffffffffffffffffffffffffffffff73716c2f67ffffffffffffffffff7469645f736c6176655fffffffffffffffffff706f7300ffffffffffffff2022-02-21
14:43:31 0 [ERROR] InnoDB: Database page corruption on disk or a
failed read of file './mysql/innodb_table_stats.ibd' page [page id:
space=1, page number=0]. You may have to recover from a backup.

002022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):

Without --innodb_use_native_aio=0 as a container argument this starts
without error.

$ sudo losetup --direct-io=off  -f btrfs.blk also exhibits the failure

Observed failures in:
* 5.17.0-0.rc4.96.fc36.x86_64
* 5.16.8 (on nixos)
* 5.15.6

No observed failure:
* 5.15.14-200.fc35.x86_64
* 5.10

references:
* https://jira.mariadb.org/browse/MDEV-27900
* https://github.com/NixOS/nixpkgs/issues/160516
* https://jira.mariadb.org/browse/MDEV-27449

Fwd: btrfs / io-uring corrupting reads

[Daniel Black]

Per references at the bottom btfs + iouring manage to corrupt the
reading of a file.

Using podman here however docker or another container runtime will
probably work. As will any MariaDB-10.6 on a distro with a native
liburing userspace. Apologies for the older and bloated container
image.

Reproduction of bug:

using a btrfs:

$ dd if=/dev/zero  of=../btrfs.blk bs=1M count=2K
$ sudo losetup --direct-io=on  -f ../btrfs.blk
$ sudo mkfs.btrfs /dev/loop6
$ sudo mount /dev/loop/6 /mnt/btrfstest
$ sudo mkdir /mnt/btrfstest/noaio
$ sudo chown dan: /mnt/btrfstest/noaio

Initialize database on directory:

$ podman run --name mdbinit --rm -v
/mnt/btrfstest/noaio/:/var/lib/mysql:Z -e
MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
--innodb_use_native_aio=0

$ podman kill mdbinit

Switch to using uring to read:

$ podman run --rm -v /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e
MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
--innodb_use_native_aio=1

Failure observed on startup:

2022-02-21 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk
or a failed read of file './ibdata1' page [page id: space=0, page
number=9]. You may have to recover from a backup.

2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):

 len 16384; hex 00000000000000092022-02-21 14:43:31 0 [ERROR] InnoDB:
Database page corruption on disk or a failed read of file './ibdata1'
page [page id: space=0, page number=243]. You may have to recover from
a backup.

ffffffffffffff2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii
and hex (16384 bytes):

 len 16384; hex
00000000000000f3ffffff0000000000009c2045bf00ffffff0000000000ffffff00000000000000020171000900000000015500020006000700000000000000090000000000000000000000000500000000000000020572000000000000000204b208010000030085690000006e66696d756d000908a0d3000300000803000073757072656d756d000000000013080000100500a000000000000000000000000000000b5359535f464f524549474e18080000180500c0000000000000000c5359ffffff535f464fff0000524549474e5f434f4c531308ffffff0000200500ff0000db000000000000000d5359535f56495254550000000000000056414c00000000000020080000280501030000ffffffff0000ffffffff0000000000000000000e6d7973000000716c2f696e6e6f64625f7461626c655f7300ffffffff0000ffffffff0000000000746174732000ffff080000300501ffff0000ffffffff000005d6692b000000000000000f6d7973716c2f696e6e6f64625f696e6465785fd2007374617473220800003805015500000000000000106d7973716c2f7472616e73616374696f6e5f72656769737472791c0800004005000000f4ffffff74000000000000ffffff00116d79ffffffffffffffffffffffffffffffffffff73716c2f67ffffffffffffffffff7469645f736c6176655fffffffffffffffffff706f7300ffffffffffffff2022-02-21
14:43:31 0 [ERROR] InnoDB: Database page corruption on disk or a
failed read of file './mysql/innodb_table_stats.ibd' page [page id:
space=1, page number=0]. You may have to recover from a backup.

002022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):

Without --innodb_use_native_aio=0 as a container argument this starts
without error.

$ sudo losetup --direct-io=off  -f btrfs.blk also exhibits the failure

Observed failures in:
* 5.17.0-0.rc4.96.fc36.x86_64
* 5.16.8 (on nixos)
* 5.15.6

No observed failure:
* 5.15.14-200.fc35.x86_64
* 5.10

references:
* https://jira.mariadb.org/browse/MDEV-27900
* https://github.com/NixOS/nixpkgs/issues/160516
* https://jira.mariadb.org/browse/MDEV-27449

Re: Fwd: btrfs / io-uring corrupting reads

[Filipe Manana]

On Tue, Feb 22, 2022 at 08:53:02AM +1100, Daniel Black wrote:
> Per references at the bottom btfs + iouring manage to corrupt the
> reading of a file.
> 
> Using podman here however docker or another container runtime will
> probably work. As will any MariaDB-10.6 on a distro with a native
> liburing userspace. Apologies for the older and bloated container
> image.
> 
> Reproduction of bug:
> 
> using a btrfs:
> 
> $ dd if=/dev/zero  of=../btrfs.blk bs=1M count=2K
> $ sudo losetup --direct-io=on  -f ../btrfs.blk
> $ sudo mkfs.btrfs /dev/loop6
> $ sudo mount /dev/loop/6 /mnt/btrfstest
> $ sudo mkdir /mnt/btrfstest/noaio
> $ sudo chown dan: /mnt/btrfstest/noaio
> 
> Initialize database on directory:
> 
> $ podman run --name mdbinit --rm -v
> /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e
> MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
> quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
> --innodb_use_native_aio=0
> 
> $ podman kill mdbinit
> 
> Switch to using uring to read:
> 
> $ podman run --rm -v /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e
> MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
> quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
> --innodb_use_native_aio=1
> 
> Failure observed on startup:
> 
> 2022-02-21 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk
> or a failed read of file './ibdata1' page [page id: space=0, page
> number=9]. You may have to recover from a backup.

I gave it a try, but it fails setting up io_uring:

2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1
2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool

So that's why it doesn't fail here, as it fallbacks to no aio mode.

Any idea why it's failing to setup io_uring?

I have the liburing2 and liburing-dev packages installed on debian, and
tried with a 5.17-rc4 kernel.

I can run fio with io_uring as the ioengine (works perferctly so far).

Thanks.

> 
> 2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):
> 
>  len 16384; hex 00000000000000092022-02-21 14:43:31 0 [ERROR] InnoDB:
> Database page corruption on disk or a failed read of file './ibdata1'
> page [page id: space=0, page number=243]. You may have to recover from
> a backup.
> 
> ffffffffffffff2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii
> and hex (16384 bytes):
> 
>  len 16384; hex
> 00000000000000f3ffffff0000000000009c2045bf00ffffff0000000000ffffff00000000000000020171000900000000015500020006000700000000000000090000000000000000000000000500000000000000020572000000000000000204b208010000030085690000006e66696d756d000908a0d3000300000803000073757072656d756d000000000013080000100500a000000000000000000000000000000b5359535f464f524549474e18080000180500c0000000000000000c5359ffffff535f464fff0000524549474e5f434f4c531308ffffff0000200500ff0000db000000000000000d5359535f56495254550000000000000056414c00000000000020080000280501030000ffffffff0000ffffffff0000000000000000000e6d7973000000716c2f696e6e6f64625f7461626c655f7300ffffffff0000ffffffff0000000000746174732000ffff080000300501ffff0000ffffffff000005d6692b000000000000000f6d7973716c2f696e6e6f64625f696e6465785fd2007374617473220800003805015500000000000000106d7973716c2f7472616e73616374696f6e5f72656769737472791c0800004005000000f4ffffff74000000000000ffffff00116d79ffffffffffffffffffffffffffffffffffff73716c2f67ffffffffffffffffff7469645f736c6176655fffffffffffffffffff706f7300ffffffffffffff2022-02-21
> 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk or a
> failed read of file './mysql/innodb_table_stats.ibd' page [page id:
> space=1, page number=0]. You may have to recover from a backup.
> 
> 002022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):
> 
> Without --innodb_use_native_aio=0 as a container argument this starts
> without error.
> 
> $ sudo losetup --direct-io=off  -f btrfs.blk also exhibits the failure
> 
> Observed failures in:
> * 5.17.0-0.rc4.96.fc36.x86_64
> * 5.16.8 (on nixos)
> * 5.15.6
> 
> No observed failure:
> * 5.15.14-200.fc35.x86_64
> * 5.10
> 
> references:
> * https://jira.mariadb.org/browse/MDEV-27900
> * https://github.com/NixOS/nixpkgs/issues/160516
> * https://jira.mariadb.org/browse/MDEV-27449

Re: Fwd: btrfs / io-uring corrupting reads

[Daniel Black]

On Tue, Feb 22, 2022 at 10:42 PM Filipe Manana <fdmanana@kernel.org> wrote:

> I gave it a try, but it fails setting up io_uring:
>
> 2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
> 2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1
> 2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
> 2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
> 2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool
>
> So that's why it doesn't fail here, as it fallbacks to no aio mode.

error 1 is EPERM. Seems it needs --privileged on the container startup
as a podman argument (before the image name). Sorry I missed that

> Any idea why it's failing to setup io_uring?
>
> I have the liburing2 and liburing-dev packages installed on debian, and
> tried with a 5.17-rc4 kernel.

Taking https://packages.debian.org/bookworm/mariadb-server-core-10.6 package:

mariadb-install-db --no-defaults --datadir=/empty/btrfs/path
--innodb-use-native-aio=0

mariadbd --no-defaults --datadir=/empty/btrfs/path --innodb-use-native-aio=1

should achieve the same thing.

Re: Fwd: btrfs / io-uring corrupting reads

[Filipe Manana]

On Tue, Feb 22, 2022 at 12:22 PM Daniel Black <daniel@mariadb.org> wrote:
>
> On Tue, Feb 22, 2022 at 10:42 PM Filipe Manana <fdmanana@kernel.org> wrote:
>
> > I gave it a try, but it fails setting up io_uring:
> >
> > 2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
> > 2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1
> > 2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
> > 2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
> > 2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool
> >
> > So that's why it doesn't fail here, as it fallbacks to no aio mode.
>
> error 1 is EPERM. Seems it needs --privileged on the container startup
> as a podman argument (before the image name). Sorry I missed that
>
> > Any idea why it's failing to setup io_uring?
> >
> > I have the liburing2 and liburing-dev packages installed on debian, and
> > tried with a 5.17-rc4 kernel.
>
> Taking https://packages.debian.org/bookworm/mariadb-server-core-10.6 package:
>
> mariadb-install-db --no-defaults --datadir=/empty/btrfs/path
> --innodb-use-native-aio=0
>
> mariadbd --no-defaults --datadir=/empty/btrfs/path --innodb-use-native-aio=1
>
> should achieve the same thing.

Sorry, I have no experience with mariadb and podman. How am I supposed
to run that?
Is that supposed to run inside the container, on the host? Do I need
to change the podman command lines?

What I did before was:

DEV=/dev/sdh
MNT=/mnt/sdh

mkfs.btrfs -f $DEV
mount $DEV $MNT

mkdir $MNT/noaio
chown fdmanana: $MNT/noaio

podman run --name mdbinit --rm -v $MNT/noaio/:/var/lib/mysql:Z -e
MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
--innodb_use_native_aio=0


Then in another shell:

podman kill --all

podman run --rm -v $MNT/noaio/:/var/lib/mysql:Z -e
MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
--innodb_use_native_aio=1


What should I change or add in there?

Thanks.

Re: Fwd: btrfs / io-uring corrupting reads

[Filipe Manana]

On Tue, Feb 22, 2022 at 12:46 PM Filipe Manana <fdmanana@kernel.org> wrote:
>
> On Tue, Feb 22, 2022 at 12:22 PM Daniel Black <daniel@mariadb.org> wrote:
> >
> > On Tue, Feb 22, 2022 at 10:42 PM Filipe Manana <fdmanana@kernel.org> wrote:
> >
> > > I gave it a try, but it fails setting up io_uring:
> > >
> > > 2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
> > > 2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1
> > > 2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
> > > 2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
> > > 2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool
> > >
> > > So that's why it doesn't fail here, as it fallbacks to no aio mode.
> >
> > error 1 is EPERM. Seems it needs --privileged on the container startup
> > as a podman argument (before the image name). Sorry I missed that
> >
> > > Any idea why it's failing to setup io_uring?
> > >
> > > I have the liburing2 and liburing-dev packages installed on debian, and
> > > tried with a 5.17-rc4 kernel.
> >
> > Taking https://packages.debian.org/bookworm/mariadb-server-core-10.6 package:
> >
> > mariadb-install-db --no-defaults --datadir=/empty/btrfs/path
> > --innodb-use-native-aio=0
> >
> > mariadbd --no-defaults --datadir=/empty/btrfs/path --innodb-use-native-aio=1
> >
> > should achieve the same thing.
>
> Sorry, I have no experience with mariadb and podman. How am I supposed
> to run that?
> Is that supposed to run inside the container, on the host? Do I need
> to change the podman command lines?
>
> What I did before was:
>
> DEV=/dev/sdh
> MNT=/mnt/sdh
>
> mkfs.btrfs -f $DEV
> mount $DEV $MNT
>
> mkdir $MNT/noaio
> chown fdmanana: $MNT/noaio
>
> podman run --name mdbinit --rm -v $MNT/noaio/:/var/lib/mysql:Z -e
> MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
> quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
> --innodb_use_native_aio=0
>
>
> Then in another shell:
>
> podman kill --all
>
> podman run --rm -v $MNT/noaio/:/var/lib/mysql:Z -e
> MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
> quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
> --innodb_use_native_aio=1
>
>
> What should I change or add in there?

Ok, just passing  --privileged to both podman commands triggered the
bug as in your report.
I'll see if I can figure out what's causing the read corruption.


>
> Thanks.

Re: Fwd: btrfs / io-uring corrupting reads

[Daniel Black]

Filipe,

DId you find anything? This is starting to be noticed by our mutual users.

https://jira.mariadb.org/browse/MDEV-27900
https://mariadb.zulipchat.com/#narrow/stream/118759-general/topic/Corrupt.20database.20page.20when.20updating.20from.2010.2E5

On Tue, Feb 22, 2022 at 11:55 PM Filipe Manana <fdmanana@kernel.org> wrote:
>
> On Tue, Feb 22, 2022 at 12:46 PM Filipe Manana <fdmanana@kernel.org> wrote:
> >
> > On Tue, Feb 22, 2022 at 12:22 PM Daniel Black <daniel@mariadb.org> wrote:
> > >
> > > On Tue, Feb 22, 2022 at 10:42 PM Filipe Manana <fdmanana@kernel.org> wrote:
> > >
> > > > I gave it a try, but it fails setting up io_uring:
> > > >
> > > > 2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
> > > > 2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1
> > > > 2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
> > > > 2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
> > > > 2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool
> > > >
> > > > So that's why it doesn't fail here, as it fallbacks to no aio mode.
> > >
> > > error 1 is EPERM. Seems it needs --privileged on the container startup
> > > as a podman argument (before the image name). Sorry I missed that
> > >
> > > > Any idea why it's failing to setup io_uring?
> > > >
> > > > I have the liburing2 and liburing-dev packages installed on debian, and
> > > > tried with a 5.17-rc4 kernel.
> > >
> > > Taking https://packages.debian.org/bookworm/mariadb-server-core-10.6 package:
> > >
> > > mariadb-install-db --no-defaults --datadir=/empty/btrfs/path
> > > --innodb-use-native-aio=0
> > >
> > > mariadbd --no-defaults --datadir=/empty/btrfs/path --innodb-use-native-aio=1
> > >
> > > should achieve the same thing.
> >
> > Sorry, I have no experience with mariadb and podman. How am I supposed
> > to run that?
> > Is that supposed to run inside the container, on the host? Do I need
> > to change the podman command lines?
> >
> > What I did before was:
> >
> > DEV=/dev/sdh
> > MNT=/mnt/sdh
> >
> > mkfs.btrfs -f $DEV
> > mount $DEV $MNT
> >
> > mkdir $MNT/noaio
> > chown fdmanana: $MNT/noaio
> >
> > podman run --name mdbinit --rm -v $MNT/noaio/:/var/lib/mysql:Z -e
> > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
> > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
> > --innodb_use_native_aio=0
> >
> >
> > Then in another shell:
> >
> > podman kill --all
> >
> > podman run --rm -v $MNT/noaio/:/var/lib/mysql:Z -e
> > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
> > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
> > --innodb_use_native_aio=1
> >
> >
> > What should I change or add in there?
>
> Ok, just passing  --privileged to both podman commands triggered the
> bug as in your report.
> I'll see if I can figure out what's causing the read corruption.
>
>
> >
> > Thanks.

Re: Fwd: btrfs / io-uring corrupting reads

[Filipe Manana]

On Wed, Mar 02, 2022 at 08:51:53AM +1100, Daniel Black wrote:
> Filipe,
> 
> DId you find anything? This is starting to be noticed by our mutual users.

Yes, I've sent a patch to address it:

https://lore.kernel.org/linux-btrfs/39c96b5608ed99b7d666d4d159f8d135e86b9606.1646219178.git.fdmanana@suse.com/T/#u

The details are in the changelog.

Basically, MariaDB is getting a short read - it asks to read 16K of data
but it only gets the first 4K of data (or the first 8K or 12K). It treats
such partial read as a corruption, but in fact it could check that it's a
short read and then try to read the remaining data.

I understand it's an unexpected result, since it knows its read requests
don't cross the EOF boundary and knows exactly the length and file offsets
of its pages/blocks, so it expects a read requesting 16K of data to return
exactly 16K of data, and no less than that. I've worked in the database
industry before, and the same assumptions existed on the engines I worked on.
Either way it was a behaviour change in btrfs and it can, and should be,
addressed in btrfs.

And yes, before your report, there was at least another on the btrfs mailing
list from someone getting MariaDB corruptions on btrfs only. However yours had
a useful reproducer that made it easier to dig into.

Thanks.

> 
> https://jira.mariadb.org/browse/MDEV-27900
> https://mariadb.zulipchat.com/#narrow/stream/118759-general/topic/Corrupt.20database.20page.20when.20updating.20from.2010.2E5
> 
> On Tue, Feb 22, 2022 at 11:55 PM Filipe Manana <fdmanana@kernel.org> wrote:
> >
> > On Tue, Feb 22, 2022 at 12:46 PM Filipe Manana <fdmanana@kernel.org> wrote:
> > >
> > > On Tue, Feb 22, 2022 at 12:22 PM Daniel Black <daniel@mariadb.org> wrote:
> > > >
> > > > On Tue, Feb 22, 2022 at 10:42 PM Filipe Manana <fdmanana@kernel.org> wrote:
> > > >
> > > > > I gave it a try, but it fails setting up io_uring:
> > > > >
> > > > > 2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
> > > > > 2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1
> > > > > 2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
> > > > > 2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
> > > > > 2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool
> > > > >
> > > > > So that's why it doesn't fail here, as it fallbacks to no aio mode.
> > > >
> > > > error 1 is EPERM. Seems it needs --privileged on the container startup
> > > > as a podman argument (before the image name). Sorry I missed that
> > > >
> > > > > Any idea why it's failing to setup io_uring?
> > > > >
> > > > > I have the liburing2 and liburing-dev packages installed on debian, and
> > > > > tried with a 5.17-rc4 kernel.
> > > >
> > > > Taking https://packages.debian.org/bookworm/mariadb-server-core-10.6 package:
> > > >
> > > > mariadb-install-db --no-defaults --datadir=/empty/btrfs/path
> > > > --innodb-use-native-aio=0
> > > >
> > > > mariadbd --no-defaults --datadir=/empty/btrfs/path --innodb-use-native-aio=1
> > > >
> > > > should achieve the same thing.
> > >
> > > Sorry, I have no experience with mariadb and podman. How am I supposed
> > > to run that?
> > > Is that supposed to run inside the container, on the host? Do I need
> > > to change the podman command lines?
> > >
> > > What I did before was:
> > >
> > > DEV=/dev/sdh
> > > MNT=/mnt/sdh
> > >
> > > mkfs.btrfs -f $DEV
> > > mount $DEV $MNT
> > >
> > > mkdir $MNT/noaio
> > > chown fdmanana: $MNT/noaio
> > >
> > > podman run --name mdbinit --rm -v $MNT/noaio/:/var/lib/mysql:Z -e
> > > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
> > > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
> > > --innodb_use_native_aio=0
> > >
> > >
> > > Then in another shell:
> > >
> > > podman kill --all
> > >
> > > podman run --rm -v $MNT/noaio/:/var/lib/mysql:Z -e
> > > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
> > > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
> > > --innodb_use_native_aio=1
> > >
> > >
> > > What should I change or add in there?
> >
> > Ok, just passing  --privileged to both podman commands triggered the
> > bug as in your report.
> > I'll see if I can figure out what's causing the read corruption.
> >
> >
> > >
> > > Thanks.