Fwd: btrfs / io-uring corrupting reads

[Daniel Black <daniel@mariadb.org>]

Per references at the bottom btfs + iouring manage to corrupt the
reading of a file.

Using podman here however docker or another container runtime will
probably work. As will any MariaDB-10.6 on a distro with a native
liburing userspace. Apologies for the older and bloated container
image.

Reproduction of bug:

using a btrfs:

$ dd if=/dev/zero  of=../btrfs.blk bs=1M count=2K
$ sudo losetup --direct-io=on  -f ../btrfs.blk
$ sudo mkfs.btrfs /dev/loop6
$ sudo mount /dev/loop/6 /mnt/btrfstest
$ sudo mkdir /mnt/btrfstest/noaio
$ sudo chown dan: /mnt/btrfstest/noaio

Initialize database on directory:

$ podman run --name mdbinit --rm -v
/mnt/btrfstest/noaio/:/var/lib/mysql:Z -e
MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
--innodb_use_native_aio=0

$ podman kill mdbinit

Switch to using uring to read:

$ podman run --rm -v /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e
MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
--innodb_use_native_aio=1

Failure observed on startup:

2022-02-21 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk
or a failed read of file './ibdata1' page [page id: space=0, page
number=9]. You may have to recover from a backup.

2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):

 len 16384; hex 00000000000000092022-02-21 14:43:31 0 [ERROR] InnoDB:
Database page corruption on disk or a failed read of file './ibdata1'
page [page id: space=0, page number=243]. You may have to recover from
a backup.

ffffffffffffff2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii
and hex (16384 bytes):

 len 16384; hex
00000000000000f3ffffff0000000000009c2045bf00ffffff0000000000ffffff00000000000000020171000900000000015500020006000700000000000000090000000000000000000000000500000000000000020572000000000000000204b208010000030085690000006e66696d756d000908a0d3000300000803000073757072656d756d000000000013080000100500a000000000000000000000000000000b5359535f464f524549474e18080000180500c0000000000000000c5359ffffff535f464fff0000524549474e5f434f4c531308ffffff0000200500ff0000db000000000000000d5359535f56495254550000000000000056414c00000000000020080000280501030000ffffffff0000ffffffff0000000000000000000e6d7973000000716c2f696e6e6f64625f7461626c655f7300ffffffff0000ffffffff0000000000746174732000ffff080000300501ffff0000ffffffff000005d6692b000000000000000f6d7973716c2f696e6e6f64625f696e6465785fd2007374617473220800003805015500000000000000106d7973716c2f7472616e73616374696f6e5f72656769737472791c0800004005000000f4ffffff74000000000000ffffff00116d79ffffffffffffffffffffffffffffffffffff73716c2f67ffffffffffffffffff7469645f736c6176655fffffffffffffffffff706f7300ffffffffffffff2022-02-21
14:43:31 0 [ERROR] InnoDB: Database page corruption on disk or a
failed read of file './mysql/innodb_table_stats.ibd' page [page id:
space=1, page number=0]. You may have to recover from a backup.

002022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):

Without --innodb_use_native_aio=0 as a container argument this starts
without error.

$ sudo losetup --direct-io=off  -f btrfs.blk also exhibits the failure

Observed failures in:
* 5.17.0-0.rc4.96.fc36.x86_64
* 5.16.8 (on nixos)
* 5.15.6

No observed failure:
* 5.15.14-200.fc35.x86_64
* 5.10

references:
* https://jira.mariadb.org/browse/MDEV-27900
* https://github.com/NixOS/nixpkgs/issues/160516
* https://jira.mariadb.org/browse/MDEV-27449