From: Qu Wenruo <[email protected]>
To: "[email protected]" <[email protected]>,
Linux FS Devel <[email protected]>,
[email protected]
Subject: Possible io_uring related race leads to btrfs data csum mismatch
Date: Wed, 16 Aug 2023 14:52:20 +0800 [thread overview]
Message-ID: <[email protected]> (raw)
Hi,
Recently I'm digging into a very rare failure during btrfs/06[234567],
where btrfs scrub detects unrepairable data corruption.
After days of digging, I have a much smaller reproducer:
```
fail()
{
echo "!!! FAILED !!!"
exit 1
}
workload()
{
mkfs.btrfs -f -m single -d single --csum sha256 $dev1
mount $dev1 $mnt
# There are around 10 more combinations with different
# seed and -p/-n parameters, but this is the smallest one
# I found so far.
$fsstress -p 7 -n 50 -s 1691396493 -w -d $mnt
umount $mnt
btrfs check --check-data-csum $dev1 || fail
}
runtime=1024
for (( i = 0; i < $runtime; i++ )); do
echo "=== $i / $runtime ==="
workload
done
```
At least here, with a VM with 6 cores (host has 8C/16T), fast enough
storage (PCIE4.0 NVME, with unsafe cache mode), it has the chance around
1/100 to hit the error.
Checking the fsstress verbose log against the failed file, it turns out
to be an io_uring write.
And with uring_write disabled in fsstress, I have no longer reproduced
the csum mismatch, even with much larger -n and -p parameters.
However I didn't see any io_uring related callback inside btrfs code,
any advice on the io_uring part would be appreciated.
Thanks,
Qu
next reply other threads:[~2023-08-16 6:53 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-08-16 6:52 Qu Wenruo [this message]
2023-08-16 14:33 ` Possible io_uring related race leads to btrfs data csum mismatch Jens Axboe
2023-08-16 14:49 ` Jens Axboe
2023-08-16 21:46 ` Qu Wenruo
2023-08-16 22:28 ` Jens Axboe
2023-08-17 1:05 ` Qu Wenruo
2023-08-17 1:12 ` Jens Axboe
2023-08-17 1:19 ` Qu Wenruo
2023-08-17 1:23 ` Jens Axboe
2023-08-17 1:31 ` Qu Wenruo
2023-08-17 1:32 ` Jens Axboe
2023-08-19 23:59 ` Qu Wenruo
2023-08-20 0:22 ` Qu Wenruo
2023-08-20 13:26 ` Jens Axboe
2023-08-20 14:11 ` Jens Axboe
2023-08-20 18:18 ` Matthew Wilcox
2023-08-20 18:40 ` Jens Axboe
2023-08-21 0:38 ` Qu Wenruo
2023-08-21 14:57 ` Jens Axboe
2023-08-21 21:42 ` Qu Wenruo
2023-08-16 22:36 ` Jens Axboe
2023-08-17 0:40 ` Qu Wenruo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox