public inbox for [email protected]
 help / color / mirror / Atom feed
From: Andres Freund <[email protected]>
To: Jens Axboe <[email protected]>
Cc: [email protected], [email protected],
	[email protected], [email protected],
	[email protected]
Subject: Re: [PATCHSET v5 0/12] Add support for async buffered reads
Date: Fri, 5 Jun 2020 12:56:17 -0700	[thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>

Hi,

On 2020-06-03 18:30:45 -0700, Andres Freund wrote:
> > I'll try and reproduce this, any chance you have a test case that can
> > be run so I don't have to write one from scratch? The more detailed
> > instructions the better.
>
> It shouldn't be too hard to write you a detailed script for reproducing
> the issue. But it'd not be an all that minimal reproducer, unless it
> also triggers on smaller scale (it's a 130GB database that triggers the
> problem reliably, and small tables don't seem to do so reliably).
>
> I'll try to write that up after I set up kvm / repro there.

Sorry, took a bit longer. Ran into some issues with my aio branch when
running on 5.6. Not yet sure if that's my fault, or io_uring's.


Here's a recipe that repros reliably for me. For me it triggers reliably
at this size (-s 1000 ends up around ~16GB).


# Build aio branch of postgres
sudo mkdir /srv/src
sudo chown andres. /srv/src
cd /srv/src
git clone -b aio https://github.com/anarazel/postgres.git pg
cd pg
./configure --enable-depend --with-liburing --prefix=/home/andres/bin/
make -j48 -s install

# check that target directory is in PATH
type psql
psql is /home/andres/bin/bin/psql

in one shell:
# create database directory, start postgres
echo 1200 | sudo tee /proc/sys/vm/nr_hugepages
initdb /srv/pguringcrash
postgres -D /srv/pguringcrash/ -c bgwriter_delay=10ms -c bgwriter_lru_maxpages=1000 -c wal_level=minimal -c max_wal_senders=0 -c wal_buffers=128MB -c max_wal_size=100GB -c shared_buffers=2GB -c max_parallel_workers_per_gather=8 -c huge_pages=on

in the other shell:
# fill with test data (skipping index creation etc)
pgbench -i -s 1000 postgres -IdtGv -q

# and trigger a parallel sequential scan, using 8 additional workers
echo 3 | sudo tee /proc/sys/vm/drop_caches && psql -c 'SET max_parallel_workers_per_gather=8' -c 'SELECT SUM(abalance) FROM pgbench_accounts ;'


For me that faily reliably triggers the issue within a few seconds.

[  323.312854] BUG: unable to handle page fault for address: 00007f0f1428c000
[  323.319879] #PF: supervisor read access in kernel mode
[  323.321391] #PF: error_code(0x0001) - permissions violation
[  323.323019] PGD 102e760067 P4D 102e760067 PUD 1037878067 PMD 8000000fc7e000e7
[  323.325065] Oops: 0001 [#1] SMP NOPTI
[  323.326248] CPU: 1 PID: 3145 Comm: postgres Not tainted 5.7.0-rc7-andres-00133-gc8707bf69395 #43
[  323.328653] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-1 04/01/2014
[  323.330365] RIP: 0010:__wake_up_common+0x53/0x140
[  323.331276] Code: 41 f6 01 04 0f 85 a3 00 00 00 48 8b 43 08 48 8d 78 e8 48 8d 43 08 48 89 c1 48 89 04 24 48 8d 47 18 48 39 c1 0f 84 b8 00 00 00 <48> 8b 47 18 89 54 24 14 31 ed 4c 8d 60 e8 8b 1f f6 c3 04 75 52 48
[  323.334419] RSP: 0018:ffffc90000dcbdc8 EFLAGS: 00010086
[  323.335409] RAX: 00007f0f1428c000 RBX: ffff889038b189f8 RCX: ffff889038b18a00
[  323.336723] RDX: 0000000000000001 RSI: 0000000000000003 RDI: 00007f0f1428bfe8
[  323.338035] RBP: 0000000000000246 R08: 0000000000000001 R09: ffffc90000dcbe18
[  323.339331] R10: ffff889036a5e750 R11: 0000000000000000 R12: 0000000000000001
[  323.340312] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000001
[  323.341204] FS:  00007f0f6f271740(0000) GS:ffff88903f840000(0000) knlGS:0000000000000000
[  323.342217] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  323.342950] CR2: 00007f0f1428c000 CR3: 0000001037ed0005 CR4: 0000000000760ee0
[  323.343836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  323.344724] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  323.345613] PKRU: 55555554
[  323.346017] Call Trace:
[  323.346435]  __wake_up_common_lock+0x7c/0xc0
[  323.347015]  ep_free+0x30/0xd0
[  323.347468]  ep_eventpoll_release+0x16/0x20
[  323.348040]  __fput+0xda/0x240
[  323.348498]  task_work_run+0x62/0x90
[  323.349015]  exit_to_usermode_loop+0xbd/0xe0
[  323.349573]  do_syscall_64+0xf2/0x130
[  323.349987]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  323.350523] RIP: 0033:0x7f0f6f5a5f13
[  323.350906] Code: 00 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb bb 0f 1f 80 00 00 00 00 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 45 c3 0f 1f 40 00 48 83 ec 18 89 7c 24 0c e8
[  323.352501] RSP: 002b:00007ffdf3eab5a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
[  323.353277] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f0f6f5a5f13
[  323.353947] RDX: 0000000000000001 RSI: 000055ef9f5cb360 RDI: 000000000000002a
[  323.354587] RBP: 000055ef9f5cb2e8 R08: 00000000000280e8 R09: 0000000000000143
[  323.355227] R10: 000000000000000a R11: 0000000000000246 R12: 000055ef9f5cb2e8
[  323.355872] R13: 0000000000000008 R14: 00000000ffffffff R15: 00007f0f6d3407a4
[  323.356510] Modules linked in: 9pnet_virtio isst_if_common iTCO_wdt 9pnet xhci_pci iTCO_vendor_support xhci_hcd
[  323.357477] CR2: 00007f0f1428c000
[  323.357866] ---[ end trace 081898b3c6b5ab1b ]---


I'll try your new branch next.

Greetings,

Andres Freund

  reply	other threads:[~2020-06-05 19:56 UTC|newest]

Thread overview: 63+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-26 19:51 [PATCHSET v5 0/12] Add support for async buffered reads Jens Axboe
2020-05-26 19:51 ` [PATCH 01/12] block: read-ahead submission should imply no-wait as well Jens Axboe
2020-05-26 19:51 ` [PATCH 02/12] mm: allow read-ahead with IOCB_NOWAIT set Jens Axboe
2020-05-26 20:45   ` Johannes Weiner
2020-05-26 19:51 ` [PATCH 03/12] mm: abstract out wake_page_match() from wake_page_function() Jens Axboe
2020-05-26 21:02   ` Johannes Weiner
2020-06-01 14:16   ` Matthew Wilcox
2020-05-26 19:51 ` [PATCH 04/12] mm: add support for async page locking Jens Axboe
2020-05-26 21:59   ` Johannes Weiner
2020-05-26 22:01     ` Jens Axboe
2020-05-27 16:02       ` Johannes Weiner
2020-06-01 14:26   ` Matthew Wilcox
2020-06-01 17:15     ` Jens Axboe
2020-05-26 19:51 ` [PATCH 05/12] mm: support async buffered reads in generic_file_buffered_read() Jens Axboe
2020-05-26 22:00   ` Johannes Weiner
2020-05-26 19:51 ` [PATCH 06/12] fs: add FMODE_BUF_RASYNC Jens Axboe
2020-05-26 19:51 ` [PATCH 07/12] ext4: flag as supporting buffered async reads Jens Axboe
2020-05-26 19:51 ` [PATCH 08/12] block: flag block devices as supporting IOCB_WAITQ Jens Axboe
2020-05-26 19:51 ` [PATCH 09/12] xfs: flag files as supporting buffered async reads Jens Axboe
2020-05-28 17:53   ` Darrick J. Wong
2020-05-28 19:23     ` Jens Axboe
2020-05-26 19:51 ` [PATCH 10/12] btrfs: " Jens Axboe
2020-05-26 19:57   ` Chris Mason
2020-05-26 19:51 ` [PATCH 11/12] mm: add kiocb_wait_page_queue_init() helper Jens Axboe
2020-05-26 22:01   ` Johannes Weiner
2020-05-26 19:51 ` [PATCH 12/12] io_uring: support true async buffered reads, if file provides it Jens Axboe
     [not found] ` <CA+icZUWfX+QmroE6j74C7o-BdfMF5=6PdYrA=5W_JCKddqkJgQ@mail.gmail.com>
2020-05-28 17:06   ` [PATCHSET v5 0/12] Add support for async buffered reads Jens Axboe
2020-05-28 17:12     ` Sedat Dilek
2020-05-28 17:14       ` Jens Axboe
2020-05-28 18:20         ` Sedat Dilek
2020-05-29 10:02     ` Sedat Dilek
2020-05-29 11:22       ` Sedat Dilek
2020-05-30 13:36         ` Sedat Dilek
2020-05-30 18:57           ` Sedat Dilek
2020-05-31  1:57             ` Jens Axboe
     [not found]               ` <CA+icZUXxmOA-5+dukCgxfSp4eVHB+QaAHO6tsgq0iioQs3Af-w@mail.gmail.com>
2020-05-31  7:12                 ` Sedat Dilek
2020-06-01 13:35                   ` Sedat Dilek
2020-06-01 14:04                     ` Jens Axboe
2020-06-01 14:13                       ` Sedat Dilek
2020-06-01 14:14                         ` Jens Axboe
2020-06-01 14:35                           ` Jens Axboe
2020-06-01 14:43                             ` Sedat Dilek
2020-06-01 14:46                               ` Jens Axboe
2020-06-01 14:51                                 ` Sedat Dilek
2020-06-01 20:18                                 ` Sedat Dilek
     [not found] ` <[email protected]>
2020-06-04  1:04   ` Jens Axboe
2020-06-04  1:30     ` Andres Freund
2020-06-05 19:56       ` Andres Freund [this message]
2020-06-05 14:42     ` Jens Axboe
2020-06-05 20:20       ` Andres Freund
2020-06-05 20:21         ` Jens Axboe
2020-06-05 20:36         ` Andres Freund
2020-06-05 20:53           ` Jens Axboe
2020-06-05 21:13             ` Jens Axboe
2020-06-05 21:21               ` Jens Axboe
2020-06-05 22:30                 ` Andres Freund
2020-06-05 22:36                   ` Andres Freund
2020-06-05 22:49                     ` Jens Axboe
2020-06-05 22:54                       ` Andres Freund
2020-06-05 22:56                         ` Jens Axboe
2020-06-05 23:02                           ` Andres Freund
2020-06-06  0:33                 ` Sedat Dilek
2020-06-06 16:04                   ` Jens Axboe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox