Crash on using the poll ring

Crash on using the poll ring

[Glauber Costa]

[-- Attachment #1: Type: text/plain, Size: 589 bytes --]

Hi

Today I found a crash when adding code for the poll ring to my implementation.
Kernel is 2b58a38ef46e91edd68eec58bdb817c42474cad6

Here's how to reproduce:

code at
https://github.com/glommer/seastar.git branch poll-ring

1. same as previous steps to configure seastar, but compile with:
ninja -C build/release apps/io_tester/io_tester

2. Download the yaml file attached

3. Run with:

./build/release/apps/io_tester/io_tester --conf ~/test.yaml --duration
15 --directory /var/disk1  --reactor-backend=uring --smp 1

(directory must be on xfs because we do c++ but we're not savages)

[-- Attachment #2: poll-ring.txt --]
[-- Type: text/plain, Size: 5156 bytes --]

[35739.749588] BUG: kernel NULL pointer dereference, address: 0000000000000000
[35739.749659] #PF: supervisor instruction fetch in kernel mode
[35739.749707] #PF: error_code(0x0010) - not-present page
[35739.749754] PGD 0 P4D 0 
[35739.749786] Oops: 0010 [#1] SMP NOPTI
[35739.749823] CPU: 0 PID: 14845 Comm: io_tester Not tainted 5.6.0-rc1+ #14
[35739.749881] Hardware name: Intel Corporation S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
[35739.749971] RIP: 0010:0x0
[35739.750004] Code: Bad RIP value.
[35739.750036] RSP: 0018:ffffb4e9cf357ba8 EFLAGS: 00010246
[35739.750083] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[35739.750148] RDX: 0000000000000001 RSI: ffff8af120830680 RDI: ffffffffb4248de0
[35739.750211] RBP: ffff8bc59a234070 R08: ffff8af12082b8d0 R09: ffff8af12082b8d0
[35739.750272] R10: 0000000000000321 R11: ffff8af120829ca4 R12: ffff8bc59a234b74
[35739.750333] R13: ffff8bc59a234000 R14: ffff8bc59a234001 R15: ffff8bc59dfd37b8
[35739.750397] FS:  00007f150b6f9900(0000) GS:ffff8af120800000(0000) knlGS:0000000000000000
[35739.750465] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[35739.750518] CR2: ffffffffffffffd6 CR3: 000000e7dd60a005 CR4: 00000000007606f0
[35739.750581] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[35739.750642] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[35739.750705] PKRU: 55555554
[35739.750732] Call Trace:
[35739.750768]  __task_work_run+0x67/0xa0
[35739.750811]  schedule+0xb1/0xf0
[35739.750845]  schedule_timeout+0x20f/0x300
[35739.750886]  ? __kfifo_to_user_r+0x90/0x90
[35739.750927]  ? __percpu_ref_switch_mode+0xd7/0x190
[35739.750976]  ? __wake_up_common_lock+0x8a/0xc0
[35739.751021]  wait_for_completion+0x119/0x160
[35739.751065]  ? wake_up_q+0xa0/0xa0
[35739.751103]  exit_aio+0xdc/0xf0
[35739.751132]  mmput+0x35/0x140
[35739.751160]  do_exit+0x30d/0xb90
[35739.751192]  ? __memcg_kmem_uncharge+0x34/0xa0
[35739.751231]  do_group_exit+0x3a/0xa0
[35739.751267]  get_signal+0x15b/0x890
[35739.751304]  do_signal+0x36/0x650
[35739.751338]  ? io_cqring_wait+0x1fc/0x220
[35739.751374]  ? io_uring_poll+0x80/0x80
[35739.751412]  exit_to_usermode_loop+0x9d/0x130
[35739.751452]  do_syscall_64+0x1a4/0x1c0
[35739.751489]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[35739.752868] RIP: 0033:0x7f150b7fa1ed
[35739.754195] Code: Bad RIP value.
[35739.755513] RSP: 002b:00007fff933aab38 EFLAGS: 00000216 ORIG_RAX: 00000000000001aa
[35739.756885] RAX: fffffffffffffffc RBX: 0000000000000000 RCX: 00007f150b7fa1ed
[35739.758267] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000007
[35739.759654] RBP: 0000600000076020 R08: 0000600000076020 R09: 0000000000000008
[35739.761005] R10: 0000000000000001 R11: 0000000000000216 R12: 0000000000000000
[35739.761907] R13: 00006000000c1060 R14: 0000000000000000 R15: 0000000000000001
[35739.762729] Modules linked in: ip6t_REJECT nf_reject_ipv6 ip6t_rpfilter ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib vfat fat ib_umad rpcrdma sunrpc intel_rapl_msr rdma_ucm intel_rapl_common ib_iser rdma_cm isst_if_common iw_cm ib_cm libiscsi skx_edac scsi_transport_iscsi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass iTCO_wdt crct10dif_pclmul iTCO_vendor_support i40iw crc32_pclmul ipmi_ssif ib_uverbs ghash_clmulni_intel intel_cstate ib_core intel_uncore ipmi_si intel_rapl_perf joydev ioatdma mei_me pcspkr switchtec i2c_i801 ipmi_devintf lpc_ich mei dca dax_pmem ipmi_msghandler dax_pmem_core acpi_power_meter acpi_pad
[35739.762774]  ip_tables xfs libcrc32c rfkill nd_pmem nd_btt ast i2c_algo_bit drm_vram_helper drm_ttm_helper ttm drm_kms_helper cec drm i40e nvme crc32c_intel megaraid_sas nvme_core nfit libnvdimm wmi pkcs8_key_parser
[35739.771965] CR2: 0000000000000000
[35739.772675] ---[ end trace d8d16013f6fe32fe ]---
[35739.855668] RIP: 0010:0x0
[35739.856235] Code: Bad RIP value.
[35739.856752] RSP: 0018:ffffb4e9cf357ba8 EFLAGS: 00010246
[35739.857265] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[35739.857772] RDX: 0000000000000001 RSI: ffff8af120830680 RDI: ffffffffb4248de0
[35739.858282] RBP: ffff8bc59a234070 R08: ffff8af12082b8d0 R09: ffff8af12082b8d0
[35739.858789] R10: 0000000000000321 R11: ffff8af120829ca4 R12: ffff8bc59a234b74
[35739.859299] R13: ffff8bc59a234000 R14: ffff8bc59a234001 R15: ffff8bc59dfd37b8
[35739.859815] FS:  00007f150b6f9900(0000) GS:ffff8af120800000(0000) knlGS:0000000000000000
[35739.860341] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[35739.860869] CR2: ffffffffffffffd6 CR3: 000000e7dd60a005 CR4: 00000000007606f0
[35739.861407] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[35739.861947] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[35739.862481] PKRU: 55555554


[-- Attachment #3: test.yaml --]
[-- Type: application/x-yaml, Size: 139 bytes --]

Re: Crash on using the poll ring

[Jens Axboe]

On 2/21/20 6:17 AM, Glauber Costa wrote:
> Hi
> 
> Today I found a crash when adding code for the poll ring to my implementation.
> Kernel is 2b58a38ef46e91edd68eec58bdb817c42474cad6
> 
> Here's how to reproduce:
> 
> code at
> https://github.com/glommer/seastar.git branch poll-ring
> 
> 1. same as previous steps to configure seastar, but compile with:
> ninja -C build/release apps/io_tester/io_tester
> 
> 2. Download the yaml file attached
> 
> 3. Run with:
> 
> ./build/release/apps/io_tester/io_tester --conf ~/test.yaml --duration
> 15 --directory /var/disk1  --reactor-backend=uring --smp 1
> 
> (directory must be on xfs because we do c++ but we're not savages)

This is due to killing the dummy callback function on the task work.
I'll play with this a bit and see how we can fix it.

-- 
Jens Axboe

Re: Crash on using the poll ring

[Jens Axboe]

On 2/21/20 7:51 AM, Jens Axboe wrote:
> On 2/21/20 6:17 AM, Glauber Costa wrote:
>> Hi
>>
>> Today I found a crash when adding code for the poll ring to my implementation.
>> Kernel is 2b58a38ef46e91edd68eec58bdb817c42474cad6
>>
>> Here's how to reproduce:
>>
>> code at
>> https://github.com/glommer/seastar.git branch poll-ring
>>
>> 1. same as previous steps to configure seastar, but compile with:
>> ninja -C build/release apps/io_tester/io_tester
>>
>> 2. Download the yaml file attached
>>
>> 3. Run with:
>>
>> ./build/release/apps/io_tester/io_tester --conf ~/test.yaml --duration
>> 15 --directory /var/disk1  --reactor-backend=uring --smp 1
>>
>> (directory must be on xfs because we do c++ but we're not savages)
> 
> This is due to killing the dummy callback function on the task work.
> I'll play with this a bit and see how we can fix it.

I re-did the code to use task_works instead, can you try the current
one? Same branch, sha is currently 9ba3cd1b8923.

-- 
Jens Axboe

Re: Crash on using the poll ring

[Glauber Costa]

On Fri, Feb 21, 2020 at 2:04 PM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 2/21/20 7:51 AM, Jens Axboe wrote:
> > On 2/21/20 6:17 AM, Glauber Costa wrote:
> >> Hi
> >>
> >> Today I found a crash when adding code for the poll ring to my implementation.
> >> Kernel is 2b58a38ef46e91edd68eec58bdb817c42474cad6
> >>
> >> Here's how to reproduce:
> >>
> >> code at
> >> https://github.com/glommer/seastar.git branch poll-ring
> >>
> >> 1. same as previous steps to configure seastar, but compile with:
> >> ninja -C build/release apps/io_tester/io_tester
> >>
> >> 2. Download the yaml file attached
> >>
> >> 3. Run with:
> >>
> >> ./build/release/apps/io_tester/io_tester --conf ~/test.yaml --duration
> >> 15 --directory /var/disk1  --reactor-backend=uring --smp 1
> >>
> >> (directory must be on xfs because we do c++ but we're not savages)
> >
> > This is due to killing the dummy callback function on the task work.
> > I'll play with this a bit and see how we can fix it.
>
> I re-did the code to use task_works instead, can you try the current
> one? Same branch, sha is currently 9ba3cd1b8923.

Tnx. I don't see a crash any longer.


>
> --
> Jens Axboe
>