Re: crash on connect

[Glauber Costa <glauber@scylladb.com>]

On Thu, Feb 20, 2020 at 2:19 PM Glauber Costa <glauber@scylladb.com> wrote:
>
> On Thu, Feb 20, 2020 at 2:12 PM Jens Axboe <axboe@kernel.dk> wrote:
> >
> > On 2/20/20 11:45 AM, Glauber Costa wrote:
> > > On Thu, Feb 20, 2020 at 12:28 PM Jens Axboe <axboe@kernel.dk> wrote:
> > >>
> > >> On 2/20/20 9:52 AM, Glauber Costa wrote:
> > >>> On Thu, Feb 20, 2020 at 11:39 AM Jens Axboe <axboe@kernel.dk> wrote:
> > >>>>
> > >>>> On 2/20/20 9:34 AM, Glauber Costa wrote:
> > >>>>> On Thu, Feb 20, 2020 at 11:29 AM Jens Axboe <axboe@kernel.dk> wrote:
> > >>>>>>
> > >>>>>> On 2/20/20 9:17 AM, Jens Axboe wrote:
> > >>>>>>> On 2/20/20 7:19 AM, Glauber Costa wrote:
> > >>>>>>>> Hi there, me again
> > >>>>>>>>
> > >>>>>>>> Kernel is at 043f0b67f2ab8d1af418056bc0cc6f0623d31347
> > >>>>>>>>
> > >>>>>>>> This test is easier to explain: it essentially issues a connect and a
> > >>>>>>>> shutdown right away.
> > >>>>>>>>
> > >>>>>>>> It currently fails due to no fault of io_uring. But every now and then
> > >>>>>>>> it crashes (you may have to run more than once to get it to crash)
> > >>>>>>>>
> > >>>>>>>> Instructions are similar to my last test.
> > >>>>>>>> Except the test to build is now "tests/unit/connect_test"
> > >>>>>>>> Code is at git@github.com:glommer/seastar.git  branch io-uring-connect-crash
> > >>>>>>>>
> > >>>>>>>> Run it with ./build/release/tests/unit/connect_test -- -c1
> > >>>>>>>> --reactor-backend=uring
> > >>>>>>>>
> > >>>>>>>> Backtrace attached
> > >>>>>>>
> > >>>>>>> Perfect thanks, I'll take a look!
> > >>>>>>
> > >>>>>> Haven't managed to crash it yet, but every run complains:
> > >>>>>>
> > >>>>>> got to shutdown of 10 with refcnt: 2
> > >>>>>> Refs being all dropped, calling forget for 10
> > >>>>>> terminate called after throwing an instance of 'fmt::v6::format_error'
> > >>>>>>   what():  argument index out of range
> > >>>>>> unknown location(0): fatal error: in "unixdomain_server": signal: SIGABRT (application abort requested)
> > >>>>>>
> > >>>>>> Not sure if that's causing it not to fail here.
> > >>>>>
> > >>>>> Ok, that means it "passed". (I was in the process of figuring out
> > >>>>> where I got this wrong when I started seeing the crashes)
> > >>>>
> > >>>> Can you do, in your kernel dir:
> > >>>>
> > >>>> $ gdb vmlinux
> > >>>> [...]
> > >>>> (gdb) l *__io_queue_sqe+0x4a
> > >>>>
> > >>>> and see what it says?
> > >>>
> > >>> 0xffffffff81375ada is in __io_queue_sqe (fs/io_uring.c:4814).
> > >>> 4809 struct io_kiocb *linked_timeout;
> > >>> 4810 struct io_kiocb *nxt = NULL;
> > >>> 4811 int ret;
> > >>> 4812
> > >>> 4813 again:
> > >>> 4814 linked_timeout = io_prep_linked_timeout(req);
> > >>> 4815
> > >>> 4816 ret = io_issue_sqe(req, sqe, &nxt, true);
> > >>> 4817
> > >>> 4818 /*
> > >>>
> > >>> (I am not using timeouts, just async_cancel)
> > >>
> > >> Can't seem to hit it here, went through thousands of iterations...
> > >> I'll keep trying.
> > >>
> > >> If you have time, you can try and enable CONFIG_KASAN=y and see if
> > >> you can hit it with that.
> > >
> > > I can
> > >
> > > Attaching full dmesg
> >
> > Can you try the latest? It's sha d8154e605f84.

10 runs, no crashes.

Thanks!

>> Before you do, can you
> > do the lookup on __io_queue_sqe+0x639 with gdb?
>
> Moving to that hash now. In the meantime, so I don't delay your fun:
>
> ) l *__io_queue_sqe+0x639
> 0xffffffff81566c19 is in __io_queue_sqe (./include/linux/compiler.h:226).
> 221 {
> 222 switch (size) {
> 223 case 1: *(volatile __u8 *)p = *(__u8 *)res; break;
> 224 case 2: *(volatile __u16 *)p = *(__u16 *)res; break;
> 225 case 4: *(volatile __u32 *)p = *(__u32 *)res; break;
> 226 case 8: *(volatile __u64 *)p = *(__u64 *)res; break;
> 227 default:
> 228 barrier();
> 229 __builtin_memcpy((void *)p, (const void *)res, size);
> 230 barrier();
>
>
> >
> > --
> > Jens Axboe
> >