Re: crash on connect

[Glauber Costa <glauber@scylladb.com>]

On Thu, Feb 20, 2020 at 2:12 PM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 2/20/20 11:45 AM, Glauber Costa wrote:
> > On Thu, Feb 20, 2020 at 12:28 PM Jens Axboe <axboe@kernel.dk> wrote:
> >>
> >> On 2/20/20 9:52 AM, Glauber Costa wrote:
> >>> On Thu, Feb 20, 2020 at 11:39 AM Jens Axboe <axboe@kernel.dk> wrote:
> >>>>
> >>>> On 2/20/20 9:34 AM, Glauber Costa wrote:
> >>>>> On Thu, Feb 20, 2020 at 11:29 AM Jens Axboe <axboe@kernel.dk> wrote:
> >>>>>>
> >>>>>> On 2/20/20 9:17 AM, Jens Axboe wrote:
> >>>>>>> On 2/20/20 7:19 AM, Glauber Costa wrote:
> >>>>>>>> Hi there, me again
> >>>>>>>>
> >>>>>>>> Kernel is at 043f0b67f2ab8d1af418056bc0cc6f0623d31347
> >>>>>>>>
> >>>>>>>> This test is easier to explain: it essentially issues a connect and a
> >>>>>>>> shutdown right away.
> >>>>>>>>
> >>>>>>>> It currently fails due to no fault of io_uring. But every now and then
> >>>>>>>> it crashes (you may have to run more than once to get it to crash)
> >>>>>>>>
> >>>>>>>> Instructions are similar to my last test.
> >>>>>>>> Except the test to build is now "tests/unit/connect_test"
> >>>>>>>> Code is at git@github.com:glommer/seastar.git  branch io-uring-connect-crash
> >>>>>>>>
> >>>>>>>> Run it with ./build/release/tests/unit/connect_test -- -c1
> >>>>>>>> --reactor-backend=uring
> >>>>>>>>
> >>>>>>>> Backtrace attached
> >>>>>>>
> >>>>>>> Perfect thanks, I'll take a look!
> >>>>>>
> >>>>>> Haven't managed to crash it yet, but every run complains:
> >>>>>>
> >>>>>> got to shutdown of 10 with refcnt: 2
> >>>>>> Refs being all dropped, calling forget for 10
> >>>>>> terminate called after throwing an instance of 'fmt::v6::format_error'
> >>>>>>   what():  argument index out of range
> >>>>>> unknown location(0): fatal error: in "unixdomain_server": signal: SIGABRT (application abort requested)
> >>>>>>
> >>>>>> Not sure if that's causing it not to fail here.
> >>>>>
> >>>>> Ok, that means it "passed". (I was in the process of figuring out
> >>>>> where I got this wrong when I started seeing the crashes)
> >>>>
> >>>> Can you do, in your kernel dir:
> >>>>
> >>>> $ gdb vmlinux
> >>>> [...]
> >>>> (gdb) l *__io_queue_sqe+0x4a
> >>>>
> >>>> and see what it says?
> >>>
> >>> 0xffffffff81375ada is in __io_queue_sqe (fs/io_uring.c:4814).
> >>> 4809 struct io_kiocb *linked_timeout;
> >>> 4810 struct io_kiocb *nxt = NULL;
> >>> 4811 int ret;
> >>> 4812
> >>> 4813 again:
> >>> 4814 linked_timeout = io_prep_linked_timeout(req);
> >>> 4815
> >>> 4816 ret = io_issue_sqe(req, sqe, &nxt, true);
> >>> 4817
> >>> 4818 /*
> >>>
> >>> (I am not using timeouts, just async_cancel)
> >>
> >> Can't seem to hit it here, went through thousands of iterations...
> >> I'll keep trying.
> >>
> >> If you have time, you can try and enable CONFIG_KASAN=y and see if
> >> you can hit it with that.
> >
> > I can
> >
> > Attaching full dmesg
>
> Can you try the latest? It's sha d8154e605f84. Before you do, can you
> do the lookup on __io_queue_sqe+0x639 with gdb?

Moving to that hash now. In the meantime, so I don't delay your fun:

) l *__io_queue_sqe+0x639
0xffffffff81566c19 is in __io_queue_sqe (./include/linux/compiler.h:226).
221 {
222 switch (size) {
223 case 1: *(volatile __u8 *)p = *(__u8 *)res; break;
224 case 2: *(volatile __u16 *)p = *(__u16 *)res; break;
225 case 4: *(volatile __u32 *)p = *(__u32 *)res; break;
226 case 8: *(volatile __u64 *)p = *(__u64 *)res; break;
227 default:
228 barrier();
229 __builtin_memcpy((void *)p, (const void *)res, size);
230 barrier();


>
> --
> Jens Axboe
>