Re: crash on connect

[Jens Axboe <axboe@kernel.dk>]

On 2/20/20 12:36 PM, Glauber Costa wrote:
> On Thu, Feb 20, 2020 at 2:19 PM Glauber Costa <glauber@scylladb.com> wrote:
>>
>> On Thu, Feb 20, 2020 at 2:12 PM Jens Axboe <axboe@kernel.dk> wrote:
>>>
>>> On 2/20/20 11:45 AM, Glauber Costa wrote:
>>>> On Thu, Feb 20, 2020 at 12:28 PM Jens Axboe <axboe@kernel.dk> wrote:
>>>>>
>>>>> On 2/20/20 9:52 AM, Glauber Costa wrote:
>>>>>> On Thu, Feb 20, 2020 at 11:39 AM Jens Axboe <axboe@kernel.dk> wrote:
>>>>>>>
>>>>>>> On 2/20/20 9:34 AM, Glauber Costa wrote:
>>>>>>>> On Thu, Feb 20, 2020 at 11:29 AM Jens Axboe <axboe@kernel.dk> wrote:
>>>>>>>>>
>>>>>>>>> On 2/20/20 9:17 AM, Jens Axboe wrote:
>>>>>>>>>> On 2/20/20 7:19 AM, Glauber Costa wrote:
>>>>>>>>>>> Hi there, me again
>>>>>>>>>>>
>>>>>>>>>>> Kernel is at 043f0b67f2ab8d1af418056bc0cc6f0623d31347
>>>>>>>>>>>
>>>>>>>>>>> This test is easier to explain: it essentially issues a connect and a
>>>>>>>>>>> shutdown right away.
>>>>>>>>>>>
>>>>>>>>>>> It currently fails due to no fault of io_uring. But every now and then
>>>>>>>>>>> it crashes (you may have to run more than once to get it to crash)
>>>>>>>>>>>
>>>>>>>>>>> Instructions are similar to my last test.
>>>>>>>>>>> Except the test to build is now "tests/unit/connect_test"
>>>>>>>>>>> Code is at git@github.com:glommer/seastar.git  branch io-uring-connect-crash
>>>>>>>>>>>
>>>>>>>>>>> Run it with ./build/release/tests/unit/connect_test -- -c1
>>>>>>>>>>> --reactor-backend=uring
>>>>>>>>>>>
>>>>>>>>>>> Backtrace attached
>>>>>>>>>>
>>>>>>>>>> Perfect thanks, I'll take a look!
>>>>>>>>>
>>>>>>>>> Haven't managed to crash it yet, but every run complains:
>>>>>>>>>
>>>>>>>>> got to shutdown of 10 with refcnt: 2
>>>>>>>>> Refs being all dropped, calling forget for 10
>>>>>>>>> terminate called after throwing an instance of 'fmt::v6::format_error'
>>>>>>>>>   what():  argument index out of range
>>>>>>>>> unknown location(0): fatal error: in "unixdomain_server": signal: SIGABRT (application abort requested)
>>>>>>>>>
>>>>>>>>> Not sure if that's causing it not to fail here.
>>>>>>>>
>>>>>>>> Ok, that means it "passed". (I was in the process of figuring out
>>>>>>>> where I got this wrong when I started seeing the crashes)
>>>>>>>
>>>>>>> Can you do, in your kernel dir:
>>>>>>>
>>>>>>> $ gdb vmlinux
>>>>>>> [...]
>>>>>>> (gdb) l *__io_queue_sqe+0x4a
>>>>>>>
>>>>>>> and see what it says?
>>>>>>
>>>>>> 0xffffffff81375ada is in __io_queue_sqe (fs/io_uring.c:4814).
>>>>>> 4809 struct io_kiocb *linked_timeout;
>>>>>> 4810 struct io_kiocb *nxt = NULL;
>>>>>> 4811 int ret;
>>>>>> 4812
>>>>>> 4813 again:
>>>>>> 4814 linked_timeout = io_prep_linked_timeout(req);
>>>>>> 4815
>>>>>> 4816 ret = io_issue_sqe(req, sqe, &nxt, true);
>>>>>> 4817
>>>>>> 4818 /*
>>>>>>
>>>>>> (I am not using timeouts, just async_cancel)
>>>>>
>>>>> Can't seem to hit it here, went through thousands of iterations...
>>>>> I'll keep trying.
>>>>>
>>>>> If you have time, you can try and enable CONFIG_KASAN=y and see if
>>>>> you can hit it with that.
>>>>
>>>> I can
>>>>
>>>> Attaching full dmesg
>>>
>>> Can you try the latest? It's sha d8154e605f84.
> 
> 10 runs, no crashes.
> 
> Thanks!

Great! Thanks for reporting and the quick testing.

-- 
Jens Axboe