Re: crash on accept

Re: crash on accept

[Jens Axboe]

On 2/19/20 9:23 AM, Glauber Costa wrote:
> Hi,
> 
> I started using af0a72622a1fb7179cf86ae714d52abadf7d8635 today so I could consume the new fast poll flag, and one of my tests that was previously passing now crashes

Thanks for testing the new stuff! As always, would really appreciate a
test case that I can run, makes my job so much easier.

-- 
Jens Axboe

Re: crash on accept

[Glauber Costa]

On Wed, Feb 19, 2020 at 3:09 PM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 2/19/20 9:23 AM, Glauber Costa wrote:
> > Hi,
> >
> > I started using af0a72622a1fb7179cf86ae714d52abadf7d8635 today so I could consume the new fast poll flag, and one of my tests that was previously passing now crashes
>
> Thanks for testing the new stuff! As always, would really appreciate a
> test case that I can run, makes my job so much easier.

Trigger warning:
It's in C++.

I am finishing refactoring some of my code now. It's nothing
substantial so I am positive it will hit again. Once I re-reproduce
I'll send you instructions.

Reading the code it's not obvious to me how it happens, so it'll be
harder for me to cook up a simple C reproducer ATM.


>
> --
> Jens Axboe
>

Re: crash on accept

[Jens Axboe]

On 2/19/20 1:11 PM, Glauber Costa wrote:
> On Wed, Feb 19, 2020 at 3:09 PM Jens Axboe <axboe@kernel.dk> wrote:
>>
>> On 2/19/20 9:23 AM, Glauber Costa wrote:
>>> Hi,
>>>
>>> I started using af0a72622a1fb7179cf86ae714d52abadf7d8635 today so I could consume the new fast poll flag, and one of my tests that was previously passing now crashes
>>
>> Thanks for testing the new stuff! As always, would really appreciate a
>> test case that I can run, makes my job so much easier.
> 
> Trigger warning:
> It's in C++.

As long as it reproduces, I don't really have to look at it :-)

> I am finishing refactoring some of my code now. It's nothing
> substantial so I am positive it will hit again. Once I re-reproduce
> I'll send you instructions.
> 
> Reading the code it's not obvious to me how it happens, so it'll be
> harder for me to cook up a simple C reproducer ATM.

I'll look here as well, as time permits.


-- 
Jens Axboe

Re: crash on accept

[Glauber Costa]

On Wed, Feb 19, 2020 at 3:13 PM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 2/19/20 1:11 PM, Glauber Costa wrote:
> > On Wed, Feb 19, 2020 at 3:09 PM Jens Axboe <axboe@kernel.dk> wrote:
> >>
> >> On 2/19/20 9:23 AM, Glauber Costa wrote:
> >>> Hi,
> >>>
> >>> I started using af0a72622a1fb7179cf86ae714d52abadf7d8635 today so I could consume the new fast poll flag, and one of my tests that was previously passing now crashes
> >>
> >> Thanks for testing the new stuff! As always, would really appreciate a
> >> test case that I can run, makes my job so much easier.
> >
> > Trigger warning:
> > It's in C++.
>
> As long as it reproduces, I don't really have to look at it :-)

Instructions:
1. clone https://github.com/glommer/seastar.git, branch uring-accept-crash
2. git submodule update --recursive --init, because we have a shit-ton
of submodules because why not.
3. install all dependencies with ./install-dependencies.sh
    note: that does not install liburing yet, you need to have at
least 0.4 (I trust you do), with the patch I just sent to add the fast
poll flag. It still fails sometimes in my system if liburing is
installed in /usr/lib instead of /usr/lib64 because cmake is made by
the devil.
3. ./configure.py --mode=release
4. ninja -C build/release tests/unit/unix_domain_test
5. crash your system (hopefully) by executing
./build/release/tests/unit/unix_domain_test -- -c1
--reactor-backend=uring


>
> > I am finishing refactoring some of my code now. It's nothing
> > substantial so I am positive it will hit again. Once I re-reproduce
> > I'll send you instructions.
> >
> > Reading the code it's not obvious to me how it happens, so it'll be
> > harder for me to cook up a simple C reproducer ATM.
>
> I'll look here as well, as time permits.
>
>
> --
> Jens Axboe
>

Re: crash on accept

[Avi Kivity]

On 2/19/20 10:25 PM, Glauber Costa wrote:
> On Wed, Feb 19, 2020 at 3:13 PM Jens Axboe <axboe@kernel.dk> wrote:
>> On 2/19/20 1:11 PM, Glauber Costa wrote:
>>> On Wed, Feb 19, 2020 at 3:09 PM Jens Axboe <axboe@kernel.dk> wrote:
>>>> On 2/19/20 9:23 AM, Glauber Costa wrote:
>>>>> Hi,
>>>>>
>>>>> I started using af0a72622a1fb7179cf86ae714d52abadf7d8635 today so I could consume the new fast poll flag, and one of my tests that was previously passing now crashes
>>>> Thanks for testing the new stuff! As always, would really appreciate a
>>>> test case that I can run, makes my job so much easier.
>>> Trigger warning:
>>> It's in C++.
>> As long as it reproduces, I don't really have to look at it :-)
> Instructions:
> 1. clone https://github.com/glommer/seastar.git, branch uring-accept-crash
> 2. git submodule update --recursive --init, because we have a shit-ton
> of submodules because why not.


Actually, seastar has only one submodule (dpdk) and it is optional, so 
you need not clone it.


> 3. install all dependencies with ./install-dependencies.sh
>      note: that does not install liburing yet, you need to have at
> least 0.4 (I trust you do), with the patch I just sent to add the fast
> poll flag. It still fails sometimes in my system if liburing is
> installed in /usr/lib instead of /usr/lib64 because cmake is made by
> the devil.
> 3. ./configure.py --mode=release


--mode dev will compile many times faster


> 4. ninja -C build/release tests/unit/unix_domain_test
> 5. crash your system (hopefully) by executing
> ./build/release/tests/unit/unix_domain_test -- -c1
> --reactor-backend=uring
>
s/release/dev/ in steps 4, 5 if you use dev mode.


>>> I am finishing refactoring some of my code now. It's nothing
>>> substantial so I am positive it will hit again. Once I re-reproduce
>>> I'll send you instructions.
>>>
>>> Reading the code it's not obvious to me how it happens, so it'll be
>>> harder for me to cook up a simple C reproducer ATM.
>> I'll look here as well, as time permits.
>>
>>
>> --
>> Jens Axboe
>>

Re: crash on accept

[Jens Axboe]

On 2/19/20 1:29 PM, Avi Kivity wrote:
> On 2/19/20 10:25 PM, Glauber Costa wrote:
>> On Wed, Feb 19, 2020 at 3:13 PM Jens Axboe <axboe@kernel.dk> wrote:
>>> On 2/19/20 1:11 PM, Glauber Costa wrote:
>>>> On Wed, Feb 19, 2020 at 3:09 PM Jens Axboe <axboe@kernel.dk> wrote:
>>>>> On 2/19/20 9:23 AM, Glauber Costa wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I started using af0a72622a1fb7179cf86ae714d52abadf7d8635 today so I could consume the new fast poll flag, and one of my tests that was previously passing now crashes
>>>>> Thanks for testing the new stuff! As always, would really appreciate a
>>>>> test case that I can run, makes my job so much easier.
>>>> Trigger warning:
>>>> It's in C++.
>>> As long as it reproduces, I don't really have to look at it :-)
>> Instructions:
>> 1. clone https://github.com/glommer/seastar.git, branch uring-accept-crash
>> 2. git submodule update --recursive --init, because we have a shit-ton
>> of submodules because why not.
> 
> 
> Actually, seastar has only one submodule (dpdk) and it is optional, so 
> you need not clone it.
> 
> 
>> 3. install all dependencies with ./install-dependencies.sh
>>      note: that does not install liburing yet, you need to have at
>> least 0.4 (I trust you do), with the patch I just sent to add the fast
>> poll flag. It still fails sometimes in my system if liburing is
>> installed in /usr/lib instead of /usr/lib64 because cmake is made by
>> the devil.
>> 3. ./configure.py --mode=release
> 
> 
> --mode dev will compile many times faster
> 
> 
>> 4. ninja -C build/release tests/unit/unix_domain_test
>> 5. crash your system (hopefully) by executing
>> ./build/release/tests/unit/unix_domain_test -- -c1
>> --reactor-backend=uring
>>
> s/release/dev/ in steps 4, 5 if you use dev mode.

Thanks, this is great, I can reproduce!


-- 
Jens Axboe

Re: crash on accept

[Jens Axboe]

On 2/19/20 4:09 PM, Jens Axboe wrote:
> On 2/19/20 1:29 PM, Avi Kivity wrote:
>> On 2/19/20 10:25 PM, Glauber Costa wrote:
>>> On Wed, Feb 19, 2020 at 3:13 PM Jens Axboe <axboe@kernel.dk> wrote:
>>>> On 2/19/20 1:11 PM, Glauber Costa wrote:
>>>>> On Wed, Feb 19, 2020 at 3:09 PM Jens Axboe <axboe@kernel.dk> wrote:
>>>>>> On 2/19/20 9:23 AM, Glauber Costa wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I started using af0a72622a1fb7179cf86ae714d52abadf7d8635 today so I could consume the new fast poll flag, and one of my tests that was previously passing now crashes
>>>>>> Thanks for testing the new stuff! As always, would really appreciate a
>>>>>> test case that I can run, makes my job so much easier.
>>>>> Trigger warning:
>>>>> It's in C++.
>>>> As long as it reproduces, I don't really have to look at it :-)
>>> Instructions:
>>> 1. clone https://github.com/glommer/seastar.git, branch uring-accept-crash
>>> 2. git submodule update --recursive --init, because we have a shit-ton
>>> of submodules because why not.
>>
>>
>> Actually, seastar has only one submodule (dpdk) and it is optional, so 
>> you need not clone it.
>>
>>
>>> 3. install all dependencies with ./install-dependencies.sh
>>>      note: that does not install liburing yet, you need to have at
>>> least 0.4 (I trust you do), with the patch I just sent to add the fast
>>> poll flag. It still fails sometimes in my system if liburing is
>>> installed in /usr/lib instead of /usr/lib64 because cmake is made by
>>> the devil.
>>> 3. ./configure.py --mode=release
>>
>>
>> --mode dev will compile many times faster
>>
>>
>>> 4. ninja -C build/release tests/unit/unix_domain_test
>>> 5. crash your system (hopefully) by executing
>>> ./build/release/tests/unit/unix_domain_test -- -c1
>>> --reactor-backend=uring
>>>
>> s/release/dev/ in steps 4, 5 if you use dev mode.
> 
> Thanks, this is great, I can reproduce!

Can you try the current branch? Should be 77aac7e7738 (or newer).

-- 
Jens Axboe

Re: crash on accept

[Glauber Costa]

I don't see a crash now, thanks.

I can now go back to trying to figure out why the test is just hanging
forever, as I was doing earlier =)
(99.9% I broke something with the last rework).

Out of curiosity, as it may help me with the above: I notice you
didn't add a patch on top, but rather rebased the tree.

What was the problem leading to the crash ?

On Wed, Feb 19, 2020 at 8:37 PM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 2/19/20 4:09 PM, Jens Axboe wrote:
> > On 2/19/20 1:29 PM, Avi Kivity wrote:
> >> On 2/19/20 10:25 PM, Glauber Costa wrote:
> >>> On Wed, Feb 19, 2020 at 3:13 PM Jens Axboe <axboe@kernel.dk> wrote:
> >>>> On 2/19/20 1:11 PM, Glauber Costa wrote:
> >>>>> On Wed, Feb 19, 2020 at 3:09 PM Jens Axboe <axboe@kernel.dk> wrote:
> >>>>>> On 2/19/20 9:23 AM, Glauber Costa wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I started using af0a72622a1fb7179cf86ae714d52abadf7d8635 today so I could consume the new fast poll flag, and one of my tests that was previously passing now crashes
> >>>>>> Thanks for testing the new stuff! As always, would really appreciate a
> >>>>>> test case that I can run, makes my job so much easier.
> >>>>> Trigger warning:
> >>>>> It's in C++.
> >>>> As long as it reproduces, I don't really have to look at it :-)
> >>> Instructions:
> >>> 1. clone https://github.com/glommer/seastar.git, branch uring-accept-crash
> >>> 2. git submodule update --recursive --init, because we have a shit-ton
> >>> of submodules because why not.
> >>
> >>
> >> Actually, seastar has only one submodule (dpdk) and it is optional, so
> >> you need not clone it.
> >>
> >>
> >>> 3. install all dependencies with ./install-dependencies.sh
> >>>      note: that does not install liburing yet, you need to have at
> >>> least 0.4 (I trust you do), with the patch I just sent to add the fast
> >>> poll flag. It still fails sometimes in my system if liburing is
> >>> installed in /usr/lib instead of /usr/lib64 because cmake is made by
> >>> the devil.
> >>> 3. ./configure.py --mode=release
> >>
> >>
> >> --mode dev will compile many times faster
> >>
> >>
> >>> 4. ninja -C build/release tests/unit/unix_domain_test
> >>> 5. crash your system (hopefully) by executing
> >>> ./build/release/tests/unit/unix_domain_test -- -c1
> >>> --reactor-backend=uring
> >>>
> >> s/release/dev/ in steps 4, 5 if you use dev mode.
> >
> > Thanks, this is great, I can reproduce!
>
> Can you try the current branch? Should be 77aac7e7738 (or newer).
>
> --
> Jens Axboe
>

Re: crash on accept

[Jens Axboe]

On 2/19/20 7:52 PM, Glauber Costa wrote:
> I don't see a crash now, thanks.

Updated again, should be more solid now. Just in case you run into
issues.

> I can now go back to trying to figure out why the test is just hanging
> forever, as I was doing earlier =)
> (99.9% I broke something with the last rework).
> 
> Out of curiosity, as it may help me with the above: I notice you
> didn't add a patch on top, but rather rebased the tree.

Yeah, the poll based async bits are very much a work in progress,
and I'll just fold fixes for now.

> What was the problem leading to the crash ?

It had to do with repeated retry. Eg we want to read from something,
we try and get -EAGAIN. Arm the poll handler, poll results says we're
good to go. Retry, get -EAGAIN again. Now we give up, but the state
wasn't restored properly.

-- 
Jens Axboe