Re: [PATCH liburing 3/4] tests: add tests for zerocopy send and notifications

[Eli Schwartz <eschwartz93@gmail.com>]

On 7/25/22 8:08 AM, Ammar Faizi wrote:
> On 7/25/22 6:28 PM, Pavel Begunkov wrote:
>> By the way, while we're at it, what is T_EXIT_ERROR? Why it's not used
>> anywhere
>> and how it's different from T_EXIT_FAIL?
> 
> [ Adding Eli to the participants. ]
> 
> Ummm... yeah. I am curious about it too now. I just took a look at commit:
> 
>    ed430fbeb33367 ("tests: migrate some tests to use enum-based exit
> codes").
> 
> Eli said:
> 
>     From: Eli Schwartz <eschwartz93@gmail.com>
>     Date: Mon, 27 Jun 2022 14:39:05 -0400
>     Subject: [PATCH] tests: migrate some tests to use enum-based exit codes
> 
>     For maintainability and clarity, eschew the use of integer literals in
>     reporting test statuses. Instead, use a helper enum which contains
>     various values from the GNU exitcode protocol. Returning 0 or 1 is
>     obvious, and in the previous commit the ability to read "skip" (77) was
>     implemented. The final exit status is 99, which indicates some kind of
>     error in running the test itself.
> 
>     A partial migration of existing pass/fail values in test sources is
>     included.
> 
>     Signed-off-by: Eli Schwartz <eschwartz93@gmail.com>
> 
> 
> That T_EXIT_ERROR is 99 here. Not sure when to use it in liburing test.
> Eli?


A test failure means that the test ran "successfully", and reported that
something is wrong with the software being tested. It is collected as a
test result which needs to be investigated to fix the software, but only
usually. It's also possible for some test harnesses to have a list of
tests where you know you have a bug, you know it's not fixed, and you
don't want to be constantly bothered by it, so you mark it as
"should_fail: true" (Meson) or XFAIL_TESTS= (GNU autotools) so that the
test report harness doesn't list it either as a "pass" or a "fail".
Instead it's listed as "expected/known fail" and the test harness itself
doesn't count it as an overall failure.

(If that test starts to pass, it's reported as an "unexpected pass" and
the test harness itself counts it as an overall failure. The idea is
that you then want to do your bookkeeping and record that whatever
commit caused the unexpected pass, has fixed a longstanding bug.)

...


T_EXIT_ERROR is different. It doesn't mean the test ran, and reported
something wrong with the software (e.g. liburing). Instead, an ERROR
return value indicates that the test itself broke and cannot even be
relied on to accurately test for a bug/regression. For example, if that
test was designated as an expected failure, it still knows that in this
case, error != fail, and it won't ignore the result as an expected failure.

Also in general, if you see test errors you know to look at bugs in the
testsuite instead of trying to debug the software. :)

I added T_EXIT_ERROR because it may be useful, without knowing in
advance whether I would have cause to use it anywhere. It's a valid
possible state.


To quote
https://www.gnu.org/software/automake/manual/html_node/Generalities-about-Testing.html


"""
Many testing environments and frameworks distinguish between test
failures and hard errors. As we’ve seen, a test failure happens when
some invariant or expected behaviour of the software under test is not
met. A hard error happens when e.g., the set-up of a test case scenario
fails, or when some other unexpected or highly undesirable condition is
encountered (for example, the program under test experiences a
segmentation fault).
"""

-- 
Eli Schwartz