[PATCH v1 0/1] Fix a stack misalign bug on _start

[PATCH v1 0/1] Fix a stack misalign bug on _start

[Ammar Faizi]

Hi Willy,

Just a single quick fix.

The ABI mandates that the %esp register must be a multiple of 16 when
executing a call instruction.

Commit 2ab446336b17 simplified the _start function, but it didn't take
care of the %esp alignment, causing SIGSEGV on SSE and AVX programs that
use aligned move instruction (e.g., movdqa, movaps, and vmovdqa).

  $eax   : 0x56559000  →  0x00003f90
  $ebx   : 0x56559000  →  0x00003f90
  $ecx   : 0x1
  $edx   : 0xf7fcaaa0  →   endbr32 
  $esp   : 0xffffcdbc  →  0x00000001
  $ebp   : 0x0
  $esi   : 0xffffce7c  →  0xffffd096
  $edi   : 0x56556060  →  <_start+0> xor %ebp, %ebp
  $eip   : 0x56556489  →  <sse_pq_add+25> movaps %xmm0, 0x30(%esp)

    <sse_pq_add+11>  pop    %eax
    <sse_pq_add+12>  add    $0x2b85, %eax
    <sse_pq_add+18>  movups -0x1fd0(%eax), %xmm0
  → <sse_pq_add+25>  movaps %xmm0, 0x30(%esp)     <== trapping instruction
    <sse_pq_add+30>  movups -0x1fe0(%eax), %xmm1
    <sse_pq_add+37>  movaps %xmm1, 0x20(%esp)
    <sse_pq_add+42>  movups -0x1ff0(%eax), %xmm2
    <sse_pq_add+49>  movaps %xmm2, 0x10(%esp)
    <sse_pq_add+54>  movups -0x2000(%eax), %xmm3

  [#0] Id 1, Name: "test", stopped 0x56556489 in sse_pq_add (), reason: SIGSEGV

  (gdb)  bt
  #0  0x56556489 in sse_pq_add ()
  #1  0x5655608e in main ()

Ensure the %esp is a multiple of 16 when executing the call instruction.

Signed-off-by: Ammar Faizi <ammarfaizi2@gnuweeb.org>
---
Ammar Faizi (1):
  tools/nolibc: i386: Fix a stack misalign bug on _start

 tools/include/nolibc/arch-i386.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)


base-commit: 6269320850097903b30be8f07a5c61d9f7592393
-- 
Ammar Faizi

[PATCH v1 1/1] tools/nolibc: i386: Fix a stack misalign bug on _start

[Ammar Faizi]

The ABI mandates that the %esp register must be a multiple of 16 when
executing a call instruction.

Commit 2ab446336b17 simplified the _start function, but it didn't take
care of the %esp alignment, causing SIGSEGV on SSE and AVX programs that
use aligned move instruction (e.g., movdqa, movaps, and vmovdqa).

Ensure the %esp is a multiple of 16 when executing the call instruction.

Cc: Zhangjin Wu <falcon@tinylab.org>
Fixes: 2ab446336b17aad362c6decee29b4efd83a01979 ("tools/nolibc: i386: shrink _start with _start_c")
Reported-by: Nicholas Rosenberg <inori@vnlx.org>
Signed-off-by: Ammar Faizi <ammarfaizi2@gnuweeb.org>
---
 tools/include/nolibc/arch-i386.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/include/nolibc/arch-i386.h b/tools/include/nolibc/arch-i386.h
index 64415b9fac77f996..4edf238eeac67aa2 100644
--- a/tools/include/nolibc/arch-i386.h
+++ b/tools/include/nolibc/arch-i386.h
@@ -167,7 +167,8 @@ void __attribute__((weak, noreturn, optimize("Os", "omit-frame-pointer"))) __no_
 	__asm__ volatile (
 		"xor  %ebp, %ebp\n"       /* zero the stack frame                                */
 		"mov  %esp, %eax\n"       /* save stack pointer to %eax, as arg1 of _start_c     */
-		"and  $-16, %esp\n"       /* last pushed argument must be 16-byte aligned        */
+		"and  $-16, %esp\n"       /* align stack to 16 bytes                             */
+		"sub  $12, %esp\n"        /* last pushed argument must be 16-byte aligned        */
 		"push %eax\n"             /* push arg1 on stack to support plain stack modes too */
 		"call _start_c\n"         /* transfer to c runtime                               */
 		"hlt\n"                   /* ensure it does not return                           */
-- 
Ammar Faizi

Re: [PATCH v1 1/1] tools/nolibc: i386: Fix a stack misalign bug on _start

[Zhangjin Wu]

Hi, Ammar, Hi, Nicholas

Thanks very much for your report and fixup, sorry for the missing alignment for
the new single one push instruction.

> The ABI mandates that the %esp register must be a multiple of 16 when
> executing a call instruction.
> 
> Commit 2ab446336b17 simplified the _start function, but it didn't take
> care of the %esp alignment, causing SIGSEGV on SSE and AVX programs that
> use aligned move instruction (e.g., movdqa, movaps, and vmovdqa).
> 
> Ensure the %esp is a multiple of 16 when executing the call instruction.
> 
> Cc: Zhangjin Wu <falcon@tinylab.org>
> Fixes: 2ab446336b17aad362c6decee29b4efd83a01979 ("tools/nolibc: i386: shrink _start with _start_c")
> Reported-by: Nicholas Rosenberg <inori@vnlx.org>
> Signed-off-by: Ammar Faizi <ammarfaizi2@gnuweeb.org>
> ---
>  tools/include/nolibc/arch-i386.h | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/include/nolibc/arch-i386.h b/tools/include/nolibc/arch-i386.h
> index 64415b9fac77f996..4edf238eeac67aa2 100644
> --- a/tools/include/nolibc/arch-i386.h
> +++ b/tools/include/nolibc/arch-i386.h
> @@ -167,7 +167,8 @@ void __attribute__((weak, noreturn, optimize("Os", "omit-frame-pointer"))) __no_
>  	__asm__ volatile (
>  		"xor  %ebp, %ebp\n"       /* zero the stack frame                                */
>  		"mov  %esp, %eax\n"       /* save stack pointer to %eax, as arg1 of _start_c     */
> -		"and  $-16, %esp\n"       /* last pushed argument must be 16-byte aligned        */
> +		"and  $-16, %esp\n"       /* align stack to 16 bytes                             */
> +		"sub  $12, %esp\n"        /* last pushed argument must be 16-byte aligned        */

Ammar, the new call of _start_c() only requires a single push, it pushes
the argument and minus %esp by 4, so, the alignment of %esp requires to
minus 12 to reserve 16-byte alignment, is this description right?

If so, What about further?

    "and  $-16, %esp\n"       /* align stack to 16 bytes                             */
    "sub ($16 - $4), %esp"    /* the 'push %eax' breaks stack alignment, fix up it   */

Thanks,
Zhangjin Wu

>  		"push %eax\n"             /* push arg1 on stack to support plain stack modes too */
>  		"call _start_c\n"         /* transfer to c runtime                               */
>  		"hlt\n"                   /* ensure it does not return                           */
> -- 
> Ammar Faizi

Re: [PATCH v1 0/1] Fix a stack misalign bug on _start

[Zhangjin Wu]

Hi, Ammar

> Hi Willy,
> 
> Just a single quick fix.
> 
> The ABI mandates that the %esp register must be a multiple of 16 when
> executing a call instruction.
>
> Commit 2ab446336b17 simplified the _start function, but it didn't take
> care of the %esp alignment, causing SIGSEGV on SSE and AVX programs that
> use aligned move instruction (e.g., movdqa, movaps, and vmovdqa).
>

Yeah, I have learned carefully about the old 'sub $4, %esp' instruction
for the old 3 'push' instructions, but at last forgot to add a new
instruction for the new single 'push' instruction to reserve the
16-byte alignment, very sorry for this bad regression. 

>   $eax   : 0x56559000  →  0x00003f90
>   $ebx   : 0x56559000  →  0x00003f90
>   $ecx   : 0x1
>   $edx   : 0xf7fcaaa0  →   endbr32 
>   $esp   : 0xffffcdbc  →  0x00000001
>   $ebp   : 0x0
>   $esi   : 0xffffce7c  →  0xffffd096
>   $edi   : 0x56556060  →  <_start+0> xor %ebp, %ebp
>   $eip   : 0x56556489  →  <sse_pq_add+25> movaps %xmm0, 0x30(%esp)
> 
>     <sse_pq_add+11>  pop    %eax
>     <sse_pq_add+12>  add    $0x2b85, %eax
>     <sse_pq_add+18>  movups -0x1fd0(%eax), %xmm0
>   → <sse_pq_add+25>  movaps %xmm0, 0x30(%esp)     <== trapping instruction
>     <sse_pq_add+30>  movups -0x1fe0(%eax), %xmm1
>     <sse_pq_add+37>  movaps %xmm1, 0x20(%esp)
>     <sse_pq_add+42>  movups -0x1ff0(%eax), %xmm2
>     <sse_pq_add+49>  movaps %xmm2, 0x10(%esp)
>     <sse_pq_add+54>  movups -0x2000(%eax), %xmm3
> 
>   [#0] Id 1, Name: "test", stopped 0x56556489 in sse_pq_add (), reason: SIGSEGV
> 
>   (gdb)  bt
>   #0  0x56556489 in sse_pq_add ()
>   #1  0x5655608e in main ()
>

Since we have a new 'startup' test group, do you have a short function
to trigger this error?

Perhaps it is time for us to add a new 'stack alignment' test case for
all of the architectures.

Thanks,
Zhangjin

> Ensure the %esp is a multiple of 16 when executing the call instruction.
> 
> Signed-off-by: Ammar Faizi <ammarfaizi2@gnuweeb.org>
> ---
> Ammar Faizi (1):
>   tools/nolibc: i386: Fix a stack misalign bug on _start
> 
>  tools/include/nolibc/arch-i386.h | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> 
> base-commit: 6269320850097903b30be8f07a5c61d9f7592393
> -- 
> Ammar Faizi

Re: [PATCH v1 1/1] tools/nolibc: i386: Fix a stack misalign bug on _start

[Ammar Faizi]

On Sat, Aug 26, 2023 at 11:07:04PM +0800, Zhangjin Wu wrote:
> > @@ -167,7 +167,8 @@ void __attribute__((weak, noreturn, optimize("Os", "omit-frame-pointer"))) __no_
> >  	__asm__ volatile (
> >  		"xor  %ebp, %ebp\n"       /* zero the stack frame                                */
> >  		"mov  %esp, %eax\n"       /* save stack pointer to %eax, as arg1 of _start_c     */
> > -		"and  $-16, %esp\n"       /* last pushed argument must be 16-byte aligned        */
> > +		"and  $-16, %esp\n"       /* align stack to 16 bytes                             */
> > +		"sub  $12, %esp\n"        /* last pushed argument must be 16-byte aligned        */
> 
> Ammar, the new call of _start_c() only requires a single push, it pushes
> the argument and minus %esp by 4, so, the alignment of %esp requires to
> minus 12 to reserve 16-byte alignment, is this description right?

Yes, that's correct.

> If so, What about further?
> 
>     "and  $-16, %esp\n"       /* align stack to 16 bytes                             */
>     "sub ($16 - $4), %esp"    /* the 'push %eax' breaks stack alignment, fix up it   */

The sub part should have been:

    "sub  $(16 - 4), %esp\n"

It's fine to me writing it that way too.

-- 
Ammar Faizi

Re: [PATCH v1 0/1] Fix a stack misalign bug on _start

[Ammar Faizi]

On Sat, Aug 26, 2023 at 11:20:24PM +0800, Zhangjin Wu wrote:
> >   $eax   : 0x56559000  →  0x00003f90
> >   $ebx   : 0x56559000  →  0x00003f90
> >   $ecx   : 0x1
> >   $edx   : 0xf7fcaaa0  →   endbr32 
> >   $esp   : 0xffffcdbc  →  0x00000001
> >   $ebp   : 0x0
> >   $esi   : 0xffffce7c  →  0xffffd096
> >   $edi   : 0x56556060  →  <_start+0> xor %ebp, %ebp
> >   $eip   : 0x56556489  →  <sse_pq_add+25> movaps %xmm0, 0x30(%esp)
> > 
> >     <sse_pq_add+11>  pop    %eax
> >     <sse_pq_add+12>  add    $0x2b85, %eax
> >     <sse_pq_add+18>  movups -0x1fd0(%eax), %xmm0
> >   → <sse_pq_add+25>  movaps %xmm0, 0x30(%esp)     <== trapping instruction
> >     <sse_pq_add+30>  movups -0x1fe0(%eax), %xmm1
> >     <sse_pq_add+37>  movaps %xmm1, 0x20(%esp)
> >     <sse_pq_add+42>  movups -0x1ff0(%eax), %xmm2
> >     <sse_pq_add+49>  movaps %xmm2, 0x10(%esp)
> >     <sse_pq_add+54>  movups -0x2000(%eax), %xmm3
> > 
> >   [#0] Id 1, Name: "test", stopped 0x56556489 in sse_pq_add (), reason: SIGSEGV
> > 
> >   (gdb)  bt
> >   #0  0x56556489 in sse_pq_add ()
> >   #1  0x5655608e in main ()
> >
> 
> Since we have a new 'startup' test group, do you have a short function
> to trigger this error?

Here is a simple program to test the stack alignment.

#include "tools/include/nolibc/nolibc.h"

__asm__ (
"main:\n"
    /*
     * When the call main is executed, the
     * %esp is 16 bytes aligned.
     *
     * Then, on function entry (%esp mod 16) == 12
     * because the call instruction pushes 4 bytes
     * onto the stack.
     *
     * subl $12, %esp will make (%esp mod 16) == 0
     * again.
     */
    "subl  $12, %esp\n"

    /*
     * These move instructions will crash if %esp is
     * not a multiple of 16.
     */
    "movdqa (%esp), %xmm0\n"
    "movdqa %xmm0, (%esp)\n"
    "movaps (%esp), %xmm0\n"
    "movaps %xmm0, (%esp)\n"

    "addl   $12, %esp\n"
    "xorl   %eax, %eax\n"
    "ret\n"
);

> Perhaps it is time for us to add a new 'stack alignment' test case for
> all of the architectures.

I don't know the alignment rules for other architectures (I only work on
x86 and x86-64). While waiting for the maintainers' comment, I'll leave
the test case decision to you. Feel free to take the above code.

Extra:
It's also fine if you take my patch with the 'sub $(16 - 4), %esp'
change and batch it together in your next series.

-- 
Ammar Faizi

Re: [PATCH v1 0/1] Fix a stack misalign bug on _start

[Zhangjin Wu]

Hi, Ammar

> On Sat, Aug 26, 2023 at 11:20:24PM +0800, Zhangjin Wu wrote:
> > >   $eax   : 0x56559000  →  0x00003f90
> > >   $ebx   : 0x56559000  →  0x00003f90
> > >   $ecx   : 0x1
> > >   $edx   : 0xf7fcaaa0  →   endbr32 
> > >   $esp   : 0xffffcdbc  →  0x00000001
> > >   $ebp   : 0x0
> > >   $esi   : 0xffffce7c  →  0xffffd096
> > >   $edi   : 0x56556060  →  <_start+0> xor %ebp, %ebp
> > >   $eip   : 0x56556489  →  <sse_pq_add+25> movaps %xmm0, 0x30(%esp)
> > > 
> > >     <sse_pq_add+11>  pop    %eax
> > >     <sse_pq_add+12>  add    $0x2b85, %eax
> > >     <sse_pq_add+18>  movups -0x1fd0(%eax), %xmm0
> > >   → <sse_pq_add+25>  movaps %xmm0, 0x30(%esp)     <== trapping instruction
> > >     <sse_pq_add+30>  movups -0x1fe0(%eax), %xmm1
> > >     <sse_pq_add+37>  movaps %xmm1, 0x20(%esp)
> > >     <sse_pq_add+42>  movups -0x1ff0(%eax), %xmm2
> > >     <sse_pq_add+49>  movaps %xmm2, 0x10(%esp)
> > >     <sse_pq_add+54>  movups -0x2000(%eax), %xmm3
> > > 
> > >   [#0] Id 1, Name: "test", stopped 0x56556489 in sse_pq_add (), reason: SIGSEGV
> > > 
> > >   (gdb)  bt
> > >   #0  0x56556489 in sse_pq_add ()
> > >   #1  0x5655608e in main ()
> > >
> > 
> > Since we have a new 'startup' test group, do you have a short function
> > to trigger this error?
> 
> Here is a simple program to test the stack alignment.
> 
> #include "tools/include/nolibc/nolibc.h"
> 
> __asm__ (
> "main:\n"
>     /*
>      * When the call main is executed, the
>      * %esp is 16 bytes aligned.
>      *
>      * Then, on function entry (%esp mod 16) == 12
>      * because the call instruction pushes 4 bytes
>      * onto the stack.
>      *
>      * subl $12, %esp will make (%esp mod 16) == 0
>      * again.
>      */
>     "subl  $12, %esp\n"
> 
>     /*
>      * These move instructions will crash if %esp is
>      * not a multiple of 16.
>      */
>     "movdqa (%esp), %xmm0\n"
>     "movdqa %xmm0, (%esp)\n"
>     "movaps (%esp), %xmm0\n"
>     "movaps %xmm0, (%esp)\n"
> 
>     "addl   $12, %esp\n"
>     "xorl   %eax, %eax\n"
>     "ret\n"
> );
>

Thanks very much for sharing this code.

> > Perhaps it is time for us to add a new 'stack alignment' test case for
> > all of the architectures.
> 
> I don't know the alignment rules for other architectures (I only work on
> x86 and x86-64). While waiting for the maintainers' comment, I'll leave
> the test case decision to you. Feel free to take the above code.
>

Yes, the stack alignment rule is architecture dependent, so, we need
more discussion and more work, not sure if there is a 'C' test function
for all, let's delay this after v6.6.

> Extra:
> It's also fine if you take my patch with the 'sub $(16 - 4), %esp'
> change and batch it together in your next series.
>

Ammar, your fixup patch is urgent since our _start_c() is for v6.6-rc1 (already
in linux-next), let's wait for comments from Thomas or Willy, they will
determine that merge it directly or require a v2. I'm ok with v1 code, but the
old comment looks not that clear.

Thanks,
Zhangjin

> -- 
> Ammar Faizi

Re: [PATCH v1 1/1] tools/nolibc: i386: Fix a stack misalign bug on _start

[Thomas Weißschuh]

Hi Ammar,

On 2023-08-26 21:16:32+0700, Ammar Faizi wrote:
> The ABI mandates that the %esp register must be a multiple of 16 when
> executing a call instruction.
> 
> Commit 2ab446336b17 simplified the _start function, but it didn't take

Afaik references to commits in commit messages are supposed to follow
the same format as in the Fixes: tag.

"Commit 2ab446336b17 ("tools/nolibc: i386: shrink _start with _start_c")
simplified the _start function, [...]"

> care of the %esp alignment, causing SIGSEGV on SSE and AVX programs that
> use aligned move instruction (e.g., movdqa, movaps, and vmovdqa).
> 
> Ensure the %esp is a multiple of 16 when executing the call instruction.
> 
> Cc: Zhangjin Wu <falcon@tinylab.org>
> Fixes: 2ab446336b17aad362c6decee29b4efd83a01979 ("tools/nolibc: i386: shrink _start with _start_c")
> Reported-by: Nicholas Rosenberg <inori@vnlx.org>
> Signed-off-by: Ammar Faizi <ammarfaizi2@gnuweeb.org>
> ---
>  tools/include/nolibc/arch-i386.h | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/include/nolibc/arch-i386.h b/tools/include/nolibc/arch-i386.h
> index 64415b9fac77f996..4edf238eeac67aa2 100644
> --- a/tools/include/nolibc/arch-i386.h
> +++ b/tools/include/nolibc/arch-i386.h
> @@ -167,7 +167,8 @@ void __attribute__((weak, noreturn, optimize("Os", "omit-frame-pointer"))) __no_
>  	__asm__ volatile (
>  		"xor  %ebp, %ebp\n"       /* zero the stack frame                                */
>  		"mov  %esp, %eax\n"       /* save stack pointer to %eax, as arg1 of _start_c     */
> -		"and  $-16, %esp\n"       /* last pushed argument must be 16-byte aligned        */
> +		"and  $-16, %esp\n"       /* align stack to 16 bytes                             */
> +		"sub  $12, %esp\n"        /* last pushed argument must be 16-byte aligned        */
>  		"push %eax\n"             /* push arg1 on stack to support plain stack modes too */
>  		"call _start_c\n"         /* transfer to c runtime                               */
>  		"hlt\n"                   /* ensure it does not return                           */

Thanks for the fix!

In general:

Acked-by: Thomas Weißschuh <linux@weissschuh.net>


Though I'd like to have the following part of your follow-up mail as
part of the commit message [0]:

"""
When the call main is executed, the
%esp is 16 bytes aligned.

Then, on function entry (%esp mod 16) == 12
because the call instruction pushes 4 bytes
onto the stack.

subl $12, %esp will make (%esp mod 16) == 0
again.
"""

It's much clearer to someone not that familiar with ASM and its
intricacies; like me.

Also a Link: tag for the Reported-by: would be nice.

[0] https://lore.kernel.org/lkml/ZOoindMFj1UKqo+s@biznet-home.integral.gnuweeb.org/


Thomas

Re: [PATCH v1 1/1] tools/nolibc: i386: Fix a stack misalign bug on _start

[Ammar Faizi]

On Sun, Aug 27, 2023 at 08:53:26AM +0200, Thomas Weißschuh wrote:
> Afaik references to commits in commit messages are supposed to follow
> the same format as in the Fixes: tag.

Copied.

> Thanks for the fix!
> 
> In general:
> 
> Acked-by: Thomas Weißschuh <linux@weissschuh.net>

I'll fix the reference format, append your acked by tag, and explain
more about the logic behind 'sub $12, %esp' in v2.

I'm currently traveling, will send the v2 revision on Monday morning
Jakarta time.

> Also a Link: tag for the Reported-by: would be nice.

I would have linked the discussion if the report were on the list.
Unfortunately, Nicholas reported it offlist.

-- 
Ammar Faizi