[PATCHSET v1 RFC liburing 0/6] Add no libc support for x86-64 arch

[PATCHSET v1 RFC liburing 0/6] Add no libc support for x86-64 arch

[Ammar Faizi]

Hi everyone,

This is the v1 of RFC to support build liburing without libc.

In this RFC, I introduce no libc support for x86-64 arch. Hopefully,
one day we can get support for other architectures as well.

Motivation:
Currently liburing depends on libc. We want to make liburing can be
built without libc.

This idea firstly posted as an issue on the liburing GitHub
repository here: https://github.com/axboe/liburing/issues/443

The subject of the issue is: "An option to use liburing without libc?".

On Mon, Sep 27, 2021 at 4:18 PM Mahdi Rakhshandehroo <notifications@github.com> wrote:
> There are a couple of issues with liburing's libc dependency:
> 
>  1) libc implementations of errno, malloc, pthread etc. tend to
>     pollute the binary with unwanted global/thread-local state.
>     This makes reentrancy impossible and relocations expensive.
>  2) libc doesn't play nice with non-POSIX threading models, like
>     green threads with small stack sizes, or direct use of the
>     clone() system call. This makes interop with other
>     languages/runtimes difficult.
> 
> One could use the raw syscall interface to io_uring to address these
> concerns, but that would be somewhat painful, so it would be nice
> for liburing to support this use case out of the box. Perhaps
> something like a NOLIBC macro could be added which, if defined,
> would patch out libc constructs and replace them with non-libc
> wrappers where applicable. A few API changes might be necessary for
> the non-libc case (e.g. io_uring_get_probe/io_uring_free_probe), but
> it shouldn't break existing applications as long as it's opt-in.

----------------------------------------------------------------

Explanation about the changes:

- New directory for arch dependent files. We create a new directory
  `src/arch`. This is where the arch dependent sources live. Currently
  we only have single arch support `src/arch/x86`. This directory
  contains crafted syscalls written in inline Assembly and get page
  size function.

- Currently, liburing uses 4 libc functions, they are:
   1) `malloc`
   2) `free`
   3) `memset`
   4) `sysconf` (to get the page size).
  
  To support no libc build, we wrap all libc functions used by
  liburing as `static inline` functions. They are all defined in
  `src/lib.h`.

  If we build liburing with libc, it will still use the functions from
  libc. If we build liburing without libc, it will use the functions
  in `src/no_libc.c` or arch dependent function from `src/arch` if
  required.

- Procedure to free the return value of `io_uring_get_probe_{,ring}`.
  Currently, several tests use `free()` to free the return value of
  this *probe* functions. But since these changes we should always
  use `io_uring_free_probe()`. Don't use `free()`.

- Don't use `errno` to check error from liburing functions on tests.
  We want the tests still work properly with liburing no libc.

----------------------------------------------------------------

How to build liburing without libc?

You can just simply add `export LIBURING_NOLIBC=y` before run the
build. Be sure to run `make clean` if you have dirty build, just to
ensure consistency.

  ammarfaizi2@integral:~/project/now/liburing$ export LIBURING_NOLIBC=y
  ammarfaizi2@integral:~/project/now/liburing$ ./configure
  prefix                        /usr
  includedir                    /usr/include
  libdir                        /usr/lib
  libdevdir                     /usr/lib
  relativelibdir                
  mandir                        /usr/man
  datadir                       /usr/share
  stringop_overflow             yes
  array_bounds                  yes
  __kernel_rwf_t                yes
  __kernel_timespec             yes
  open_how                      no
  statx                         yes
  C++                           yes
  has_ucontext                  yes
  has_memfd_create              yes
  LIBURING_NOLIBC               yes
  CC                            gcc
  CXX                           g++
  ammarfaizi2@integral:~/project/now/liburing$ taskset -c 0,1,2,3 make -j4

Make sure you see the `LIBURING_NOLIBC` with `yes`.

----------------------------------------------------------------

Extra improvements of using liburing build without libc:

1) The file size of liburing.so is reduced.

  With libc:
    116984 src/liburing.so.2.1.0

  Without libc:
    104656 src/liburing.so.2.1.0

2) Efficient function call. We inline all `syscall` instructions with
   inline Assembly. This greatly reduces the data movement, as syscall
   only clobbers %rax, %rcx and %r11. Plus it is compatible with the
   kernel style return value, so no need a branch to catch error from
   `errno` variable anymore.

   With libc, we may spend more extra time to save caller saved
   registers just to perform a syscall, because if we use libc, every
   syscall is wrapped with a function call.

   Another extra thing is when we need to check `errno` variable, that
   will cost more extra call to `__errno_location` and extra branches
   (as per we implement the kernel style return value).

   Without libc, the generated Assembly code is also smaller. For
   example, we can take a look at this generated Assembly code of
   `__io_uring_sqring_wait` function.

  With libc:

    0000000000003340 <__io_uring_sqring_wait>:
      3340: f3 0f 1e fa           endbr64 
      3344: 48 83 ec 10           sub    $0x10,%rsp
      3348: 8b b7 c4 00 00 00     mov    0xc4(%rdi),%esi
      334e: 31 c9                 xor    %ecx,%ecx
      3350: 31 d2                 xor    %edx,%edx
      3352: 6a 08                 push   $0x8
      3354: 41 b8 04 00 00 00     mov    $0x4,%r8d
      335a: 45 31 c9              xor    %r9d,%r9d
      335d: bf aa 01 00 00        mov    $0x1aa,%edi
      3362: 31 c0                 xor    %eax,%eax
      3364: e8 17 ef ff ff        call   2280 <syscall@plt>
      3369: 5a                    pop    %rdx
      336a: 59                    pop    %rcx
      336b: 41 89 c0              mov    %eax,%r8d
      336e: 85 c0                 test   %eax,%eax
      3370: 79 0b                 jns    337d <__io_uring_sqring_wait+0x3d>
      3372: e8 49 ee ff ff        call   21c0 <__errno_location@plt>
      3377: 44 8b 00              mov    (%rax),%r8d
      337a: 41 f7 d8              neg    %r8d
      337d: 44 89 c0              mov    %r8d,%eax
      3380: 48 83 c4 08           add    $0x8,%rsp
      3384: c3                    ret
      3385: 66 2e 0f 1f 84 00 00  cs nopw 0x0(%rax,%rax,1)
      338c: 00 00 00 
      338f: 90                    nop


  Without libc:

    0000000000001e20 <__io_uring_sqring_wait>:
      1e20: f3 0f 1e fa           endbr64 
      1e24: 31 d2                 xor    %edx,%edx
      1e26: 8b bf c4 00 00 00     mov    0xc4(%rdi),%edi
      1e2c: 45 31 c0              xor    %r8d,%r8d
      1e2f: b8 aa 01 00 00        mov    $0x1aa,%eax
      1e34: 41 ba 04 00 00 00     mov    $0x4,%r10d
      1e3a: 41 b9 08 00 00 00     mov    $0x8,%r9d
      1e40: 89 d6                 mov    %edx,%esi
      1e42: 0f 05                 syscall
      1e44: c3                    ret

3) More portable shared library. Sometimes we meet a case where libc
   version is not compatible with other versions of libc.

   Now, as we do not depend on libc, it's easier to distribute the
   liburing.so without worrying about libc version anymore. As long as
   the architecture is the same and the kernel version is compatible,
   that should not be a problem.
----------------------------------------------------------------

How to apply and test?

This work is based on Jens Axboe's liburing repo on master branch, commit
a9895111798af2003f5298abb1d5bdaf11ca549e ("Fix typo "timout" -> "timeout"").

If you want a Git repository you can pull from

The following changes since commit a9895111798af2003f5298abb1d5bdaf11ca549e:

  Fix typo "timout" -> "timeout" (2021-10-05 16:42:02 -0600)

are available in the Git repository at:

  https://github.com/ammarfaizi2/liburing tags/nolibc-x86-64

for you to fetch changes up to 73ea5bf2b08c8ea100f16b4494353107ba5fe6d4:

  src/{queue,register,setup}: Clean up unused includes (2021-10-06 18:33:20 +0700)

Please review and comment!
----------------------------------------------------------------
Ammar Faizi (6):
      configure: Add LIBURING_NOLIBC variable
      Add no libc support
      Add x86-64 no libc build support
      test/cq-size: Don't use `errno` to check liburing's functions
      test/{iopoll,read-write}: Use `io_uring_free_probe()` instead of `free()`
      src/{queue,register,setup}: Clean up unused includes

 configure              |   7 ++
 src/Makefile           |  13 ++-
 src/arch/x86/lib.h     |  26 ++++++
 src/arch/x86/syscall.h | 200 +++++++++++++++++++++++++++++++++++++++++++
 src/lib.h              |  77 +++++++++++++++++
 src/no_libc.c          |  48 +++++++++++
 src/queue.c            |  14 +--
 src/register.c         |  12 +--
 src/setup.c            |  28 ++----
 src/syscall.c          |  11 ++-
 src/syscall.h          |  71 +++++++++++----
 test/Makefile          |  23 ++++-
 test/cq-size.c         |  10 ++-
 test/iopoll.c          |   2 +-
 test/read-write.c      |   2 +-
 15 files changed, 480 insertions(+), 64 deletions(-)
 create mode 100644 src/arch/x86/lib.h
 create mode 100644 src/arch/x86/syscall.h
 create mode 100644 src/lib.h
 create mode 100644 src/no_libc.c

-- 
Ammar Faizi

[PATCH v1 RFC liburing 1/6] configure: Add LIBURING_NOLIBC variable

[Ammar Faizi]

To support build liburing without libc.

Link: https://github.com/axboe/liburing/issues/443
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 configure | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/configure b/configure
index a7caa07..3dec81a 100755
--- a/configure
+++ b/configure
@@ -357,6 +357,13 @@ print_config "has_memfd_create" "$has_memfd_create"
 
 
 #############################################################################
+if test "$LIBURING_NOLIBC" = "y" -o "$LIBURING_NOLIBC" = "yes" -o "$LIBURING_NOLIBC" = "1"; then
+  output_sym "LIBURING_NOLIBC"
+  LIBURING_NOLIBC="yes"
+else
+  LIBURING_NOLIBC="no"
+fi
+print_config "LIBURING_NOLIBC" "$LIBURING_NOLIBC"
 
 if test "$__kernel_rwf_t" = "yes"; then
   output_sym "CONFIG_HAVE_KERNEL_RWF_T"
-- 
2.30.2

[PATCH v1 RFC liburing 2/6] Add no libc support

[Ammar Faizi]

Create `src/no_libc.c` to substitute libc functions like `memset()`,
`malloc()` and `free()`. Only build this file when we build liburing
without libc.

Wrap libc functions with `static inline` functions, defined in
`src/lib.h`, currently we have:
 1) `get_page_size`
 2) `uring_memset`
 3) `uring_malloc`
 4) `uring_free`

Extra notes for tests:
 1) Functions in `src/syscall.c` require libc to work.
 2) Tests require functions in `src/syscall.c`.

So we build `src/syscall.c` manually from test's Makefile.

The Makefile in `src/` dir still builds `src/syscall.c` when we
compile liburing with libc.

Cc: Bedirhan KURT <windowz414@gnuweeb.org>
Cc: Louvian Lyndal <louvianlyndal@gmail.com>
Link: https://github.com/axboe/liburing/issues/443
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 src/Makefile   | 13 +++++++++-
 src/lib.h      | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++
 src/no_libc.c  | 48 +++++++++++++++++++++++++++++++++++
 src/queue.c    |  1 +
 src/register.c |  1 +
 src/setup.c    | 16 +++++-------
 src/syscall.c  | 11 +++++++-
 test/Makefile  | 23 ++++++++++++++---
 8 files changed, 167 insertions(+), 15 deletions(-)
 create mode 100644 src/lib.h
 create mode 100644 src/no_libc.c

diff --git a/src/Makefile b/src/Makefile
index 5e46a9d..3ca2ac7 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -32,11 +32,22 @@ endif
 
 all: $(all_targets)
 
-liburing_srcs := setup.c queue.c syscall.c register.c
+liburing_srcs := setup.c queue.c register.c
+
+ifeq ($(LIBURING_NOLIBC),y)
+	liburing_srcs += no_libc.c
+	override CFLAGS += -nostdlib -nolibc -nodefaultlibs -ffreestanding -fno-stack-protector -fpic
+	override CPPFLAGS += -nostdlib -nolibc -nodefaultlibs -ffreestanding -fno-stack-protector -fpic
+	override LINK_FLAGS += -nostdlib -nolibc -nodefaultlibs -fpic
+else
+	liburing_srcs += syscall.c
+endif
 
 liburing_objs := $(patsubst %.c,%.ol,$(liburing_srcs))
 liburing_sobjs := $(patsubst %.c,%.os,$(liburing_srcs))
 
+$(liburing_srcs): syscall.h lib.h
+
 $(liburing_objs) $(liburing_sobjs): include/liburing/io_uring.h
 
 %.os: %.c
diff --git a/src/lib.h b/src/lib.h
new file mode 100644
index 0000000..171eee7
--- /dev/null
+++ b/src/lib.h
@@ -0,0 +1,69 @@
+/* SPDX-License-Identifier: MIT */
+#ifndef LIBURING_LIB_H
+#define LIBURING_LIB_H
+
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#ifndef offsetof
+# define offsetof(TYPE, FIELD) ((size_t) &((TYPE *)0)->FIELD)
+#endif
+
+#ifndef container_of
+# define container_of(PTR, TYPE, FIELD) ({				\
+	__typeof__(((TYPE *)0)->FIELD) *__FIELD_PTR = (PTR);		\
+	(TYPE *)((char *) __FIELD_PTR - offsetof(TYPE, FIELD));		\
+})
+#endif
+
+
+static inline long get_page_size(void)
+{
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_get_page_size();
+#else
+	long page_size;
+
+	page_size = sysconf(_SC_PAGESIZE);
+	if (page_size < 0)
+		page_size = 4096;
+
+	return page_size;
+#endif
+}
+
+void *__uring_memset(void *s, int c, size_t n);
+
+static inline void *uring_memset(void *s, int c, size_t n)
+{
+#ifdef LIBURING_NOLIBC
+	return __uring_memset(s, c, n);
+#else
+	return memset(s, c, n);
+#endif
+}
+
+void *__uring_malloc(size_t len);
+
+static inline void *uring_malloc(size_t len)
+{
+#ifdef LIBURING_NOLIBC
+	return __uring_malloc(len);
+#else
+	return malloc(len);
+#endif
+}
+
+void __uring_free(void *p);
+
+static inline void uring_free(void *p)
+{
+#ifdef LIBURING_NOLIBC
+	__uring_free(p);
+#else
+	free(p);
+#endif
+}
+
+#endif /* #ifndef LIBURING_LIB_H */
diff --git a/src/no_libc.c b/src/no_libc.c
new file mode 100644
index 0000000..149426c
--- /dev/null
+++ b/src/no_libc.c
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: MIT */
+
+#ifndef LIBURING_NOLIBC
+# error "This file should only be compiled for no libc build"
+#endif
+
+#include "lib.h"
+#include "syscall.h"
+
+void *__uring_memset(void *s, int c, size_t n)
+{
+	size_t i;
+	unsigned char *p = s;
+
+	for (i = 0; i < n; i++)
+		p[i] = (unsigned char) c;
+
+	return s;
+}
+
+struct uring_heap {
+	size_t		len;
+	char		user_p[];
+};
+
+void *__uring_malloc(size_t len)
+{
+	struct uring_heap *heap;
+
+	heap = uring_mmap(NULL, sizeof(*heap) + len, PROT_READ | PROT_WRITE,
+			  MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+	if (IS_ERR(heap))
+		return NULL;
+
+	heap->len = sizeof(*heap) + len;
+	return heap->user_p;
+}
+
+void __uring_free(void *p)
+{
+	struct uring_heap *heap;
+
+	if (uring_unlikely(!p))
+		return;
+
+	heap = container_of(p, struct uring_heap, user_p);
+	uring_munmap(heap, heap->len);
+}
diff --git a/src/queue.c b/src/queue.c
index 9af29d5..cd76048 100644
--- a/src/queue.c
+++ b/src/queue.c
@@ -13,6 +13,7 @@
 #include "liburing.h"
 #include "liburing/barrier.h"
 
+#include "lib.h"
 #include "syscall.h"
 
 /*
diff --git a/src/register.c b/src/register.c
index 074223f..f8e88cf 100644
--- a/src/register.c
+++ b/src/register.c
@@ -12,6 +12,7 @@
 #include "liburing/io_uring.h"
 #include "liburing.h"
 
+#include "lib.h"
 #include "syscall.h"
 
 int io_uring_register_buffers_update_tag(struct io_uring *ring, unsigned off,
diff --git a/src/setup.c b/src/setup.c
index 4f006de..0f64a35 100644
--- a/src/setup.c
+++ b/src/setup.c
@@ -12,6 +12,7 @@
 #include "liburing/io_uring.h"
 #include "liburing.h"
 
+#include "lib.h"
 #include "syscall.h"
 
 static void io_uring_unmap_rings(struct io_uring_sq *sq, struct io_uring_cq *cq)
@@ -161,7 +162,7 @@ int io_uring_queue_init(unsigned entries, struct io_uring *ring, unsigned flags)
 {
 	struct io_uring_params p;
 
-	memset(&p, 0, sizeof(p));
+	uring_memset(&p, 0, sizeof(p));
 	p.flags = flags;
 
 	return io_uring_queue_init_params(entries, ring, &p);
@@ -184,16 +185,16 @@ struct io_uring_probe *io_uring_get_probe_ring(struct io_uring *ring)
 	int r;
 
 	len = sizeof(*probe) + 256 * sizeof(struct io_uring_probe_op);
-	probe = malloc(len);
+	probe = uring_malloc(len);
 	if (!probe)
 		return NULL;
-	memset(probe, 0, len);
+	uring_memset(probe, 0, len);
 
 	r = io_uring_register_probe(ring, probe, 256);
 	if (r >= 0)
 		return probe;
 
-	free(probe);
+	uring_free(probe);
 	return NULL;
 }
 
@@ -214,7 +215,7 @@ struct io_uring_probe *io_uring_get_probe(void)
 
 void io_uring_free_probe(struct io_uring_probe *probe)
 {
-	free(probe);
+	uring_free(probe);
 }
 
 static int __fls(int x)
@@ -336,10 +337,7 @@ ssize_t io_uring_mlock_size_params(unsigned entries, struct io_uring_params *p)
 		cq_entries = 2 * entries;
 	}
 
-	page_size = sysconf(_SC_PAGESIZE);
-	if (page_size < 0)
-		page_size = 4096;
-
+	page_size = get_page_size();
 	return rings_size(entries, cq_entries, page_size);
 }
 
diff --git a/src/syscall.c b/src/syscall.c
index 5923fbb..cfb3ee2 100644
--- a/src/syscall.c
+++ b/src/syscall.c
@@ -1,6 +1,16 @@
 /* SPDX-License-Identifier: MIT */
 #define _DEFAULT_SOURCE
 
+/*
+ * Functions in this file require libc, only build them when we use libc.
+ *
+ * Note:
+ * liburing's tests still need these functions.
+ */
+#if defined(LIBURING_NOLIBC) && !defined(LIBURING_BUILD_TEST)
+# error "This file should only be compiled for libc build"
+#endif
+
 /*
  * Will go away once libc support is there
  */
@@ -11,7 +21,6 @@
 #include "liburing/io_uring.h"
 #include "syscall.h"
 
-
 int __sys_io_uring_register(int fd, unsigned opcode, const void *arg,
 			    unsigned nr_args)
 {
diff --git a/test/Makefile b/test/Makefile
index 2936469..ddb589b 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -21,8 +21,8 @@ ifdef CONFIG_HAVE_ARRAY_BOUNDS
 endif
 
 CXXFLAGS ?= $(CFLAGS)
-override CFLAGS += $(XCFLAGS)
-override CXXFLAGS += $(XCFLAGS) -std=c++11
+override CFLAGS += $(XCFLAGS) -DLIBURING_BUILD_TEST
+override CXXFLAGS += $(XCFLAGS) -std=c++11 -DLIBURING_BUILD_TEST
 LDFLAGS ?=
 override LDFLAGS += -L../src/ -luring
 
@@ -153,11 +153,26 @@ test_targets += sq-full-cpp
 endif
 all_targets += sq-full-cpp
 
-helpers = helpers.o
+#
+# Note:
+#
+# Build ../src/syscall.c manually from test's Makefile to support
+# build liburing without libc.
+#
+# Functions in ../src/syscall.c require libc to work with, if we
+# build liburing.so without libc, we don't have those functions
+# in liburing.so.
+#
+# This makes the tests can run for liburing.so without libc.
+#
+helpers = helpers.o ../src/syscall.o
 
 all: ${helpers} $(test_targets)
 
-helpers.o: helpers.c helpers.c
+../src/syscall.o: ../src/syscall.c
+	$(QUIET_CC)$(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $<
+
+helpers.o: helpers.c
 	$(QUIET_CC)$(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $<
 
 %: %.c ${helpers} helpers.h
-- 
2.30.2

[PATCH v1 RFC liburing 3/6] Add x86-64 no libc build support

[Ammar Faizi]

1) Initiate no libc support from x86-64 arch.
2) Create `src/arch` directory. This directory is used to save arch
   dependent files.
3) Add x86-64 syscalls and lib support.

All of these are supposed to support build liburing without libc.

Cc: Bedirhan KURT <windowz414@gnuweeb.org>
Cc: Louvian Lyndal <louvianlyndal@gmail.com>
Link: https://github.com/axboe/liburing/issues/443
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 src/arch/x86/lib.h     |  26 ++++++
 src/arch/x86/syscall.h | 200 +++++++++++++++++++++++++++++++++++++++++
 src/lib.h              |   8 ++
 src/syscall.h          |  71 +++++++++++----
 4 files changed, 290 insertions(+), 15 deletions(-)
 create mode 100644 src/arch/x86/lib.h
 create mode 100644 src/arch/x86/syscall.h

diff --git a/src/arch/x86/lib.h b/src/arch/x86/lib.h
new file mode 100644
index 0000000..0d4b321
--- /dev/null
+++ b/src/arch/x86/lib.h
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: MIT */
+
+#ifndef LIBURING_ARCH_X86_LIB_H
+#define LIBURING_ARCH_X86_LIB_H
+
+#ifndef LIBURING_LIB_H
+#  error "This file should be included from src/lib.h (liburing)"
+#endif
+
+#if defined(__x86_64__)
+
+static inline long __arch_impl_get_page_size(void)
+{
+	return 4096;
+}
+
+#else /* #if defined(__x86_64__) */
+
+/*
+ * TODO: Add x86 (32-bit) support here.
+ */
+#error "x86 (32-bit) is currently not supported"
+
+#endif /* #if defined(__x86_64__) */
+
+#endif /* #ifndef LIBURING_ARCH_X86_LIB_H */
diff --git a/src/arch/x86/syscall.h b/src/arch/x86/syscall.h
new file mode 100644
index 0000000..1f36310
--- /dev/null
+++ b/src/arch/x86/syscall.h
@@ -0,0 +1,200 @@
+/* SPDX-License-Identifier: MIT */
+
+#ifndef LIBURING_ARCH_X86_SYSCALL_H
+#define LIBURING_ARCH_X86_SYSCALL_H
+
+#ifndef LIBURING_SYSCALL_H
+#  error "This file should be included from src/syscall.h (liburing)"
+#endif
+
+#if defined(__x86_64__)
+/**
+ * Note for syscall registers usage (x86-64):
+ *   - %rax is the syscall number.
+ *   - %rax is also the return value.
+ *   - %rdi is the 1st argument.
+ *   - %rsi is the 2nd argument.
+ *   - %rdx is the 3rd argument.
+ *   - %r10 is the 4th argument (**yes it's %r10, not %rcx!**).
+ *   - %r8  is the 5th argument.
+ *   - %r9  is the 6th argument.
+ *
+ * `syscall` instruction will clobber %r11 and %rcx.
+ *
+ * After the syscall returns to userspace:
+ *   - %r11 will contain %rflags.
+ *   - %rcx will contain the return address.
+ *
+ * IOW, after the syscall returns to userspace:
+ *   %r11 == %rflags and %rcx == %rip.
+ */
+
+static inline void *__arch_impl_mmap(void *addr, size_t length, int prot,
+				     int flags, int fd, off_t offset)
+{
+	void *rax;
+	register int r10 __asm__("r10") = flags;
+	register int r8 __asm__("r8") = fd;
+	register int r9 __asm__("r9") = offset;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)
+		: "a"(__NR_mmap),	/* %rax */
+		  "D"(addr),		/* %rdi */
+		  "S"(length),		/* %rsi */
+		  "d"(prot),		/* %rdx */
+		  "r"(r10),		/* %r10 */
+		  "r"(r8),		/* %r8  */
+		  "r"(r9)		/* %r9  */
+		: "memory", "rcx", "r11"
+	);
+	return rax;
+}
+
+static inline int __arch_impl_munmap(void *addr, size_t length)
+{
+	int eax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(eax)		/* %rax */
+		: "a"(__NR_munmap),	/* %rax */
+		  "D"(addr),		/* %rdi */
+		  "S"(length)		/* %rsi */
+		: "memory", "rcx", "r11"
+	);
+	return eax;
+}
+
+static inline int __arch_impl_madvise(void *addr, size_t length, int advice)
+{
+	int eax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(eax)		/* %rax */
+		: "a"(__NR_madvise),	/* %rax */
+		  "D"(addr),		/* %rdi */
+		  "S"(length),		/* %rsi */
+		  "d"(advice)		/* %rdx */
+		: "memory", "rcx", "r11"
+	);
+	return eax;
+}
+
+static inline int __arch_impl_getrlimit(int resource, struct rlimit *rlim)
+{
+	int eax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(eax)		/* %rax */
+		: "a"(__NR_getrlimit),	/* %rax */
+		  "D"(resource),	/* %rdi */
+		  "S"(rlim)		/* %rsi */
+		: "memory", "rcx", "r11"
+	);
+	return eax;
+}
+
+static inline int __arch_impl_setrlimit(int resource, const struct rlimit *rlim)
+{
+	int eax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(eax)		/* %rax */
+		: "a"(__NR_setrlimit),	/* %rax */
+		  "D"(resource),	/* %rdi */
+		  "S"(rlim)		/* %rsi */
+		: "memory", "rcx", "r11"
+	);
+	return eax;
+}
+
+static inline int __arch_impl_close(int fd)
+{
+	int eax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(eax)		/* %rax */
+		: "a"(__NR_close),	/* %rax */
+		  "D"(fd)		/* %rdi */
+		: "memory", "rcx", "r11"
+	);
+	return eax;
+}
+
+static inline int __arch_impl_io_uring_register(int fd, unsigned opcode,
+						const void *arg,
+						unsigned nr_args)
+{
+	int eax;
+	register unsigned r10 __asm__("r10") = nr_args;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(eax)			/* %rax */
+		: "a"(__NR_io_uring_register),	/* %rax */
+		  "D"(fd),			/* %rdi */
+		  "S"(opcode),			/* %rsi */
+		  "d"(arg),			/* %rdx */
+		  "r"(r10)			/* %r10 */
+		: "memory", "rcx", "r11"
+	);
+	return eax;
+}
+
+static inline int __arch_impl_io_uring_setup(unsigned entries,
+					     struct io_uring_params *p)
+{
+	int eax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(eax)			/* %rax */
+		: "a"(__NR_io_uring_setup),	/* %rax */
+		  "D"(entries),			/* %rdi */
+		  "S"(p)			/* %rsi */
+		: "memory", "rcx", "r11"
+	);
+	return eax;
+}
+
+static inline int __arch_impl_io_uring_enter(int fd, unsigned to_submit,
+					     unsigned min_complete,
+					     unsigned flags, sigset_t *sig,
+					     int sz)
+{
+	int eax;
+	register unsigned r10 __asm__("r10") = flags;
+	register sigset_t *r8 __asm__("r8") = sig;
+	register int r9 __asm__("r9") = sz;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(eax)
+		: "a"(__NR_io_uring_enter),	/* %rax */
+		  "D"(fd),			/* %rdi */
+		  "S"(to_submit),		/* %rsi */
+		  "d"(min_complete),		/* %rdx */
+		  "r"(r10),			/* %r10 */
+		  "r"(r8),			/* %r8  */
+		  "r"(r9)			/* %r9  */
+		: "memory", "rcx", "r11"
+	);
+	return eax;
+}
+
+#else /* #if defined(__x86_64__) */
+
+/*
+ * TODO: Add x86 (32-bit) support here.
+ */
+#error "x86 (32-bit) is currently not supported"
+
+#endif /* #if defined(__x86_64__) */
+
+#endif /* #ifndef LIBURING_ARCH_X86_SYSCALL_H */
diff --git a/src/lib.h b/src/lib.h
index 171eee7..baacabe 100644
--- a/src/lib.h
+++ b/src/lib.h
@@ -6,6 +6,14 @@
 #include <string.h>
 #include <unistd.h>
 
+#ifdef LIBURING_NOLIBC
+#  if defined(__x86_64__) || defined(__i386__)
+#    include "arch/x86/lib.h"
+#  else
+#    error "The arch is currently not supported to build liburing without libc"
+#  endif
+#endif
+
 #ifndef offsetof
 # define offsetof(TYPE, FIELD) ((size_t) &((TYPE *)0)->FIELD)
 #endif
diff --git a/src/syscall.h b/src/syscall.h
index 9eff968..82f5db0 100644
--- a/src/syscall.h
+++ b/src/syscall.h
@@ -29,13 +29,13 @@
 # endif
 #elif defined __mips__
 # ifndef __NR_io_uring_setup
-#  define __NR_io_uring_setup           (__NR_Linux + 425)
+#  define __NR_io_uring_setup		(__NR_Linux + 425)
 # endif
 # ifndef __NR_io_uring_enter
-#  define __NR_io_uring_enter           (__NR_Linux + 426)
+#  define __NR_io_uring_enter		(__NR_Linux + 426)
 # endif
 # ifndef __NR_io_uring_register
-#  define __NR_io_uring_register        (__NR_Linux + 427)
+#  define __NR_io_uring_register	(__NR_Linux + 427)
 # endif
 #else /* !__alpha__ and !__mips__ */
 # ifndef __NR_io_uring_setup
@@ -49,9 +49,22 @@
 # endif
 #endif
 
-
+/*
+ * Don't put this below the #include "arch/$arch/syscall.h", that
+ * file may need it.
+ */
 struct io_uring_params;
 
+
+#ifdef LIBURING_NOLIBC
+#  if defined(__x86_64__) || defined(__i386__)
+#    include "arch/x86/syscall.h"
+#  else
+#    error "The arch is currently not supported to build liburing without libc"
+#  endif
+#endif
+
+
 /*
  * System calls
  */
@@ -68,12 +81,12 @@ static inline void *ERR_PTR(intptr_t n)
 	return (void *) n;
 }
 
-static inline intptr_t PTR_ERR(void *ptr)
+static inline intptr_t PTR_ERR(const void *ptr)
 {
 	return (intptr_t) ptr;
 }
 
-static inline bool IS_ERR(void *ptr)
+static inline bool IS_ERR(const void *ptr)
 {
 	return uring_unlikely((uintptr_t) ptr >= (uintptr_t) -4095UL);
 }
@@ -81,30 +94,40 @@ static inline bool IS_ERR(void *ptr)
 static inline int ____sys_io_uring_register(int fd, unsigned opcode,
 					    const void *arg, unsigned nr_args)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_io_uring_register(fd, opcode, arg, nr_args);
+#else
 	int ret;
-
 	ret = syscall(__NR_io_uring_register, fd, opcode, arg, nr_args);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int ____sys_io_uring_setup(unsigned entries,
 					 struct io_uring_params *p)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_io_uring_setup(entries, p);
+#else
 	int ret;
-
 	ret = syscall(__NR_io_uring_setup, entries, p);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int ____sys_io_uring_enter2(int fd, unsigned to_submit,
 					  unsigned min_complete, unsigned flags,
 					  sigset_t *sig, int sz)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_io_uring_enter(fd, to_submit, min_complete, flags,
+					  sig, sz);
+#else
 	int ret;
-
 	ret = syscall(__NR_io_uring_enter, fd, to_submit, min_complete, flags,
 		      sig, sz);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int ____sys_io_uring_enter(int fd, unsigned to_submit,
@@ -118,50 +141,68 @@ static inline int ____sys_io_uring_enter(int fd, unsigned to_submit,
 static inline void *uring_mmap(void *addr, size_t length, int prot, int flags,
 			       int fd, off_t offset)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_mmap(addr, length, prot, flags, fd, offset);
+#else
 	void *ret;
-
 	ret = mmap(addr, length, prot, flags, fd, offset);
 	return (ret == MAP_FAILED) ? ERR_PTR(-errno) : ret;
+#endif
 }
 
 static inline int uring_munmap(void *addr, size_t length)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_munmap(addr, length);
+#else
 	int ret;
-
 	ret = munmap(addr, length);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int uring_madvise(void *addr, size_t length, int advice)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_madvise(addr, length, advice);
+#else
 	int ret;
-
 	ret = madvise(addr, length, advice);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int uring_getrlimit(int resource, struct rlimit *rlim)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_getrlimit(resource, rlim);
+#else
 	int ret;
-
 	ret = getrlimit(resource, rlim);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int uring_setrlimit(int resource, const struct rlimit *rlim)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_setrlimit(resource, rlim);
+#else
 	int ret;
-
 	ret = setrlimit(resource, rlim);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int uring_close(int fd)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_close(fd);
+#else
 	int ret;
-
 	ret = close(fd);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 #endif
-- 
2.30.2

[PATCH v1 RFC liburing 4/6] test/cq-size: Don't use `errno` to check liburing's functions

[Ammar Faizi]

When we build liburing without libc, we can't check `errno` variable
with respect to liburing's functions. Don't do that it in test.

Note:
The tests themselves can still use `errno` to check error from
functions that come from the libc, but not liburing.

Cc: Bedirhan KURT <windowz414@gnuweeb.org>
Cc: Louvian Lyndal <louvianlyndal@gmail.com>
Link: https://github.com/axboe/liburing/issues/443
Fixes: https://github.com/axboe/liburing/issues/449
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 test/cq-size.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/test/cq-size.c b/test/cq-size.c
index b7dd5b4..4e6e3d1 100644
--- a/test/cq-size.c
+++ b/test/cq-size.c
@@ -45,14 +45,20 @@ int main(int argc, char *argv[])
 	p.cq_entries = 0;
 
 	ret = io_uring_queue_init_params(4, &ring, &p);
-	if (ret >= 0 || errno != EINVAL) {
+	if (ret >= 0) {
 		printf("zero sized cq ring succeeded\n");
+		io_uring_queue_exit(&ring);
+		goto err;
+	}
+
+	if (ret != -EINVAL) {
+		printf("io_uring_queue_init_params failed, but not with -EINVAL"
+		       ", returned error %d (%s)\n", ret, strerror(-ret));
 		goto err;
 	}
 
 done:
 	return 0;
 err:
-	io_uring_queue_exit(&ring);
 	return 1;
 }
-- 
2.30.2

[PATCH v1 RFC liburing 5/6] test/{iopoll,read-write}: Use `io_uring_free_probe()` instead of `free()`

[Ammar Faizi]

`io_uring_free_probe()` should really be used to free the return value
of `io_uring_get_probe_ring()`. As we may not always allocate it with
`malloc()`. For example, to support no libc build [1].

Cc: Bedirhan KURT <windowz414@gnuweeb.org>
Cc: Louvian Lyndal <louvianlyndal@gmail.com>
Link: https://github.com/axboe/liburing/issues/443 [1]
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 test/iopoll.c     | 2 +-
 test/read-write.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/test/iopoll.c b/test/iopoll.c
index de36473..4bfc26a 100644
--- a/test/iopoll.c
+++ b/test/iopoll.c
@@ -306,7 +306,7 @@ static int probe_buf_select(void)
 		fprintf(stdout, "Buffer select not supported, skipping\n");
 		return 0;
 	}
-	free(p);
+	io_uring_free_probe(p);
 	return 0;
 }
 
diff --git a/test/read-write.c b/test/read-write.c
index 885905b..d54ad0e 100644
--- a/test/read-write.c
+++ b/test/read-write.c
@@ -480,7 +480,7 @@ static int test_buf_select(const char *filename, int nonvec)
 		fprintf(stdout, "Buffer select not supported, skipping\n");
 		return 0;
 	}
-	free(p);
+	io_uring_free_probe(p);
 
 	/*
 	 * Write out data with known pattern
-- 
2.30.2

[PATCH v1 RFC liburing 6/6] src/{queue,register,setup}: Clean up unused includes

[Ammar Faizi]

Cc: Bedirhan KURT <windowz414@gnuweeb.org>
Cc: Louvian Lyndal <louvianlyndal@gmail.com>
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 src/queue.c    | 15 +++------------
 src/register.c | 13 +++----------
 src/setup.c    | 14 +++-----------
 3 files changed, 9 insertions(+), 33 deletions(-)

diff --git a/src/queue.c b/src/queue.c
index cd76048..eb0c736 100644
--- a/src/queue.c
+++ b/src/queue.c
@@ -1,20 +1,11 @@
 /* SPDX-License-Identifier: MIT */
 #define _POSIX_C_SOURCE 200112L
 
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#include <unistd.h>
-#include <string.h>
-#include <stdbool.h>
-
-#include "liburing/compat.h"
-#include "liburing/io_uring.h"
-#include "liburing.h"
-#include "liburing/barrier.h"
-
 #include "lib.h"
 #include "syscall.h"
+#include "liburing.h"
+#include "liburing/compat.h"
+#include "liburing/io_uring.h"
 
 /*
  * Returns true if we're not using SQ thread (thus nobody submits but us)
diff --git a/src/register.c b/src/register.c
index f8e88cf..1f2c409 100644
--- a/src/register.c
+++ b/src/register.c
@@ -1,19 +1,12 @@
 /* SPDX-License-Identifier: MIT */
 #define _POSIX_C_SOURCE 200112L
 
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#include <sys/resource.h>
-#include <unistd.h>
-#include <string.h>
-
+#include "lib.h"
+#include "syscall.h"
+#include "liburing.h"
 #include "liburing/compat.h"
 #include "liburing/io_uring.h"
-#include "liburing.h"
 
-#include "lib.h"
-#include "syscall.h"
 
 int io_uring_register_buffers_update_tag(struct io_uring *ring, unsigned off,
 					 const struct iovec *iovecs,
diff --git a/src/setup.c b/src/setup.c
index 0f64a35..083b685 100644
--- a/src/setup.c
+++ b/src/setup.c
@@ -1,19 +1,11 @@
 /* SPDX-License-Identifier: MIT */
 #define _DEFAULT_SOURCE
 
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <unistd.h>
-#include <string.h>
-#include <stdlib.h>
-#include <signal.h>
-
-#include "liburing/compat.h"
-#include "liburing/io_uring.h"
-#include "liburing.h"
-
 #include "lib.h"
 #include "syscall.h"
+#include "liburing.h"
+#include "liburing/compat.h"
+#include "liburing/io_uring.h"
 
 static void io_uring_unmap_rings(struct io_uring_sq *sq, struct io_uring_cq *cq)
 {
-- 
2.30.2

Re: [PATCHSET v1 RFC liburing 0/6] Add no libc support for x86-64 arch

[Jens Axboe]

On 10/6/21 8:49 AM, Ammar Faizi wrote:
> Hi everyone,
> 
> This is the v1 of RFC to support build liburing without libc.
> 
> In this RFC, I introduce no libc support for x86-64 arch. Hopefully,
> one day we can get support for other architectures as well.
> 
> Motivation:
> Currently liburing depends on libc. We want to make liburing can be
> built without libc.
> 
> This idea firstly posted as an issue on the liburing GitHub
> repository here: https://github.com/axboe/liburing/issues/443
> 
> The subject of the issue is: "An option to use liburing without libc?".

This series seems to be somewhat upside down. You should fix up tests
first, then add support for x86-64 nolibc, then enable it. You seem to
be doing the opposite?

-- 
Jens Axboe

Re: [PATCHSET v1 RFC liburing 0/6] Add no libc support for x86-64 arch

[Ammar Faizi]

On Thu, Oct 7, 2021 at 1:48 AM Jens Axboe <axboe@kernel.dk> wrote:
>On 10/6/21 8:49 AM, Ammar Faizi wrote:
>> Hi everyone,
>> 
>> This is the v1 of RFC to support build liburing without libc.
>> 
>> In this RFC, I introduce no libc support for x86-64 arch. Hopefully,
>> one day we can get support for other architectures as well.
>> 
>> Motivation:
>> Currently liburing depends on libc. We want to make liburing can be
>> built without libc.
>> 
>> This idea firstly posted as an issue on the liburing GitHub
>> repository here: https://github.com/axboe/liburing/issues/443
>> 
>> The subject of the issue is: "An option to use liburing without libc?".
>
>This series seems to be somewhat upside down. You should fix up tests
>first, then add support for x86-64 nolibc, then enable it. You seem to
>be doing the opposite?
>
>-- 
>Jens Axboe

Yes, that's what I am doing.

I agree with add support for x86-64 nolibc, then enable it. However,
the tests fixes happened very naturally.

I would not be able to caught those broken tests if I didn't add the
nolibc support.

There are two main problems with the tests, all of them are caught
after adding no libc support.

  1) The test uses `errno` to check error from liburing functions,
     this is only problematic with no libc build. I wouldn't be able
     to caught this without adding no libc support. I caught several
     tests failed after added no libc support, then investigated the
     failure causes and found the culprit (it's errno from the libc).

  2) The test uses `free()` to free the return value of
     `io_uring_get_probe{,_ring}` functions
     from liburing. This causes invalid free only for nolibc build.
     So it does really make sense to add no libc support first, then
     fix up the tests. Because there is no way I can know this broken
     situation earlier.

Since now I know everything about the situation, I can do so. So I
will send the RFC v2 and rebase everything based on your order.

Thanks for the comment.

-- 
Ammar Faizi

Re: [PATCHSET v1 RFC liburing 0/6] Add no libc support for x86-64 arch

[Jens Axboe]

On 10/6/21 4:20 PM, Ammar Faizi wrote:
> On Thu, Oct 7, 2021 at 1:48 AM Jens Axboe <axboe@kernel.dk> wrote:
>> On 10/6/21 8:49 AM, Ammar Faizi wrote:
>>> Hi everyone,
>>>
>>> This is the v1 of RFC to support build liburing without libc.
>>>
>>> In this RFC, I introduce no libc support for x86-64 arch. Hopefully,
>>> one day we can get support for other architectures as well.
>>>
>>> Motivation:
>>> Currently liburing depends on libc. We want to make liburing can be
>>> built without libc.
>>>
>>> This idea firstly posted as an issue on the liburing GitHub
>>> repository here: https://github.com/axboe/liburing/issues/443
>>>
>>> The subject of the issue is: "An option to use liburing without libc?".
>>
>> This series seems to be somewhat upside down. You should fix up tests
>> first, then add support for x86-64 nolibc, then enable it. You seem to
>> be doing the opposite?
>>
>> -- 
>> Jens Axboe
> 
> Yes, that's what I am doing.
> 
> I agree with add support for x86-64 nolibc, then enable it. However,
> the tests fixes happened very naturally.
> 
> I would not be able to caught those broken tests if I didn't add the
> nolibc support.

Right, but that's exactly why they should get fixed up front! If not,
you've got this weird point in the git tree that doesn't make any sense.
Each patch, when applied, should leave you with a fully functional repo.

> There are two main problems with the tests, all of them are caught
> after adding no libc support.
> 
>   1) The test uses `errno` to check error from liburing functions,
>      this is only problematic with no libc build. I wouldn't be able
>      to caught this without adding no libc support. I caught several
>      tests failed after added no libc support, then investigated the
>      failure causes and found the culprit (it's errno from the libc).
> 
>   2) The test uses `free()` to free the return value of
>      `io_uring_get_probe{,_ring}` functions
>      from liburing. This causes invalid free only for nolibc build.
>      So it does really make sense to add no libc support first, then
>      fix up the tests. Because there is no way I can know this broken
>      situation earlier.

2 is just a plain bug, and was introduced when that probe freeing
function was added. So that's definitely just a separate upfront patch
to fix that.

1 can be fixed too upfront, as it should not cause any changes in
behavior.

> Since now I know everything about the situation, I can do so. So I
> will send the RFC v2 and rebase everything based on your order.

Thanks!

-- 
Jens Axboe

Re: [PATCH v1 RFC liburing 6/6] src/{queue,register,setup}: Clean up unused includes

[Ammar Faizi]

On Thu, Oct 7, 2021 at 1:45 AM Bedirhan KURT <windowz414@gnuweeb.org> wrote:
>
> Ammar,
>
> As you told me through Telegram, I have reviewed your patch but        
> aren't inclusions important for some functions and/or commands  
> to work on compiling process?
>
> I mean you surely know more about this but, from what I know
> with my Android development experience, omitting/removing
> inclusions and/or imports from files to be compiled or
> scripts to be used might cause issues during
> compilation/booting/usage process so I guess those
> inclusions were just unnecessary?
>
> I'm also aware that Android is no match with Linux kernel but  
> I just wanted to point that out.
>
> Regards,
>
> Bedirhan KURT.
>

Hmmm...

I don't fully understand your comment. But as far as I can tell, those
inclusions are duplicate. We have them in the lib.h and syscall.h
after these patches. If this removal caused a compilation problem, I
would have caught that before sending these patches as I've at least
built it.

-- 
Ammar Faizi

[PATCHSET v2 RFC liburing 0/5] Add no libc support for x86-64 arch

[Ammar Faizi]

Hi everyone,

This is the v2 of RFC to support build liburing without libc.

In this RFC, I introduce no libc support for x86-64 arch. Hopefully,
one day we can get support for other architectures as well.

Motivation:
Currently liburing depends on libc. We want to make liburing can be
built without libc.

This idea firstly posted as an issue on the liburing GitHub
repository here: https://github.com/axboe/liburing/issues/443

The subject of the issue is: "An option to use liburing without libc?".

On Mon, Sep 27, 2021 at 4:18 PM Mahdi Rakhshandehroo <notifications@github.com> wrote:
> There are a couple of issues with liburing's libc dependency:
> 
>  1) libc implementations of errno, malloc, pthread etc. tend to
>     pollute the binary with unwanted global/thread-local state.
>     This makes reentrancy impossible and relocations expensive.
>  2) libc doesn't play nice with non-POSIX threading models, like
>     green threads with small stack sizes, or direct use of the
>     clone() system call. This makes interop with other
>     languages/runtimes difficult.
> 
> One could use the raw syscall interface to io_uring to address these
> concerns, but that would be somewhat painful, so it would be nice
> for liburing to support this use case out of the box. Perhaps
> something like a NOLIBC macro could be added which, if defined,
> would patch out libc constructs and replace them with non-libc
> wrappers where applicable. A few API changes might be necessary for
> the non-libc case (e.g. io_uring_get_probe/io_uring_free_probe), but
> it shouldn't break existing applications as long as it's opt-in.

----------------------------------------------------------------

Explanation about the changes:

- New directory for arch dependent files. We create a new directory
  `src/arch`. This is where the arch dependent sources live. Currently
  we only have single arch support `src/arch/x86`. This directory
  contains crafted syscalls written in inline Assembly and get page
  size function.

- Currently, liburing uses 4 libc functions, they are:
   1) `malloc`
   2) `free`
   3) `memset`
   4) `sysconf` (to get the page size).
  
  To support no libc build, we wrap all libc functions used by
  liburing as `static inline` functions. They are all defined in
  `src/lib.h`.

  If we build liburing with libc, it will still use the functions from
  libc. If we build liburing without libc, it will use the functions
  in `src/no_libc.c` or arch dependent function from `src/arch` if
  required.

- Procedure to free the return value of `io_uring_get_probe_{,ring}`.
  Currently, several tests use `free()` to free the return value of
  this *probe* functions. But since these changes we should always
  use `io_uring_free_probe()`. Don't use `free()`.

- Don't use `errno` to check error from liburing functions on tests.
  We want the tests still work properly with liburing no libc.

----------------------------------------------------------------

How to build liburing without libc?

You can just simply add `export LIBURING_NOLIBC=y` before run the
build. Be sure to run `make clean` if you have dirty build, just to
ensure consistency.

  ammarfaizi2@integral:~/project/now/liburing$ export LIBURING_NOLIBC=y
  ammarfaizi2@integral:~/project/now/liburing$ ./configure
  prefix                        /usr
  includedir                    /usr/include
  libdir                        /usr/lib
  libdevdir                     /usr/lib
  relativelibdir                
  mandir                        /usr/man
  datadir                       /usr/share
  stringop_overflow             yes
  array_bounds                  yes
  __kernel_rwf_t                yes
  __kernel_timespec             yes
  open_how                      no
  statx                         yes
  C++                           yes
  has_ucontext                  yes
  has_memfd_create              yes
  LIBURING_NOLIBC               yes
  CC                            gcc
  CXX                           g++
  ammarfaizi2@integral:~/project/now/liburing$ taskset -c 0,1,2,3 make -j4

Make sure you see the `LIBURING_NOLIBC` with `yes`.

----------------------------------------------------------------

Extra improvements of using liburing build without libc:

1) The file size of liburing.so is reduced.

  With libc:
    116984 src/liburing.so.2.1.0

  Without libc:
    104656 src/liburing.so.2.1.0

2) Efficient function call. We inline all `syscall` instructions with
   inline Assembly. This greatly reduces the data movement, as syscall
   only clobbers %rax, %rcx and %r11. Plus it is compatible with the
   kernel style return value, so no need a branch to catch error from
   `errno` variable anymore.

   With libc, we may spend more extra time to save caller saved
   registers just to perform a syscall, because if we use libc, every
   syscall is wrapped with a function call.

   Another extra thing is when we need to check `errno` variable, that
   will cost more extra call to `__errno_location` and extra branches
   (as per we implement the kernel style return value).

   Without libc, the generated Assembly code is also smaller. For
   example, we can take a look at this generated Assembly code of
   `__io_uring_sqring_wait` function.

  With libc:

    0000000000003340 <__io_uring_sqring_wait>:
      3340: f3 0f 1e fa           endbr64 
      3344: 48 83 ec 10           sub    $0x10,%rsp
      3348: 8b b7 c4 00 00 00     mov    0xc4(%rdi),%esi
      334e: 31 c9                 xor    %ecx,%ecx
      3350: 31 d2                 xor    %edx,%edx
      3352: 6a 08                 push   $0x8
      3354: 41 b8 04 00 00 00     mov    $0x4,%r8d
      335a: 45 31 c9              xor    %r9d,%r9d
      335d: bf aa 01 00 00        mov    $0x1aa,%edi
      3362: 31 c0                 xor    %eax,%eax
      3364: e8 17 ef ff ff        call   2280 <syscall@plt>
      3369: 5a                    pop    %rdx
      336a: 59                    pop    %rcx
      336b: 41 89 c0              mov    %eax,%r8d
      336e: 85 c0                 test   %eax,%eax
      3370: 79 0b                 jns    337d <__io_uring_sqring_wait+0x3d>
      3372: e8 49 ee ff ff        call   21c0 <__errno_location@plt>
      3377: 44 8b 00              mov    (%rax),%r8d
      337a: 41 f7 d8              neg    %r8d
      337d: 44 89 c0              mov    %r8d,%eax
      3380: 48 83 c4 08           add    $0x8,%rsp
      3384: c3                    ret
      3385: 66 2e 0f 1f 84 00 00  cs nopw 0x0(%rax,%rax,1)
      338c: 00 00 00 
      338f: 90                    nop


  Without libc:

    0000000000001e20 <__io_uring_sqring_wait>:
      1e20: f3 0f 1e fa           endbr64 
      1e24: 31 d2                 xor    %edx,%edx
      1e26: 8b bf c4 00 00 00     mov    0xc4(%rdi),%edi
      1e2c: 45 31 c0              xor    %r8d,%r8d
      1e2f: b8 aa 01 00 00        mov    $0x1aa,%eax
      1e34: 41 ba 04 00 00 00     mov    $0x4,%r10d
      1e3a: 41 b9 08 00 00 00     mov    $0x8,%r9d
      1e40: 89 d6                 mov    %edx,%esi
      1e42: 0f 05                 syscall
      1e44: c3                    ret

3) More portable shared library. Sometimes we meet a case where libc
   version is not compatible with other versions of libc.

   Now, as we do not depend on libc, it's easier to distribute the
   liburing.so without worrying about libc version anymore. As long as
   the architecture is the same and the kernel version is compatible,
   that should not be a problem.
----------------------------------------------------------------
v2:
  - Rebase the work based on commit 326ed975d49e8c7b ("configure: add
    openat2.h for open_how and RESOLVE_* flags").

  - Fix the patches order, make sure fix up the tests first, add
    nolibc sources, and then add a variable build to enable it.

  - Fix incorrect data type for `__arch_impl_mmap()` offset. It was
    `int` (that's not right). The proper data type is `off_t`.

  - Always use `long` or `void *` to contain the return value of
    syscall in `__arch_impl_*` functions.

  - Rename `src/no_libc.` to `src/nolibc.c`.

  - Reduce the number of patches to 5, it was 6.

Link: [v1] https://lore.kernel.org/io-uring/20211006144911.1181674-1-ammar.faizi@students.amikom.ac.id/T/
----------------------------------------------------------------

How to apply and test?

This work is based on Jens Axboe's liburing repo on master branch, commit
326ed975d49e8c7bff071abe8e7ac5e0c0196622 ("configure: add openat2.h for
open_how and RESOLVE_* flags").

If you want a Git repository you can pull from

The following changes since commit 326ed975d49e8c7bff071abe8e7ac5e0c0196622:

  configure: add openat2.h for open_how and RESOLVE_* flags (2021-10-06 12:08:45 -0600)

are available in the Git repository at:

  https://github.com/ammarfaizi2/liburing tags/nolibc-x86-64-v2

for you to fetch changes up to ebe8926cef4149cd6427607fab0d86537c5a2ba8:

  Add LIBURING_NOLIBC variable and edit src/Makefile (2021-10-07 13:07:35 +0700)

Please review and comment!
----------------------------------------------------------------
Ammar Faizi (5):
      test/{iopoll,read-write}: Use `io_uring_free_probe()` instead of `free()`
      test/cq-size: Don't use `errno` to check liburing's functions
      Add arch dependent directory and files
      Add no libc build support
      Add LIBURING_NOLIBC variable and edit src/Makefile

 configure              |   7 ++
 src/Makefile           |  13 +++-
 src/arch/x86/lib.h     |  26 +++++++
 src/arch/x86/syscall.h | 200 +++++++++++++++++++++++++++++++++++++++++++++++++
 src/lib.h              |  77 +++++++++++++++++++
 src/nolibc.c           |  48 ++++++++++++
 src/queue.c            |  14 +---
 src/register.c         |  12 +--
 src/setup.c            |  30 +++-----
 src/syscall.c          |  11 ++-
 src/syscall.h          |  71 ++++++++++++++----
 test/Makefile          |  19 ++++-
 test/cq-size.c         |  10 ++-
 test/iopoll.c          |   2 +-
 test/read-write.c      |   2 +-
 15 files changed, 477 insertions(+), 65 deletions(-)
 create mode 100644 src/arch/x86/lib.h
 create mode 100644 src/arch/x86/syscall.h
 create mode 100644 src/lib.h
 create mode 100644 src/nolibc.c

-- 
Ammar Faizi

[PATCH v2 RFC liburing 1/5] test/{iopoll,read-write}: Use `io_uring_free_probe()` instead of `free()`

[Ammar Faizi]

`io_uring_free_probe()` should really be used to free the return value
of `io_uring_get_probe_ring()`. As we may not always allocate it with
`malloc()`. For example, to support no libc build [1].

Link: https://github.com/axboe/liburing/issues/443 [1]
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 test/iopoll.c     | 2 +-
 test/read-write.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/test/iopoll.c b/test/iopoll.c
index de36473..4bfc26a 100644
--- a/test/iopoll.c
+++ b/test/iopoll.c
@@ -306,7 +306,7 @@ static int probe_buf_select(void)
 		fprintf(stdout, "Buffer select not supported, skipping\n");
 		return 0;
 	}
-	free(p);
+	io_uring_free_probe(p);
 	return 0;
 }
 
diff --git a/test/read-write.c b/test/read-write.c
index 885905b..d54ad0e 100644
--- a/test/read-write.c
+++ b/test/read-write.c
@@ -480,7 +480,7 @@ static int test_buf_select(const char *filename, int nonvec)
 		fprintf(stdout, "Buffer select not supported, skipping\n");
 		return 0;
 	}
-	free(p);
+	io_uring_free_probe(p);
 
 	/*
 	 * Write out data with known pattern
-- 
2.30.2

[PATCH v2 RFC liburing 2/5] test/cq-size: Don't use `errno` to check liburing's functions

[Ammar Faizi]

When we build liburing without libc, we can't check `errno` variable
with respect to liburing's functions. Don't do that it in test.

Note:
The tests themselves can still use `errno` to check error from
functions that come from the libc, but not liburing.

Link: https://github.com/axboe/liburing/issues/443
Fixes: https://github.com/axboe/liburing/issues/449
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 test/cq-size.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/test/cq-size.c b/test/cq-size.c
index b7dd5b4..4e6e3d1 100644
--- a/test/cq-size.c
+++ b/test/cq-size.c
@@ -45,14 +45,20 @@ int main(int argc, char *argv[])
 	p.cq_entries = 0;
 
 	ret = io_uring_queue_init_params(4, &ring, &p);
-	if (ret >= 0 || errno != EINVAL) {
+	if (ret >= 0) {
 		printf("zero sized cq ring succeeded\n");
+		io_uring_queue_exit(&ring);
+		goto err;
+	}
+
+	if (ret != -EINVAL) {
+		printf("io_uring_queue_init_params failed, but not with -EINVAL"
+		       ", returned error %d (%s)\n", ret, strerror(-ret));
 		goto err;
 	}
 
 done:
 	return 0;
 err:
-	io_uring_queue_exit(&ring);
 	return 1;
 }
-- 
2.30.2

[PATCH v2 RFC liburing 3/5] Add arch dependent directory and files

[Ammar Faizi]

Create a new directory `src/arch` to save arch dependent sources.
Add support start from x86-64, add syscalls crafted in Assembly code
and lib (currently the lib only contains get page size function).

Link: https://github.com/axboe/liburing/issues/443
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 src/arch/x86/lib.h     |  26 ++++++
 src/arch/x86/syscall.h | 200 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 226 insertions(+)
 create mode 100644 src/arch/x86/lib.h
 create mode 100644 src/arch/x86/syscall.h

diff --git a/src/arch/x86/lib.h b/src/arch/x86/lib.h
new file mode 100644
index 0000000..0d4b321
--- /dev/null
+++ b/src/arch/x86/lib.h
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: MIT */
+
+#ifndef LIBURING_ARCH_X86_LIB_H
+#define LIBURING_ARCH_X86_LIB_H
+
+#ifndef LIBURING_LIB_H
+#  error "This file should be included from src/lib.h (liburing)"
+#endif
+
+#if defined(__x86_64__)
+
+static inline long __arch_impl_get_page_size(void)
+{
+	return 4096;
+}
+
+#else /* #if defined(__x86_64__) */
+
+/*
+ * TODO: Add x86 (32-bit) support here.
+ */
+#error "x86 (32-bit) is currently not supported"
+
+#endif /* #if defined(__x86_64__) */
+
+#endif /* #ifndef LIBURING_ARCH_X86_LIB_H */
diff --git a/src/arch/x86/syscall.h b/src/arch/x86/syscall.h
new file mode 100644
index 0000000..be18165
--- /dev/null
+++ b/src/arch/x86/syscall.h
@@ -0,0 +1,200 @@
+/* SPDX-License-Identifier: MIT */
+
+#ifndef LIBURING_ARCH_X86_SYSCALL_H
+#define LIBURING_ARCH_X86_SYSCALL_H
+
+#ifndef LIBURING_SYSCALL_H
+#  error "This file should be included from src/syscall.h (liburing)"
+#endif
+
+#if defined(__x86_64__)
+/**
+ * Note for syscall registers usage (x86-64):
+ *   - %rax is the syscall number.
+ *   - %rax is also the return value.
+ *   - %rdi is the 1st argument.
+ *   - %rsi is the 2nd argument.
+ *   - %rdx is the 3rd argument.
+ *   - %r10 is the 4th argument (**yes it's %r10, not %rcx!**).
+ *   - %r8  is the 5th argument.
+ *   - %r9  is the 6th argument.
+ *
+ * `syscall` instruction will clobber %r11 and %rcx.
+ *
+ * After the syscall returns to userspace:
+ *   - %r11 will contain %rflags.
+ *   - %rcx will contain the return address.
+ *
+ * IOW, after the syscall returns to userspace:
+ *   %r11 == %rflags and %rcx == %rip.
+ */
+
+static inline void *__arch_impl_mmap(void *addr, size_t length, int prot,
+				     int flags, int fd, off_t offset)
+{
+	void *rax;
+	register int r10 __asm__("r10") = flags;
+	register int r8 __asm__("r8") = fd;
+	register off_t r9 __asm__("r9") = offset;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)
+		: "a"(__NR_mmap),	/* %rax */
+		  "D"(addr),		/* %rdi */
+		  "S"(length),		/* %rsi */
+		  "d"(prot),		/* %rdx */
+		  "r"(r10),		/* %r10 */
+		  "r"(r8),		/* %r8  */
+		  "r"(r9)		/* %r9  */
+		: "memory", "rcx", "r11"
+	);
+	return rax;
+}
+
+static inline int __arch_impl_munmap(void *addr, size_t length)
+{
+	long rax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)		/* %rax */
+		: "a"(__NR_munmap),	/* %rax */
+		  "D"(addr),		/* %rdi */
+		  "S"(length)		/* %rsi */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+static inline int __arch_impl_madvise(void *addr, size_t length, int advice)
+{
+	long rax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)		/* %rax */
+		: "a"(__NR_madvise),	/* %rax */
+		  "D"(addr),		/* %rdi */
+		  "S"(length),		/* %rsi */
+		  "d"(advice)		/* %rdx */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+static inline int __arch_impl_getrlimit(int resource, struct rlimit *rlim)
+{
+	long rax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)		/* %rax */
+		: "a"(__NR_getrlimit),	/* %rax */
+		  "D"(resource),	/* %rdi */
+		  "S"(rlim)		/* %rsi */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+static inline int __arch_impl_setrlimit(int resource, const struct rlimit *rlim)
+{
+	long rax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)		/* %rax */
+		: "a"(__NR_setrlimit),	/* %rax */
+		  "D"(resource),	/* %rdi */
+		  "S"(rlim)		/* %rsi */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+static inline int __arch_impl_close(int fd)
+{
+	long rax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)		/* %rax */
+		: "a"(__NR_close),	/* %rax */
+		  "D"(fd)		/* %rdi */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+static inline int __arch_impl_io_uring_register(int fd, unsigned opcode,
+						const void *arg,
+						unsigned nr_args)
+{
+	long rax;
+	register unsigned r10 __asm__("r10") = nr_args;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)			/* %rax */
+		: "a"(__NR_io_uring_register),	/* %rax */
+		  "D"(fd),			/* %rdi */
+		  "S"(opcode),			/* %rsi */
+		  "d"(arg),			/* %rdx */
+		  "r"(r10)			/* %r10 */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+static inline int __arch_impl_io_uring_setup(unsigned entries,
+					     struct io_uring_params *p)
+{
+	long rax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)			/* %rax */
+		: "a"(__NR_io_uring_setup),	/* %rax */
+		  "D"(entries),			/* %rdi */
+		  "S"(p)			/* %rsi */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+static inline int __arch_impl_io_uring_enter(int fd, unsigned to_submit,
+					     unsigned min_complete,
+					     unsigned flags, sigset_t *sig,
+					     int sz)
+{
+	long rax;
+	register unsigned r10 __asm__("r10") = flags;
+	register sigset_t *r8 __asm__("r8") = sig;
+	register int r9 __asm__("r9") = sz;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)
+		: "a"(__NR_io_uring_enter),	/* %rax */
+		  "D"(fd),			/* %rdi */
+		  "S"(to_submit),		/* %rsi */
+		  "d"(min_complete),		/* %rdx */
+		  "r"(r10),			/* %r10 */
+		  "r"(r8),			/* %r8  */
+		  "r"(r9)			/* %r9  */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+#else /* #if defined(__x86_64__) */
+
+/*
+ * TODO: Add x86 (32-bit) support here.
+ */
+#error "x86 (32-bit) is currently not supported"
+
+#endif /* #if defined(__x86_64__) */
+
+#endif /* #ifndef LIBURING_ARCH_X86_SYSCALL_H */
-- 
2.30.2

[PATCH v2 RFC liburing 4/5] Add no libc build support

[Ammar Faizi]

Create `src/nolibc.c` to substitute libc functions like `memset()`,
`malloc()` and `free()`. Only build this file when we build liburing
without libc.

Wrap libc functions with `static inline` functions, defined in
`src/lib.h`, currently we have:
 1) `get_page_size`
 2) `uring_memset`
 3) `uring_malloc`
 4) `uring_free`

Add conditional preprocessor in `src/{syscall,lib}.h` to use arch
dependent and nolibc functions when we build liburing without libc.

Extra notes for tests:
 1) Functions in `src/syscall.c` require libc to work.
 2) Tests require functions in `src/syscall.c`.

So we build `src/syscall.c` manually from test's Makefile.

The Makefile in `src/` dir still builds `src/syscall.c` when we
compile liburing with libc.

Link: https://github.com/axboe/liburing/issues/443
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 src/lib.h      | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++
 src/nolibc.c   | 48 +++++++++++++++++++++++++++++++
 src/queue.c    | 14 ++-------
 src/register.c | 12 ++------
 src/setup.c    | 30 +++++++-------------
 src/syscall.c  | 11 +++++++-
 src/syscall.h  | 71 ++++++++++++++++++++++++++++++++++++----------
 test/Makefile  | 19 ++++++++++---
 8 files changed, 222 insertions(+), 60 deletions(-)
 create mode 100644 src/lib.h
 create mode 100644 src/nolibc.c

diff --git a/src/lib.h b/src/lib.h
new file mode 100644
index 0000000..baacabe
--- /dev/null
+++ b/src/lib.h
@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: MIT */
+#ifndef LIBURING_LIB_H
+#define LIBURING_LIB_H
+
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#ifdef LIBURING_NOLIBC
+#  if defined(__x86_64__) || defined(__i386__)
+#    include "arch/x86/lib.h"
+#  else
+#    error "The arch is currently not supported to build liburing without libc"
+#  endif
+#endif
+
+#ifndef offsetof
+# define offsetof(TYPE, FIELD) ((size_t) &((TYPE *)0)->FIELD)
+#endif
+
+#ifndef container_of
+# define container_of(PTR, TYPE, FIELD) ({				\
+	__typeof__(((TYPE *)0)->FIELD) *__FIELD_PTR = (PTR);		\
+	(TYPE *)((char *) __FIELD_PTR - offsetof(TYPE, FIELD));		\
+})
+#endif
+
+
+static inline long get_page_size(void)
+{
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_get_page_size();
+#else
+	long page_size;
+
+	page_size = sysconf(_SC_PAGESIZE);
+	if (page_size < 0)
+		page_size = 4096;
+
+	return page_size;
+#endif
+}
+
+void *__uring_memset(void *s, int c, size_t n);
+
+static inline void *uring_memset(void *s, int c, size_t n)
+{
+#ifdef LIBURING_NOLIBC
+	return __uring_memset(s, c, n);
+#else
+	return memset(s, c, n);
+#endif
+}
+
+void *__uring_malloc(size_t len);
+
+static inline void *uring_malloc(size_t len)
+{
+#ifdef LIBURING_NOLIBC
+	return __uring_malloc(len);
+#else
+	return malloc(len);
+#endif
+}
+
+void __uring_free(void *p);
+
+static inline void uring_free(void *p)
+{
+#ifdef LIBURING_NOLIBC
+	__uring_free(p);
+#else
+	free(p);
+#endif
+}
+
+#endif /* #ifndef LIBURING_LIB_H */
diff --git a/src/nolibc.c b/src/nolibc.c
new file mode 100644
index 0000000..149426c
--- /dev/null
+++ b/src/nolibc.c
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: MIT */
+
+#ifndef LIBURING_NOLIBC
+# error "This file should only be compiled for no libc build"
+#endif
+
+#include "lib.h"
+#include "syscall.h"
+
+void *__uring_memset(void *s, int c, size_t n)
+{
+	size_t i;
+	unsigned char *p = s;
+
+	for (i = 0; i < n; i++)
+		p[i] = (unsigned char) c;
+
+	return s;
+}
+
+struct uring_heap {
+	size_t		len;
+	char		user_p[];
+};
+
+void *__uring_malloc(size_t len)
+{
+	struct uring_heap *heap;
+
+	heap = uring_mmap(NULL, sizeof(*heap) + len, PROT_READ | PROT_WRITE,
+			  MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+	if (IS_ERR(heap))
+		return NULL;
+
+	heap->len = sizeof(*heap) + len;
+	return heap->user_p;
+}
+
+void __uring_free(void *p)
+{
+	struct uring_heap *heap;
+
+	if (uring_unlikely(!p))
+		return;
+
+	heap = container_of(p, struct uring_heap, user_p);
+	uring_munmap(heap, heap->len);
+}
diff --git a/src/queue.c b/src/queue.c
index 9af29d5..eb0c736 100644
--- a/src/queue.c
+++ b/src/queue.c
@@ -1,19 +1,11 @@
 /* SPDX-License-Identifier: MIT */
 #define _POSIX_C_SOURCE 200112L
 
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#include <unistd.h>
-#include <string.h>
-#include <stdbool.h>
-
+#include "lib.h"
+#include "syscall.h"
+#include "liburing.h"
 #include "liburing/compat.h"
 #include "liburing/io_uring.h"
-#include "liburing.h"
-#include "liburing/barrier.h"
-
-#include "syscall.h"
 
 /*
  * Returns true if we're not using SQ thread (thus nobody submits but us)
diff --git a/src/register.c b/src/register.c
index 074223f..1f2c409 100644
--- a/src/register.c
+++ b/src/register.c
@@ -1,18 +1,12 @@
 /* SPDX-License-Identifier: MIT */
 #define _POSIX_C_SOURCE 200112L
 
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#include <sys/resource.h>
-#include <unistd.h>
-#include <string.h>
-
+#include "lib.h"
+#include "syscall.h"
+#include "liburing.h"
 #include "liburing/compat.h"
 #include "liburing/io_uring.h"
-#include "liburing.h"
 
-#include "syscall.h"
 
 int io_uring_register_buffers_update_tag(struct io_uring *ring, unsigned off,
 					 const struct iovec *iovecs,
diff --git a/src/setup.c b/src/setup.c
index 4f006de..700dc07 100644
--- a/src/setup.c
+++ b/src/setup.c
@@ -1,18 +1,11 @@
 /* SPDX-License-Identifier: MIT */
 #define _DEFAULT_SOURCE
 
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <unistd.h>
-#include <string.h>
-#include <stdlib.h>
-#include <signal.h>
-
+#include "lib.h"
+#include "syscall.h"
+#include "liburing.h"
 #include "liburing/compat.h"
 #include "liburing/io_uring.h"
-#include "liburing.h"
-
-#include "syscall.h"
 
 static void io_uring_unmap_rings(struct io_uring_sq *sq, struct io_uring_cq *cq)
 {
@@ -93,7 +86,7 @@ int io_uring_queue_mmap(int fd, struct io_uring_params *p, struct io_uring *ring
 {
 	int ret;
 
-	memset(ring, 0, sizeof(*ring));
+	uring_memset(ring, 0, sizeof(*ring));
 	ret = io_uring_mmap(fd, p, &ring->sq, &ring->cq);
 	if (!ret) {
 		ring->flags = p->flags;
@@ -161,7 +154,7 @@ int io_uring_queue_init(unsigned entries, struct io_uring *ring, unsigned flags)
 {
 	struct io_uring_params p;
 
-	memset(&p, 0, sizeof(p));
+	uring_memset(&p, 0, sizeof(p));
 	p.flags = flags;
 
 	return io_uring_queue_init_params(entries, ring, &p);
@@ -184,16 +177,16 @@ struct io_uring_probe *io_uring_get_probe_ring(struct io_uring *ring)
 	int r;
 
 	len = sizeof(*probe) + 256 * sizeof(struct io_uring_probe_op);
-	probe = malloc(len);
+	probe = uring_malloc(len);
 	if (!probe)
 		return NULL;
-	memset(probe, 0, len);
+	uring_memset(probe, 0, len);
 
 	r = io_uring_register_probe(ring, probe, 256);
 	if (r >= 0)
 		return probe;
 
-	free(probe);
+	uring_free(probe);
 	return NULL;
 }
 
@@ -214,7 +207,7 @@ struct io_uring_probe *io_uring_get_probe(void)
 
 void io_uring_free_probe(struct io_uring_probe *probe)
 {
-	free(probe);
+	uring_free(probe);
 }
 
 static int __fls(int x)
@@ -336,10 +329,7 @@ ssize_t io_uring_mlock_size_params(unsigned entries, struct io_uring_params *p)
 		cq_entries = 2 * entries;
 	}
 
-	page_size = sysconf(_SC_PAGESIZE);
-	if (page_size < 0)
-		page_size = 4096;
-
+	page_size = get_page_size();
 	return rings_size(entries, cq_entries, page_size);
 }
 
diff --git a/src/syscall.c b/src/syscall.c
index 5923fbb..cfb3ee2 100644
--- a/src/syscall.c
+++ b/src/syscall.c
@@ -1,6 +1,16 @@
 /* SPDX-License-Identifier: MIT */
 #define _DEFAULT_SOURCE
 
+/*
+ * Functions in this file require libc, only build them when we use libc.
+ *
+ * Note:
+ * liburing's tests still need these functions.
+ */
+#if defined(LIBURING_NOLIBC) && !defined(LIBURING_BUILD_TEST)
+# error "This file should only be compiled for libc build"
+#endif
+
 /*
  * Will go away once libc support is there
  */
@@ -11,7 +21,6 @@
 #include "liburing/io_uring.h"
 #include "syscall.h"
 
-
 int __sys_io_uring_register(int fd, unsigned opcode, const void *arg,
 			    unsigned nr_args)
 {
diff --git a/src/syscall.h b/src/syscall.h
index 9eff968..82f5db0 100644
--- a/src/syscall.h
+++ b/src/syscall.h
@@ -29,13 +29,13 @@
 # endif
 #elif defined __mips__
 # ifndef __NR_io_uring_setup
-#  define __NR_io_uring_setup           (__NR_Linux + 425)
+#  define __NR_io_uring_setup		(__NR_Linux + 425)
 # endif
 # ifndef __NR_io_uring_enter
-#  define __NR_io_uring_enter           (__NR_Linux + 426)
+#  define __NR_io_uring_enter		(__NR_Linux + 426)
 # endif
 # ifndef __NR_io_uring_register
-#  define __NR_io_uring_register        (__NR_Linux + 427)
+#  define __NR_io_uring_register	(__NR_Linux + 427)
 # endif
 #else /* !__alpha__ and !__mips__ */
 # ifndef __NR_io_uring_setup
@@ -49,9 +49,22 @@
 # endif
 #endif
 
-
+/*
+ * Don't put this below the #include "arch/$arch/syscall.h", that
+ * file may need it.
+ */
 struct io_uring_params;
 
+
+#ifdef LIBURING_NOLIBC
+#  if defined(__x86_64__) || defined(__i386__)
+#    include "arch/x86/syscall.h"
+#  else
+#    error "The arch is currently not supported to build liburing without libc"
+#  endif
+#endif
+
+
 /*
  * System calls
  */
@@ -68,12 +81,12 @@ static inline void *ERR_PTR(intptr_t n)
 	return (void *) n;
 }
 
-static inline intptr_t PTR_ERR(void *ptr)
+static inline intptr_t PTR_ERR(const void *ptr)
 {
 	return (intptr_t) ptr;
 }
 
-static inline bool IS_ERR(void *ptr)
+static inline bool IS_ERR(const void *ptr)
 {
 	return uring_unlikely((uintptr_t) ptr >= (uintptr_t) -4095UL);
 }
@@ -81,30 +94,40 @@ static inline bool IS_ERR(void *ptr)
 static inline int ____sys_io_uring_register(int fd, unsigned opcode,
 					    const void *arg, unsigned nr_args)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_io_uring_register(fd, opcode, arg, nr_args);
+#else
 	int ret;
-
 	ret = syscall(__NR_io_uring_register, fd, opcode, arg, nr_args);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int ____sys_io_uring_setup(unsigned entries,
 					 struct io_uring_params *p)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_io_uring_setup(entries, p);
+#else
 	int ret;
-
 	ret = syscall(__NR_io_uring_setup, entries, p);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int ____sys_io_uring_enter2(int fd, unsigned to_submit,
 					  unsigned min_complete, unsigned flags,
 					  sigset_t *sig, int sz)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_io_uring_enter(fd, to_submit, min_complete, flags,
+					  sig, sz);
+#else
 	int ret;
-
 	ret = syscall(__NR_io_uring_enter, fd, to_submit, min_complete, flags,
 		      sig, sz);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int ____sys_io_uring_enter(int fd, unsigned to_submit,
@@ -118,50 +141,68 @@ static inline int ____sys_io_uring_enter(int fd, unsigned to_submit,
 static inline void *uring_mmap(void *addr, size_t length, int prot, int flags,
 			       int fd, off_t offset)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_mmap(addr, length, prot, flags, fd, offset);
+#else
 	void *ret;
-
 	ret = mmap(addr, length, prot, flags, fd, offset);
 	return (ret == MAP_FAILED) ? ERR_PTR(-errno) : ret;
+#endif
 }
 
 static inline int uring_munmap(void *addr, size_t length)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_munmap(addr, length);
+#else
 	int ret;
-
 	ret = munmap(addr, length);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int uring_madvise(void *addr, size_t length, int advice)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_madvise(addr, length, advice);
+#else
 	int ret;
-
 	ret = madvise(addr, length, advice);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int uring_getrlimit(int resource, struct rlimit *rlim)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_getrlimit(resource, rlim);
+#else
 	int ret;
-
 	ret = getrlimit(resource, rlim);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int uring_setrlimit(int resource, const struct rlimit *rlim)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_setrlimit(resource, rlim);
+#else
 	int ret;
-
 	ret = setrlimit(resource, rlim);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int uring_close(int fd)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_close(fd);
+#else
 	int ret;
-
 	ret = close(fd);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 #endif
diff --git a/test/Makefile b/test/Makefile
index 2936469..1a10a24 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -21,8 +21,8 @@ ifdef CONFIG_HAVE_ARRAY_BOUNDS
 endif
 
 CXXFLAGS ?= $(CFLAGS)
-override CFLAGS += $(XCFLAGS)
-override CXXFLAGS += $(XCFLAGS) -std=c++11
+override CFLAGS += $(XCFLAGS) -DLIBURING_BUILD_TEST
+override CXXFLAGS += $(XCFLAGS) -std=c++11 -DLIBURING_BUILD_TEST
 LDFLAGS ?=
 override LDFLAGS += -L../src/ -luring
 
@@ -153,11 +153,22 @@ test_targets += sq-full-cpp
 endif
 all_targets += sq-full-cpp
 
-helpers = helpers.o
+#
+# Build ../src/syscall.c manually from test's Makefile to support
+# liburing nolibc.
+#
+# Functions in ../src/syscall.c require libc to work with, if we
+# build liburing without libc, we don't have those functions
+# in liburing.a. So build it manually here.
+#
+helpers = helpers.o ../src/syscall.o
 
 all: ${helpers} $(test_targets)
 
-helpers.o: helpers.c helpers.c
+../src/syscall.o: ../src/syscall.c
+	$(QUIET_CC)$(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $<
+
+helpers.o: helpers.c
 	$(QUIET_CC)$(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $<
 
 %: %.c ${helpers} helpers.h
-- 
2.30.2

[PATCH v2 RFC liburing 5/5] Add LIBURING_NOLIBC variable and edit src/Makefile

[Ammar Faizi]

Conditonal variable to enable nolibc build.

Link: https://github.com/axboe/liburing/issues/443
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 configure    |  7 +++++++
 src/Makefile | 13 ++++++++++++-
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index 92f51bd..6bdcead 100755
--- a/configure
+++ b/configure
@@ -358,6 +358,13 @@ print_config "has_memfd_create" "$has_memfd_create"
 
 
 #############################################################################
+if test "$LIBURING_NOLIBC" = "y" -o "$LIBURING_NOLIBC" = "yes" -o "$LIBURING_NOLIBC" = "1"; then
+  output_sym "LIBURING_NOLIBC"
+  LIBURING_NOLIBC="yes"
+else
+  LIBURING_NOLIBC="no"
+fi
+print_config "LIBURING_NOLIBC" "$LIBURING_NOLIBC"
 
 if test "$__kernel_rwf_t" = "yes"; then
   output_sym "CONFIG_HAVE_KERNEL_RWF_T"
diff --git a/src/Makefile b/src/Makefile
index 5e46a9d..bc42675 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -32,11 +32,22 @@ endif
 
 all: $(all_targets)
 
-liburing_srcs := setup.c queue.c syscall.c register.c
+liburing_srcs := setup.c queue.c register.c
+
+ifeq ($(LIBURING_NOLIBC),y)
+	liburing_srcs += nolibc.c
+	override CFLAGS += -nostdlib -nolibc -nodefaultlibs -ffreestanding -fno-stack-protector -fpic
+	override CPPFLAGS += -nostdlib -nolibc -nodefaultlibs -ffreestanding -fno-stack-protector -fpic
+	override LINK_FLAGS += -nostdlib -nolibc -nodefaultlibs -fpic
+else
+	liburing_srcs += syscall.c
+endif
 
 liburing_objs := $(patsubst %.c,%.ol,$(liburing_srcs))
 liburing_sobjs := $(patsubst %.c,%.os,$(liburing_srcs))
 
+$(liburing_srcs): syscall.h lib.h
+
 $(liburing_objs) $(liburing_sobjs): include/liburing/io_uring.h
 
 %.os: %.c
-- 
2.30.2

Re: [PATCH v2 RFC liburing 1/5] test/{iopoll,read-write}: Use `io_uring_free_probe()` instead of `free()`

[Jens Axboe]

On 10/7/21 12:31 AM, Ammar Faizi wrote:
> `io_uring_free_probe()` should really be used to free the return value
> of `io_uring_get_probe_ring()`. As we may not always allocate it with
> `malloc()`. For example, to support no libc build [1].

Applied, thanks.

-- 
Jens Axboe

Re: [PATCH v2 RFC liburing 2/5] test/cq-size: Don't use `errno` to check liburing's functions

[Jens Axboe]

On 10/7/21 12:31 AM, Ammar Faizi wrote:
> When we build liburing without libc, we can't check `errno` variable
> with respect to liburing's functions. Don't do that it in test.
> 
> Note:
> The tests themselves can still use `errno` to check error from
> functions that come from the libc, but not liburing.

Applied, thanks.

-- 
Jens Axboe

Re: [PATCH v2 RFC liburing 4/5] Add no libc build support

[Jens Axboe]

On 10/7/21 12:31 AM, Ammar Faizi wrote:
> Create `src/nolibc.c` to substitute libc functions like `memset()`,
> `malloc()` and `free()`. Only build this file when we build liburing
> without libc.
> 
> Wrap libc functions with `static inline` functions, defined in
> `src/lib.h`, currently we have:
>  1) `get_page_size`
>  2) `uring_memset`
>  3) `uring_malloc`
>  4) `uring_free`
> 
> Add conditional preprocessor in `src/{syscall,lib}.h` to use arch
> dependent and nolibc functions when we build liburing without libc.
> 
> Extra notes for tests:
>  1) Functions in `src/syscall.c` require libc to work.
>  2) Tests require functions in `src/syscall.c`.
> 
> So we build `src/syscall.c` manually from test's Makefile.
> 
> The Makefile in `src/` dir still builds `src/syscall.c` when we
> compile liburing with libc.

Why are we jumping through hoops on the naming? You add a uring_memset()
that the lib is supposed to use, then that calls memset() with libc or
__uring_memset() if we're building without libc. Why not just have the
nolibc build provide memset() and avoids this churn? Not just in terms
of the immediate patch, that matters less. But longer term where
inevitably a memset() or similar will be added to the existing code
base.

-- 
Jens Axboe

Re: [PATCH v2 RFC liburing 4/5] Add no libc build support

[Ammar Faizi]

On Thu, Oct 7, 2021 at 7:27 PM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 10/7/21 12:31 AM, Ammar Faizi wrote:
> > Create `src/nolibc.c` to substitute libc functions like `memset()`,
> > `malloc()` and `free()`. Only build this file when we build liburing
> > without libc.
> >
> > Wrap libc functions with `static inline` functions, defined in
> > `src/lib.h`, currently we have:
> >  1) `get_page_size`
> >  2) `uring_memset`
> >  3) `uring_malloc`
> >  4) `uring_free`
> >
> > Add conditional preprocessor in `src/{syscall,lib}.h` to use arch
> > dependent and nolibc functions when we build liburing without libc.
> >
> > Extra notes for tests:
> >  1) Functions in `src/syscall.c` require libc to work.
> >  2) Tests require functions in `src/syscall.c`.
> >
> > So we build `src/syscall.c` manually from test's Makefile.
> >
> > The Makefile in `src/` dir still builds `src/syscall.c` when we
> > compile liburing with libc.
>
> Why are we jumping through hoops on the naming? You add a uring_memset()
> that the lib is supposed to use, then that calls memset() with libc or
> __uring_memset() if we're building without libc. Why not just have the
> nolibc build provide memset() and avoids this churn? Not just in terms
> of the immediate patch, that matters less. But longer term where
> inevitably a memset() or similar will be added to the existing code
> base.
>

Ah, good point. The syscall wrappers gave me this bad suggestion. I know
we have a rationale to wrap syscalls because we need to eliminate `errno`
being used in C files. But we don't actually need extra wrappers for
`memset()`, `malloc()` and `free()`.

I agree that we can just provide `memset()`, `malloc()` and `free()` for
the nolibc build without having extra wrappers like that.

I will address this and send the rest as a PATCHSET.

-- 
Ammar Faizi

[PATCHSET liburing 0/4] Add no libc support for x86-64 arch

[Ammar Faizi]

Hi everyone,

This is a patchset after 2 RFC to support build liburing without libc.

In this patchset, I introduce no libc support for x86-64 arch.
Hopefully, one day we can get support for other architectures as well.

Motivation:
Currently liburing depends on libc. We want to make liburing can be
built without libc.

This idea firstly posted as an issue on the liburing GitHub
repository here: https://github.com/axboe/liburing/issues/443

The subject of the issue is: "An option to use liburing without libc?".

On Mon, Sep 27, 2021 at 4:18 PM Mahdi Rakhshandehroo <notifications@github.com> wrote:
> There are a couple of issues with liburing's libc dependency:
> 
>  1) libc implementations of errno, malloc, pthread etc. tend to
>     pollute the binary with unwanted global/thread-local state.
>     This makes reentrancy impossible and relocations expensive.
>  2) libc doesn't play nice with non-POSIX threading models, like
>     green threads with small stack sizes, or direct use of the
>     clone() system call. This makes interop with other
>     languages/runtimes difficult.
> 
> One could use the raw syscall interface to io_uring to address these
> concerns, but that would be somewhat painful, so it would be nice
> for liburing to support this use case out of the box. Perhaps
> something like a NOLIBC macro could be added which, if defined,
> would patch out libc constructs and replace them with non-libc
> wrappers where applicable. A few API changes might be necessary for
> the non-libc case (e.g. io_uring_get_probe/io_uring_free_probe), but
> it shouldn't break existing applications as long as it's opt-in.

----------------------------------------------------------------

Explanation about the changes:

- New directory for arch dependent files. We create a new directory
  `src/arch`. This is where the arch dependent sources live. Currently
  we only have single arch support `src/arch/x86`. This directory
  contains crafted syscalls written in inline Assembly and get page
  size function.

- Currently, liburing uses 4 libc functions, they are:
   1) `malloc`
   2) `free`
   3) `memset`
   4) `sysconf` (to get the page size).
  
  To support nolibc build, we provide our own functions in `src/nolibc.c`.

- Procedure to free the return value of `io_uring_get_probe_{,ring}`.
  Currently, several tests use `free()` to free the return value of
  this *probe* functions. But since these changes we should always
  use `io_uring_free_probe()`. Don't use `free()`.

- Don't use `errno` to check error from liburing functions on tests.
  We want the tests still work properly with liburing no libc.

----------------------------------------------------------------

How to build liburing without libc?

You can just simply add `export LIBURING_NOLIBC=y` before run the
build. Be sure to run `make clean` if you have dirty build, just to
ensure consistency.

  ammarfaizi2@integral:~/project/now/liburing$ export LIBURING_NOLIBC=y
  ammarfaizi2@integral:~/project/now/liburing$ ./configure
  prefix                        /usr
  includedir                    /usr/include
  libdir                        /usr/lib
  libdevdir                     /usr/lib
  relativelibdir                
  mandir                        /usr/man
  datadir                       /usr/share
  stringop_overflow             yes
  array_bounds                  yes
  __kernel_rwf_t                yes
  __kernel_timespec             yes
  open_how                      no
  statx                         yes
  C++                           yes
  has_ucontext                  yes
  has_memfd_create              yes
  LIBURING_NOLIBC               yes
  CC                            gcc
  CXX                           g++
  ammarfaizi2@integral:~/project/now/liburing$ taskset -c 0,1,2,3 make -j4

Make sure you see the `LIBURING_NOLIBC` with `yes`.

----------------------------------------------------------------

Extra improvements of using liburing build without libc:

1) The file size of liburing.so is reduced.

  With libc:
    186906 src/liburing.a
    116136 src/liburing.so.2.1.0
    303042 total

  Without libc:
    168152 src/liburing.a
    104136 src/liburing.so.2.1.0
    272288 total

2) Efficient function call. We inline all `syscall` instructions with
   inline Assembly. This greatly reduces the data movement, as syscall
   only clobbers %rax, %rcx and %r11. Plus it is compatible with the
   kernel style return value, so no need a branch to catch error from
   `errno` variable anymore.

   With libc, we may spend more extra time to save caller saved
   registers just to perform a syscall, because if we use libc, every
   syscall is wrapped with a function call.

   Another extra thing is when we need to check `errno` variable, that
   will cost more extra call to `__errno_location` and extra branches
   (as per we implement the kernel style return value).

   Without libc, the generated Assembly code is also smaller. For
   example, we can take a look at this generated Assembly code of
   `__io_uring_sqring_wait` function.

  With libc:

    0000000000003340 <__io_uring_sqring_wait>:
      3340: f3 0f 1e fa           endbr64 
      3344: 48 83 ec 10           sub    $0x10,%rsp
      3348: 8b b7 c4 00 00 00     mov    0xc4(%rdi),%esi
      334e: 31 c9                 xor    %ecx,%ecx
      3350: 31 d2                 xor    %edx,%edx
      3352: 6a 08                 push   $0x8
      3354: 41 b8 04 00 00 00     mov    $0x4,%r8d
      335a: 45 31 c9              xor    %r9d,%r9d
      335d: bf aa 01 00 00        mov    $0x1aa,%edi
      3362: 31 c0                 xor    %eax,%eax
      3364: e8 17 ef ff ff        call   2280 <syscall@plt>
      3369: 5a                    pop    %rdx
      336a: 59                    pop    %rcx
      336b: 41 89 c0              mov    %eax,%r8d
      336e: 85 c0                 test   %eax,%eax
      3370: 79 0b                 jns    337d <__io_uring_sqring_wait+0x3d>
      3372: e8 49 ee ff ff        call   21c0 <__errno_location@plt>
      3377: 44 8b 00              mov    (%rax),%r8d
      337a: 41 f7 d8              neg    %r8d
      337d: 44 89 c0              mov    %r8d,%eax
      3380: 48 83 c4 08           add    $0x8,%rsp
      3384: c3                    ret
      3385: 66 2e 0f 1f 84 00 00  cs nopw 0x0(%rax,%rax,1)
      338c: 00 00 00 
      338f: 90                    nop


  Without libc:

    0000000000001e20 <__io_uring_sqring_wait>:
      1e20: f3 0f 1e fa           endbr64 
      1e24: 31 d2                 xor    %edx,%edx
      1e26: 8b bf c4 00 00 00     mov    0xc4(%rdi),%edi
      1e2c: 45 31 c0              xor    %r8d,%r8d
      1e2f: b8 aa 01 00 00        mov    $0x1aa,%eax
      1e34: 41 ba 04 00 00 00     mov    $0x4,%r10d
      1e3a: 41 b9 08 00 00 00     mov    $0x8,%r9d
      1e40: 89 d6                 mov    %edx,%esi
      1e42: 0f 05                 syscall
      1e44: c3                    ret

3) More portable shared library. Sometimes we meet a case where libc
   version is not compatible with other versions of libc.

   Now, as we do not depend on libc, it's easier to distribute the
   liburing.so without worrying about libc version anymore. As long as
   the architecture is the same and the kernel version is compatible,
   that should not be a problem.
----------------------------------------------------------------
This patchset:
  - Drop extra wrappers for `malloc()`, `free()` and `memset()`.

  - Fix UAF bug in test/thread-exit. I found this after I changed the
    function name `uring_free()` to `free()` for nolibc build.

  - 2 patches have been applied in RFC v2, drop them. Add one extra
    patch to fix the UAF bug. So we have 4 patches here.

RFC v2:
  - Rebase the work based on commit 326ed975d49e8c7b ("configure: add
    openat2.h for open_how and RESOLVE_* flags").

  - Fix the patches order, make sure fix up the tests first, add
    nolibc sources, and then add a variable build to enable it.

  - Fix incorrect data type for `__arch_impl_mmap()` offset. It was
    `int` (that's not right). The proper data type is `off_t`.

  - Always use `long` or `void *` to contain the return value of
    syscall in `__arch_impl_*` functions.

  - Rename `src/no_libc.` to `src/nolibc.c`.

  - Reduce the number of patches to 5, it was 6.

Link: [RFC v1] https://lore.kernel.org/io-uring/20211006144911.1181674-1-ammar.faizi@students.amikom.ac.id/T/
Link: [RFC v2] https://lore.kernel.org/io-uring/20211007063157.1311033-1-ammar.faizi@students.amikom.ac.id/
----------------------------------------------------------------
Ammar Faizi (4):
      test/thread-exit: Fix use after free bug
      Add arch dependent directory and files
      Add no libc build support
      Add LIBURING_NOLIBC variable and edit src/Makefile

 configure              |   7 ++
 src/Makefile           |  13 ++-
 src/arch/x86/lib.h     |  26 ++++++
 src/arch/x86/syscall.h | 200 +++++++++++++++++++++++++++++++++++++++++++
 src/lib.h              |  44 ++++++++++
 src/nolibc.c           |  48 +++++++++++
 src/queue.c            |  14 +--
 src/register.c         |  12 +--
 src/setup.c            |  17 +---
 src/syscall.c          |  11 ++-
 src/syscall.h          |  71 +++++++++++----
 test/Makefile          |  19 +++-
 test/thread-exit.c     |  16 +++-
 13 files changed, 442 insertions(+), 56 deletions(-)
 create mode 100644 src/arch/x86/lib.h
 create mode 100644 src/arch/x86/syscall.h
 create mode 100644 src/lib.h
 create mode 100644 src/nolibc.c

-- 
Ammar Faizi

[PATCH liburing 1/4] test/thread-exit: Fix use after free bug

[Ammar Faizi]

When I added support for nolibc x86-64, I found this test failed.

Long story short, we provide our own `free()` that always unmaps the VM
with `munmap()`. It makes the CQE return -EFAULT because the kernel
reads unmapped user memory from the pending `write()` SQE.

I believe this test can run properly with libc build because `free()`
from libc doesn't always unmap the memory, instead it uses free list on
the userspace and the freed heap may still be userspace addressable.

Fix this by deferring the free.

Cc: Jens Axboe <axboe@kernel.dk>
Fixes: 2edfa3f84bcc44612b7a04caf1f048f5406fcc7a ("Add test case for thread exiting with pending IO")
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 test/thread-exit.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/test/thread-exit.c b/test/thread-exit.c
index 7f66028..05509fb 100644
--- a/test/thread-exit.c
+++ b/test/thread-exit.c
@@ -26,8 +26,18 @@ struct d {
 	unsigned long off;
 	int pipe_fd;
 	int err;
+	int i;
 };
 
+char *g_buf[NR_IOS] = {NULL};
+
+static void free_g_buf(void)
+{
+	int i;
+	for (i = 0; i < NR_IOS; i++)
+		free(g_buf[i]);
+}
+
 static void *do_io(void *data)
 {
 	struct d *d = data;
@@ -36,6 +46,7 @@ static void *do_io(void *data)
 	int ret;
 
 	buffer = t_malloc(WSIZE);
+	g_buf[d->i] = buffer;
 	memset(buffer, 0x5a, WSIZE);
 	sqe = io_uring_get_sqe(d->ring);
 	if (!sqe) {
@@ -55,8 +66,6 @@ static void *do_io(void *data)
 	ret = io_uring_submit(d->ring);
 	if (ret != 2)
 		d->err++;
-
-	free(buffer);
 	return NULL;
 }
 
@@ -103,6 +112,7 @@ int main(int argc, char *argv[])
 	d.pipe_fd = fds[0];
 	d.err = 0;
 	for (i = 0; i < NR_IOS; i++) {
+		d.i = i;
 		memset(&thread, 0, sizeof(thread));
 		pthread_create(&thread, NULL, do_io, &d);
 		pthread_join(thread, NULL);
@@ -125,7 +135,9 @@ int main(int argc, char *argv[])
 		io_uring_cqe_seen(&ring, cqe);
 	}
 
+	free_g_buf();
 	return d.err;
 err:
+	free_g_buf();
 	return 1;
 }
-- 
2.30.2

[PATCH liburing 2/4] Add arch dependent directory and files

[Ammar Faizi]

Create a new directory `src/arch` to save arch dependent sources.
Add support start from x86-64, add syscalls crafted in Assembly code
and lib (currently the lib only contains get page size function).

Link: https://github.com/axboe/liburing/issues/443
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 src/arch/x86/lib.h     |  26 ++++++
 src/arch/x86/syscall.h | 200 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 226 insertions(+)
 create mode 100644 src/arch/x86/lib.h
 create mode 100644 src/arch/x86/syscall.h

diff --git a/src/arch/x86/lib.h b/src/arch/x86/lib.h
new file mode 100644
index 0000000..0d4b321
--- /dev/null
+++ b/src/arch/x86/lib.h
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: MIT */
+
+#ifndef LIBURING_ARCH_X86_LIB_H
+#define LIBURING_ARCH_X86_LIB_H
+
+#ifndef LIBURING_LIB_H
+#  error "This file should be included from src/lib.h (liburing)"
+#endif
+
+#if defined(__x86_64__)
+
+static inline long __arch_impl_get_page_size(void)
+{
+	return 4096;
+}
+
+#else /* #if defined(__x86_64__) */
+
+/*
+ * TODO: Add x86 (32-bit) support here.
+ */
+#error "x86 (32-bit) is currently not supported"
+
+#endif /* #if defined(__x86_64__) */
+
+#endif /* #ifndef LIBURING_ARCH_X86_LIB_H */
diff --git a/src/arch/x86/syscall.h b/src/arch/x86/syscall.h
new file mode 100644
index 0000000..be18165
--- /dev/null
+++ b/src/arch/x86/syscall.h
@@ -0,0 +1,200 @@
+/* SPDX-License-Identifier: MIT */
+
+#ifndef LIBURING_ARCH_X86_SYSCALL_H
+#define LIBURING_ARCH_X86_SYSCALL_H
+
+#ifndef LIBURING_SYSCALL_H
+#  error "This file should be included from src/syscall.h (liburing)"
+#endif
+
+#if defined(__x86_64__)
+/**
+ * Note for syscall registers usage (x86-64):
+ *   - %rax is the syscall number.
+ *   - %rax is also the return value.
+ *   - %rdi is the 1st argument.
+ *   - %rsi is the 2nd argument.
+ *   - %rdx is the 3rd argument.
+ *   - %r10 is the 4th argument (**yes it's %r10, not %rcx!**).
+ *   - %r8  is the 5th argument.
+ *   - %r9  is the 6th argument.
+ *
+ * `syscall` instruction will clobber %r11 and %rcx.
+ *
+ * After the syscall returns to userspace:
+ *   - %r11 will contain %rflags.
+ *   - %rcx will contain the return address.
+ *
+ * IOW, after the syscall returns to userspace:
+ *   %r11 == %rflags and %rcx == %rip.
+ */
+
+static inline void *__arch_impl_mmap(void *addr, size_t length, int prot,
+				     int flags, int fd, off_t offset)
+{
+	void *rax;
+	register int r10 __asm__("r10") = flags;
+	register int r8 __asm__("r8") = fd;
+	register off_t r9 __asm__("r9") = offset;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)
+		: "a"(__NR_mmap),	/* %rax */
+		  "D"(addr),		/* %rdi */
+		  "S"(length),		/* %rsi */
+		  "d"(prot),		/* %rdx */
+		  "r"(r10),		/* %r10 */
+		  "r"(r8),		/* %r8  */
+		  "r"(r9)		/* %r9  */
+		: "memory", "rcx", "r11"
+	);
+	return rax;
+}
+
+static inline int __arch_impl_munmap(void *addr, size_t length)
+{
+	long rax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)		/* %rax */
+		: "a"(__NR_munmap),	/* %rax */
+		  "D"(addr),		/* %rdi */
+		  "S"(length)		/* %rsi */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+static inline int __arch_impl_madvise(void *addr, size_t length, int advice)
+{
+	long rax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)		/* %rax */
+		: "a"(__NR_madvise),	/* %rax */
+		  "D"(addr),		/* %rdi */
+		  "S"(length),		/* %rsi */
+		  "d"(advice)		/* %rdx */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+static inline int __arch_impl_getrlimit(int resource, struct rlimit *rlim)
+{
+	long rax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)		/* %rax */
+		: "a"(__NR_getrlimit),	/* %rax */
+		  "D"(resource),	/* %rdi */
+		  "S"(rlim)		/* %rsi */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+static inline int __arch_impl_setrlimit(int resource, const struct rlimit *rlim)
+{
+	long rax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)		/* %rax */
+		: "a"(__NR_setrlimit),	/* %rax */
+		  "D"(resource),	/* %rdi */
+		  "S"(rlim)		/* %rsi */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+static inline int __arch_impl_close(int fd)
+{
+	long rax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)		/* %rax */
+		: "a"(__NR_close),	/* %rax */
+		  "D"(fd)		/* %rdi */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+static inline int __arch_impl_io_uring_register(int fd, unsigned opcode,
+						const void *arg,
+						unsigned nr_args)
+{
+	long rax;
+	register unsigned r10 __asm__("r10") = nr_args;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)			/* %rax */
+		: "a"(__NR_io_uring_register),	/* %rax */
+		  "D"(fd),			/* %rdi */
+		  "S"(opcode),			/* %rsi */
+		  "d"(arg),			/* %rdx */
+		  "r"(r10)			/* %r10 */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+static inline int __arch_impl_io_uring_setup(unsigned entries,
+					     struct io_uring_params *p)
+{
+	long rax;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)			/* %rax */
+		: "a"(__NR_io_uring_setup),	/* %rax */
+		  "D"(entries),			/* %rdi */
+		  "S"(p)			/* %rsi */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+static inline int __arch_impl_io_uring_enter(int fd, unsigned to_submit,
+					     unsigned min_complete,
+					     unsigned flags, sigset_t *sig,
+					     int sz)
+{
+	long rax;
+	register unsigned r10 __asm__("r10") = flags;
+	register sigset_t *r8 __asm__("r8") = sig;
+	register int r9 __asm__("r9") = sz;
+
+	__asm__ volatile(
+		"syscall"
+		: "=a"(rax)
+		: "a"(__NR_io_uring_enter),	/* %rax */
+		  "D"(fd),			/* %rdi */
+		  "S"(to_submit),		/* %rsi */
+		  "d"(min_complete),		/* %rdx */
+		  "r"(r10),			/* %r10 */
+		  "r"(r8),			/* %r8  */
+		  "r"(r9)			/* %r9  */
+		: "memory", "rcx", "r11"
+	);
+	return (int) rax;
+}
+
+#else /* #if defined(__x86_64__) */
+
+/*
+ * TODO: Add x86 (32-bit) support here.
+ */
+#error "x86 (32-bit) is currently not supported"
+
+#endif /* #if defined(__x86_64__) */
+
+#endif /* #ifndef LIBURING_ARCH_X86_SYSCALL_H */
-- 
2.30.2

[PATCH liburing 3/4] Add no libc build support

[Ammar Faizi]

Create `src/nolibc.c` to provide libc functions like `memset()`,
`malloc()` and `free()`. Only build this file when we build liburing
without libc.

Add `get_page_size()` function to get the page size. When we build
with libc, it uses `sysconf(_SC_PAGESIZE)`, otherwise it uses arch
dependent implementation function that we provide at `src/arch` dir.

Add conditional preprocessor in `src/syscall.h` to use arch dependent
syscalls written in Assembly when we build liburing without libc.

Extra notes for tests:
 1) Functions in `src/syscall.c` require libc to work.
 2) Tests require functions in `src/syscall.c`.

So we build `src/syscall.c` manually from the test's Makefile.

The Makefile in `src/` dir still builds `src/syscall.c` when we
build liburing with libc.

Link: https://github.com/axboe/liburing/issues/443
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 src/lib.h      | 44 +++++++++++++++++++++++++++++++
 src/nolibc.c   | 48 ++++++++++++++++++++++++++++++++++
 src/queue.c    | 14 +++-------
 src/register.c | 12 +++------
 src/setup.c    | 17 +++---------
 src/syscall.c  | 11 +++++++-
 src/syscall.h  | 71 +++++++++++++++++++++++++++++++++++++++-----------
 test/Makefile  | 19 +++++++++++---
 8 files changed, 183 insertions(+), 53 deletions(-)
 create mode 100644 src/lib.h
 create mode 100644 src/nolibc.c

diff --git a/src/lib.h b/src/lib.h
new file mode 100644
index 0000000..4c147f5
--- /dev/null
+++ b/src/lib.h
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: MIT */
+#ifndef LIBURING_LIB_H
+#define LIBURING_LIB_H
+
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#ifdef LIBURING_NOLIBC
+#  if defined(__x86_64__) || defined(__i386__)
+#    include "arch/x86/lib.h"
+#  else
+#    error "The arch is currently not supported to build liburing without libc"
+#  endif
+#endif
+
+#ifndef offsetof
+# define offsetof(TYPE, FIELD) ((size_t) &((TYPE *)0)->FIELD)
+#endif
+
+#ifndef container_of
+# define container_of(PTR, TYPE, FIELD) ({				\
+	__typeof__(((TYPE *)0)->FIELD) *__FIELD_PTR = (PTR);		\
+	(TYPE *)((char *) __FIELD_PTR - offsetof(TYPE, FIELD));		\
+})
+#endif
+
+
+static inline long get_page_size(void)
+{
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_get_page_size();
+#else
+	long page_size;
+
+	page_size = sysconf(_SC_PAGESIZE);
+	if (page_size < 0)
+		page_size = 4096;
+
+	return page_size;
+#endif
+}
+
+#endif /* #ifndef LIBURING_LIB_H */
diff --git a/src/nolibc.c b/src/nolibc.c
new file mode 100644
index 0000000..cdf0e04
--- /dev/null
+++ b/src/nolibc.c
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: MIT */
+
+#ifndef LIBURING_NOLIBC
+# error "This file should only be compiled for no libc build"
+#endif
+
+#include "lib.h"
+#include "syscall.h"
+
+void *memset(void *s, int c, size_t n)
+{
+	size_t i;
+	unsigned char *p = s;
+
+	for (i = 0; i < n; i++)
+		p[i] = (unsigned char) c;
+
+	return s;
+}
+
+struct uring_heap {
+	size_t		len;
+	char		user_p[];
+};
+
+void *malloc(size_t len)
+{
+	struct uring_heap *heap;
+
+	heap = uring_mmap(NULL, sizeof(*heap) + len, PROT_READ | PROT_WRITE,
+			  MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+	if (IS_ERR(heap))
+		return NULL;
+
+	heap->len = sizeof(*heap) + len;
+	return heap->user_p;
+}
+
+void free(void *p)
+{
+	struct uring_heap *heap;
+
+	if (uring_unlikely(!p))
+		return;
+
+	heap = container_of(p, struct uring_heap, user_p);
+	uring_munmap(heap, heap->len);
+}
diff --git a/src/queue.c b/src/queue.c
index 9af29d5..eb0c736 100644
--- a/src/queue.c
+++ b/src/queue.c
@@ -1,19 +1,11 @@
 /* SPDX-License-Identifier: MIT */
 #define _POSIX_C_SOURCE 200112L
 
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#include <unistd.h>
-#include <string.h>
-#include <stdbool.h>
-
+#include "lib.h"
+#include "syscall.h"
+#include "liburing.h"
 #include "liburing/compat.h"
 #include "liburing/io_uring.h"
-#include "liburing.h"
-#include "liburing/barrier.h"
-
-#include "syscall.h"
 
 /*
  * Returns true if we're not using SQ thread (thus nobody submits but us)
diff --git a/src/register.c b/src/register.c
index 074223f..1f2c409 100644
--- a/src/register.c
+++ b/src/register.c
@@ -1,18 +1,12 @@
 /* SPDX-License-Identifier: MIT */
 #define _POSIX_C_SOURCE 200112L
 
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#include <sys/resource.h>
-#include <unistd.h>
-#include <string.h>
-
+#include "lib.h"
+#include "syscall.h"
+#include "liburing.h"
 #include "liburing/compat.h"
 #include "liburing/io_uring.h"
-#include "liburing.h"
 
-#include "syscall.h"
 
 int io_uring_register_buffers_update_tag(struct io_uring *ring, unsigned off,
 					 const struct iovec *iovecs,
diff --git a/src/setup.c b/src/setup.c
index 4f006de..5543468 100644
--- a/src/setup.c
+++ b/src/setup.c
@@ -1,18 +1,12 @@
 /* SPDX-License-Identifier: MIT */
 #define _DEFAULT_SOURCE
 
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <unistd.h>
-#include <string.h>
-#include <stdlib.h>
-#include <signal.h>
-
+#include "lib.h"
+#include "syscall.h"
+#include "liburing.h"
 #include "liburing/compat.h"
 #include "liburing/io_uring.h"
-#include "liburing.h"
 
-#include "syscall.h"
 
 static void io_uring_unmap_rings(struct io_uring_sq *sq, struct io_uring_cq *cq)
 {
@@ -336,10 +330,7 @@ ssize_t io_uring_mlock_size_params(unsigned entries, struct io_uring_params *p)
 		cq_entries = 2 * entries;
 	}
 
-	page_size = sysconf(_SC_PAGESIZE);
-	if (page_size < 0)
-		page_size = 4096;
-
+	page_size = get_page_size();
 	return rings_size(entries, cq_entries, page_size);
 }
 
diff --git a/src/syscall.c b/src/syscall.c
index 5923fbb..cfb3ee2 100644
--- a/src/syscall.c
+++ b/src/syscall.c
@@ -1,6 +1,16 @@
 /* SPDX-License-Identifier: MIT */
 #define _DEFAULT_SOURCE
 
+/*
+ * Functions in this file require libc, only build them when we use libc.
+ *
+ * Note:
+ * liburing's tests still need these functions.
+ */
+#if defined(LIBURING_NOLIBC) && !defined(LIBURING_BUILD_TEST)
+# error "This file should only be compiled for libc build"
+#endif
+
 /*
  * Will go away once libc support is there
  */
@@ -11,7 +21,6 @@
 #include "liburing/io_uring.h"
 #include "syscall.h"
 
-
 int __sys_io_uring_register(int fd, unsigned opcode, const void *arg,
 			    unsigned nr_args)
 {
diff --git a/src/syscall.h b/src/syscall.h
index 9eff968..82f5db0 100644
--- a/src/syscall.h
+++ b/src/syscall.h
@@ -29,13 +29,13 @@
 # endif
 #elif defined __mips__
 # ifndef __NR_io_uring_setup
-#  define __NR_io_uring_setup           (__NR_Linux + 425)
+#  define __NR_io_uring_setup		(__NR_Linux + 425)
 # endif
 # ifndef __NR_io_uring_enter
-#  define __NR_io_uring_enter           (__NR_Linux + 426)
+#  define __NR_io_uring_enter		(__NR_Linux + 426)
 # endif
 # ifndef __NR_io_uring_register
-#  define __NR_io_uring_register        (__NR_Linux + 427)
+#  define __NR_io_uring_register	(__NR_Linux + 427)
 # endif
 #else /* !__alpha__ and !__mips__ */
 # ifndef __NR_io_uring_setup
@@ -49,9 +49,22 @@
 # endif
 #endif
 
-
+/*
+ * Don't put this below the #include "arch/$arch/syscall.h", that
+ * file may need it.
+ */
 struct io_uring_params;
 
+
+#ifdef LIBURING_NOLIBC
+#  if defined(__x86_64__) || defined(__i386__)
+#    include "arch/x86/syscall.h"
+#  else
+#    error "The arch is currently not supported to build liburing without libc"
+#  endif
+#endif
+
+
 /*
  * System calls
  */
@@ -68,12 +81,12 @@ static inline void *ERR_PTR(intptr_t n)
 	return (void *) n;
 }
 
-static inline intptr_t PTR_ERR(void *ptr)
+static inline intptr_t PTR_ERR(const void *ptr)
 {
 	return (intptr_t) ptr;
 }
 
-static inline bool IS_ERR(void *ptr)
+static inline bool IS_ERR(const void *ptr)
 {
 	return uring_unlikely((uintptr_t) ptr >= (uintptr_t) -4095UL);
 }
@@ -81,30 +94,40 @@ static inline bool IS_ERR(void *ptr)
 static inline int ____sys_io_uring_register(int fd, unsigned opcode,
 					    const void *arg, unsigned nr_args)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_io_uring_register(fd, opcode, arg, nr_args);
+#else
 	int ret;
-
 	ret = syscall(__NR_io_uring_register, fd, opcode, arg, nr_args);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int ____sys_io_uring_setup(unsigned entries,
 					 struct io_uring_params *p)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_io_uring_setup(entries, p);
+#else
 	int ret;
-
 	ret = syscall(__NR_io_uring_setup, entries, p);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int ____sys_io_uring_enter2(int fd, unsigned to_submit,
 					  unsigned min_complete, unsigned flags,
 					  sigset_t *sig, int sz)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_io_uring_enter(fd, to_submit, min_complete, flags,
+					  sig, sz);
+#else
 	int ret;
-
 	ret = syscall(__NR_io_uring_enter, fd, to_submit, min_complete, flags,
 		      sig, sz);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int ____sys_io_uring_enter(int fd, unsigned to_submit,
@@ -118,50 +141,68 @@ static inline int ____sys_io_uring_enter(int fd, unsigned to_submit,
 static inline void *uring_mmap(void *addr, size_t length, int prot, int flags,
 			       int fd, off_t offset)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_mmap(addr, length, prot, flags, fd, offset);
+#else
 	void *ret;
-
 	ret = mmap(addr, length, prot, flags, fd, offset);
 	return (ret == MAP_FAILED) ? ERR_PTR(-errno) : ret;
+#endif
 }
 
 static inline int uring_munmap(void *addr, size_t length)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_munmap(addr, length);
+#else
 	int ret;
-
 	ret = munmap(addr, length);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int uring_madvise(void *addr, size_t length, int advice)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_madvise(addr, length, advice);
+#else
 	int ret;
-
 	ret = madvise(addr, length, advice);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int uring_getrlimit(int resource, struct rlimit *rlim)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_getrlimit(resource, rlim);
+#else
 	int ret;
-
 	ret = getrlimit(resource, rlim);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int uring_setrlimit(int resource, const struct rlimit *rlim)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_setrlimit(resource, rlim);
+#else
 	int ret;
-
 	ret = setrlimit(resource, rlim);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 static inline int uring_close(int fd)
 {
+#ifdef LIBURING_NOLIBC
+	return __arch_impl_close(fd);
+#else
 	int ret;
-
 	ret = close(fd);
 	return (ret < 0) ? -errno : ret;
+#endif
 }
 
 #endif
diff --git a/test/Makefile b/test/Makefile
index 2936469..1a10a24 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -21,8 +21,8 @@ ifdef CONFIG_HAVE_ARRAY_BOUNDS
 endif
 
 CXXFLAGS ?= $(CFLAGS)
-override CFLAGS += $(XCFLAGS)
-override CXXFLAGS += $(XCFLAGS) -std=c++11
+override CFLAGS += $(XCFLAGS) -DLIBURING_BUILD_TEST
+override CXXFLAGS += $(XCFLAGS) -std=c++11 -DLIBURING_BUILD_TEST
 LDFLAGS ?=
 override LDFLAGS += -L../src/ -luring
 
@@ -153,11 +153,22 @@ test_targets += sq-full-cpp
 endif
 all_targets += sq-full-cpp
 
-helpers = helpers.o
+#
+# Build ../src/syscall.c manually from test's Makefile to support
+# liburing nolibc.
+#
+# Functions in ../src/syscall.c require libc to work with, if we
+# build liburing without libc, we don't have those functions
+# in liburing.a. So build it manually here.
+#
+helpers = helpers.o ../src/syscall.o
 
 all: ${helpers} $(test_targets)
 
-helpers.o: helpers.c helpers.c
+../src/syscall.o: ../src/syscall.c
+	$(QUIET_CC)$(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $<
+
+helpers.o: helpers.c
 	$(QUIET_CC)$(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $<
 
 %: %.c ${helpers} helpers.h
-- 
2.30.2

[PATCH liburing 4/4] Add LIBURING_NOLIBC variable and edit src/Makefile

[Ammar Faizi]

Conditonal variable to enable nolibc build.

Link: https://github.com/axboe/liburing/issues/443
Signed-off-by: Ammar Faizi <ammar.faizi@students.amikom.ac.id>
---
 configure    |  7 +++++++
 src/Makefile | 13 ++++++++++++-
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index 92f51bd..6bdcead 100755
--- a/configure
+++ b/configure
@@ -358,6 +358,13 @@ print_config "has_memfd_create" "$has_memfd_create"
 
 
 #############################################################################
+if test "$LIBURING_NOLIBC" = "y" -o "$LIBURING_NOLIBC" = "yes" -o "$LIBURING_NOLIBC" = "1"; then
+  output_sym "LIBURING_NOLIBC"
+  LIBURING_NOLIBC="yes"
+else
+  LIBURING_NOLIBC="no"
+fi
+print_config "LIBURING_NOLIBC" "$LIBURING_NOLIBC"
 
 if test "$__kernel_rwf_t" = "yes"; then
   output_sym "CONFIG_HAVE_KERNEL_RWF_T"
diff --git a/src/Makefile b/src/Makefile
index 5e46a9d..bc42675 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -32,11 +32,22 @@ endif
 
 all: $(all_targets)
 
-liburing_srcs := setup.c queue.c syscall.c register.c
+liburing_srcs := setup.c queue.c register.c
+
+ifeq ($(LIBURING_NOLIBC),y)
+	liburing_srcs += nolibc.c
+	override CFLAGS += -nostdlib -nolibc -nodefaultlibs -ffreestanding -fno-stack-protector -fpic
+	override CPPFLAGS += -nostdlib -nolibc -nodefaultlibs -ffreestanding -fno-stack-protector -fpic
+	override LINK_FLAGS += -nostdlib -nolibc -nodefaultlibs -fpic
+else
+	liburing_srcs += syscall.c
+endif
 
 liburing_objs := $(patsubst %.c,%.ol,$(liburing_srcs))
 liburing_sobjs := $(patsubst %.c,%.os,$(liburing_srcs))
 
+$(liburing_srcs): syscall.h lib.h
+
 $(liburing_objs) $(liburing_sobjs): include/liburing/io_uring.h
 
 %.os: %.c
-- 
2.30.2